A client came to us after their hosting bill tripled in a single month. No new features had launched. No marketing campaign was driving traffic. Their application was being hammered by bots — thousands of requests per minute hitting their API endpoints, scraping product data, and submitting junk through every form on the site. This is not unusual. It is the default state of the internet.
What Bots Actually Do to Your Application (and Your Bill)
Bot traffic is not an abstract security concern. It has direct business consequences.
Infrastructure costs climb. Every request your server processes costs money. When bots generate thousands of requests per hour, your compute and bandwidth bills reflect that. If you are running on auto-scaling infrastructure (most AWS or cloud-hosted applications are), bots can trigger scaling events that spin up additional servers to handle traffic that has zero business value. We have seen clients paying hundreds of dollars a month in unnecessary compute costs because nobody noticed the traffic was not human.
Content gets scraped. If your business depends on proprietary data — product listings, pricing, research, member-only content — bots can systematically copy it. Competitors or aggregator sites scrape your content and republish it, sometimes outranking you for your own material.
Forms become unusable. Contact forms, registration pages, login screens: bots will find and abuse them. The result is a database full of junk submissions, wasted time for staff sorting real enquiries from spam, and in the worst case, credential stuffing attacks where bots try stolen username and password combinations against your login page.
Analytics become unreliable. When a significant portion of your traffic is automated, your analytics data tells you nothing useful. Bounce rates, session durations, conversion funnels — all distorted by traffic that was never a potential customer.
Spotting the Signs
You do not always know bots are a problem until you look. Common indicators:
- Traffic spikes that do not correlate with any marketing activity or seasonal pattern
- Unusually high server resource usage relative to your known user base
- Form submissions with obviously fake data, or a sudden surge in submissions
- Requests to paths that do not exist in your application (bots probing for common vulnerabilities like
/wp-adminor/.env) - A single IP address or a small range of IPs generating disproportionate traffic
If your application runs behind Cloudflare, their analytics dashboard will show you the split between human and bot traffic immediately. If it does not, your server access logs will tell the story — look for user agents, request patterns, and volume anomalies.
Cloudflare as Your First Line of Defence
Cloudflare is the single most effective first step for bot protection, and the free tier covers more than most people expect.
Free plan. DDoS mitigation, basic bot detection via Browser Integrity Check, and a limited number of firewall rules. For a small to mid-size application with straightforward traffic patterns, this handles the bulk of unsophisticated bot traffic. We configure this as a baseline on every client project.
Pro plan. Adds the Web Application Firewall (WAF) with managed rulesets that cover the OWASP Top 10 vulnerabilities, plus more granular rate limiting. This is where protection starts getting serious — the managed rules catch common attack patterns without you needing to write custom logic.
Business plan. Advanced bot management, custom WAF rules, and more detailed analytics. For applications handling sensitive data or facing targeted attacks, this is the right tier.
One thing we have learned from experience: Cloudflare’s security settings need careful tuning when first deployed. On one project, overly aggressive challenge rules started blocking legitimate users — including the client’s own staff. The default settings are sensible, but every application has unique traffic patterns. Roll out new rules in log-only mode first, review what they would have blocked, and then enforce. This saves you from turning your security layer into a customer-blocking layer.
AWS WAF for Application-Layer Protection
If your application runs on AWS (behind an Application Load Balancer, CloudFront distribution, or API Gateway), AWS WAF gives you a second layer of protection that operates closer to your application.
Where Cloudflare catches threats at the edge before traffic reaches your infrastructure, AWS WAF lets you define rules based on your application’s specific logic. You can block requests based on:
- Request rate per IP (rate-based rules)
- Geographic origin
- Specific URI paths, query strings, or headers
- Known malicious IP reputation lists (AWS provides managed rule groups for this)
- Request body content, which is useful for protecting API endpoints
AWS WAF’s managed rule groups are particularly valuable. The AWS Managed Rules Bot Control rule group identifies and categorises bot traffic, letting you allow verified bots (search engine crawlers) while challenging or blocking everything else. The Core Rule Set covers common web exploits. You pay per rule group and per million requests evaluated, which keeps costs proportional to your actual traffic.
For one client, we configured AWS WAF rate-based rules to limit login attempts to 100 per five-minute window per IP. This stopped a credential stuffing attack within minutes of deployment, without affecting legitimate users who might retry a password a few times.
Rate Limiting and Custom Rules
Rate limiting is the single most underused protection mechanism we encounter. The concept is simple: limit how many requests a single source can make within a time window.
Effective rate limiting requires thinking about your application’s actual usage patterns. A product catalogue page might legitimately receive 10 requests per second from a user browsing quickly. Your login endpoint should never receive 50 requests per second from a single IP. Your contact form should not receive more than a few submissions per minute from anyone.
Set rate limits per endpoint, not globally. Be generous enough that legitimate users never hit the limit, strict enough that automated traffic gets caught. Both Cloudflare and AWS WAF support this, and using both gives you rate limiting at two layers — edge and application.
Custom rules let you get specific. Block traffic from countries where you have no customers. Challenge requests with suspicious user agents. Require JavaScript execution (which most simple bots cannot do) before allowing access to sensitive pages. These are not difficult to configure, but they require someone who understands both the tools and your application’s traffic patterns.
The Layered Defence Approach
No single tool stops everything. The approach we recommend and implement as part of our application support and infrastructure services:
- Cloudflare at the edge. Stops volumetric attacks, filters known bad actors, challenges suspicious traffic before it reaches your servers. This alone eliminates the majority of bot traffic.
- AWS WAF at the application layer. Catches what gets through Cloudflare, enforces application-specific rules, provides rate limiting tuned to your endpoints.
- Application-level protections. CAPTCHA or challenge pages on high-value forms. Account lockout policies. Logging and alerting for anomalous patterns.
- Monitoring and iteration. Review blocked traffic weekly. Tune rules as attack patterns change. Bots evolve; your defences need to evolve with them.
This layered model means no single point of failure. If a bot bypasses Cloudflare’s detection, WAF catches it. If a new attack pattern emerges that neither catches initially, application-level logging flags it for manual review and rule updates.
The cost of this setup ranges from free (Cloudflare free tier plus basic AWS WAF rules) to a few hundred dollars per month for comprehensive protection. Compare that to the cost of tripled hosting bills, scraped content, or a data breach from an unprotected login form.
If you are unsure where your application is exposed, we can help. Get a bot traffic audit for your application → — we will review your setup and recommend the right level of protection for your stack and budget.
Frequently Asked Questions
How do I know if bots are hitting my application? Look for traffic spikes with no matching marketing activity, disproportionate requests from a handful of IPs, and form submissions full of junk data. Cloudflare’s dashboard surfaces this immediately if you are already using it.
Is Cloudflare’s free plan enough for bot protection? Yes, for baseline defence. It covers DDoS mitigation, basic bot detection, and basic rate limiting (one rule). Step up to Pro or Business for managed rulesets and granular controls against credential stuffing or API abuse.
What’s the difference between Cloudflare and AWS WAF? Cloudflare is a reverse proxy that filters traffic at the edge; AWS WAF applies rules at the application layer inside your AWS stack. Using both gives you defence in depth.