The New Age of Bots: How ONX Stops Scrapers on the PHP Origin
Bot traffic now makes up to 40% of business application traffic. Here is how ONX stops scrapers and malicious bots on the origin — without adding latency for real guests.
Just three years ago, bots in server logs were predictable: Googlebot for indexing, UptimeRobot for monitoring, and the occasional attacker probing for SQL injections. Today, the reality is starkly different. The explosion of LLMs and web-scraping pipelines means that up to 40% of total traffic on business applications now consists of unwanted bots—ranging from academic crawlers (EMAIL-EU-Academic-Crawler) to aggressive commercial AI scrapers (MultimodalCrawler, L9OutreachBot).
For developers, sysadmins, and CTOs, this is no longer just a security concern. It directly impacts latency, triggers unwanted auto-scaling in the cloud, and inflates infrastructure invoices.
Why Traditional Defenses Fail
Relying on legacy defense mechanisms is now both operationally and technically unsustainable:
- robots.txt is Dead: For AI bots, robots.txt is an optional suggestion at best—and a roadmap of high-value targets to scrape first at worst.
- Relying on User-Agent is Naive: Advanced scraping networks dynamically rotate browser headers (Headless Chrome) and route requests through residential proxies.
- Manual Blocklists Don't Scale: Manually adding new bot signatures to Nginx/Apache configs leads to endless production deployments and bloated, unmaintainable rule sets.
How ONX Handles Bot Traffic on the Origin
ONX runs on the PHP origin. Every request that reaches PHP is inspected before your booking, form or catalogue code runs. Known crawlers are checked; the rest is scored against the endpoint profile for this site.
[ Incoming Request ]
│
▼
┌────────────────────────────────────────────────────────┐
│ LEVEL 1: Identity & Reverse DNS │
│ - Hard-check IP ranges & rDNS for Google, Bing, etc. │
│ - Mismatched IP vs. User-Agent -> block / tarpit │
└─────────────────────────┬──────────────────────────────┘
│ (Unknown / Unverified Traffic)
▼
┌────────────────────────────────────────────────────────┐
│ LEVEL 2: Endpoint scoring (ONX agent) │
│ - Parameter and URI rate limits │
│ - Behavioral scoring on this site's profile │
│ - Low score -> Challenge / Block │
└─────────────────────────┬──────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ LEVEL 3: Abuse limits │
│ - Dynamic Rate Limiting (HTTP 429) │
│ - Temporary IP quarantine │
└─────────────────────────┬──────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ EU CONSOLE (operations telemetry) │
│ - Confirmed pattern on site A can strengthen │
│ the class prior for other ONX sites │
└────────────────────────────────────────────────────────┘
1. Identity verification (PTR & A record checks)
Fake search engine crawlers are a common attack vector. Rather than trusting the User-Agent string, the ONX agent does a reverse DNS lookup (PTR) and a forward lookup (A). If a client claims to be Googlebot but comes from an unverified datacenter IP, the request is blocked or tarpitted — it never reaches booking, login or catalogue code.
2. Endpoint scoring on this site
Header spoofing is cheap. ONX scores the request against what this URL normally looks like — methods, parameters, length and frequency. A burst on /booking or a parameter shape that the hotel prior has never seen is treated as abuse, not as a guest.
3. Class prior, not instant immunity
Confirmed incidents from the EU console can strengthen the prior for the same class of site — hotel, municipality or shop. That is operations telemetry under a DPA, not a live feed of every visitor request.
If a new bot or scraping farm strains a site in the network, confirmed signals can strengthen the class prior for other ONX sites. The next hotel or shop does not have to rediscover the same pattern from zero.
Business Impact for Your Infrastructure
Bot management with ONX is about keeping the application usable:
- Less junk on PHP. Stopping scrapers on the origin reduces worker and database load on booking and catalogue URLs.
- Catalogue and booking protection. Keeps prices, calendars and product listings for real visitors, not automated harvesters.
- A warmer start for the next site. A confirmed pattern on one hotel can improve the prior for the next, after an operator has reviewed it.
- Cleaner operations view. Fewer bot hits in the console, so a real guest incident is easier to see.
Conclusion
Bot management in 2026 is no longer about reactively updating IP blocklists. With ONX, you deploy an agent on the PHP origin that learns this site and takes verified signals from the EU network. The result is a more resilient booking, form or shop — and less junk load for real guests.
Back to the journal