User-Agent spoofing works because the header has no signature. Anyone can send a Googlebot string from a residential connection. A single header does not prove who is on the other side. Treating it as proof leads to two failures: scrapers pass as crawlers, and real browsers get blocked because their string looks unusual.
We built two endpoints around that problem. Bot detector (UA) matches the string against a maintained list of search, social and malicious agents. Bot definition (IP) checks the address against published crawler ranges and hosting data. Neither is enough alone.
What User-Agent spoofing hides
User-Agent spoofing exploits the claim. The header says what the client wants you to believe about its browser, version and platform. That is useful for compatibility. Old browsers need different CSS. A mobile client may need a smaller image. The string is a hint for rendering, not a credential.
User-Agent spoofing is easy. curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" sends the same string Google uses. A script can rotate strings on every request. The header has no signature and no shared secret.
Combine three signals
Start with the claimed identity. The UA endpoint returns a bot flag, a category and the matched signature. Then check the network. Search engines publish their IP ranges, and the IP endpoint compares the address with those ranges and with hosting providers. Finally, look at behaviour: request rate, paths, session length and whether the client loads assets.
A request that claims Googlebot, comes from a residential ASN, ignores images and hits the same URL every two seconds is not a crawler. A request from a Google range with a fresh UA and a normal crawl pattern is probably fine.
curl "https://api2.globus.studio/v2/bot_ua?ua=Googlebot&format=json"
curl "https://api2.globus.studio/v2/bot_ip?ip=66.249.66.1&format=json"What to do with the result
User-Agent spoofing is cheap, so do not block on the string alone. Rate limit first. A suspicious request can be slowed, challenged or served a smaller page. Keep a log with the UA, IP, ASN and decision so you can review false positives later.
For known crawlers, verify the range before you allow a higher rate. For unknown clients, treat the UA as one feature among several. That approach protects the site without turning every new browser into a suspect.
Limits
A User-Agent spoofing check depends on a signature list that ages. New bots appear and old strings change. The IP endpoint depends on published ranges and hosting data, and some crawlers use addresses that are not obvious. The UA endpoint defaults to the request header when ua is missing, which is convenient for server-side checks but can surprise a proxy.
No single User-Agent spoofing check gives a final answer. The useful signal comes from the combination. Read the UA endpoint documentation and the IP endpoint documentation, then decide what your site should do with a mismatch.
What to log
Log the User-Agent, the client address, the ASN, the matched signature and the decision. Keep the raw string for a short period and the decision longer. When a customer reports a block, the log tells you whether the string, the network or the rate limit caused it.
A review queue helps. A request that fails two checks can be challenged instead of blocked. A request that fails one check can pass with a lower rate. The policy stays measurable, and you can change it after looking at real traffic.
Test the policy before you ship
Replay a day of access logs through the checks. Count how many real users would be challenged and how many known bots would pass. Adjust the thresholds before the rule reaches production. A policy that looks reasonable on paper can block a partner’s monitoring service on the first day.
Further reading: MDN, User-Agent header.