To detect bot traffic, start with the network and then look at the claim. A search crawler, a monitoring probe and a credential-stuffing script all look different. The address is one signal. The network owner, the request rate and the claimed identity fill in the rest.
The Bot definition (IP) endpoint checks an address against known crawler ranges and returns a bot flag with a name when it finds a match.
curl "https://api2.globus.studio/v2/bot_ip?ip=66.249.66.1&format=json"How to detect bot traffic from the response
bot_detect is true when the address belongs to a known bot range. bot_name names the operator, for example google or bing. A false result means the address is not in the list. It does not prove the client is a person.
Why the network matters
A residential address that claims to be Googlebot is suspicious. A hosting address that sends a browser User-Agent is also worth a look. The ASN and hosting flag help separate a home connection from a data center. The UA endpoint covers the claimed identity. To detect bot traffic well, compare the two answers: the network and the string should agree.
Verify before you allow
If you grant a crawler a higher rate limit, verify the range first. Search engines publish their address ranges. A request from outside those ranges should not get the crawler budget even if the User-Agent matches.
What to do with a hit
Rate limit, challenge or log. Blocking every hit will catch some legitimate monitoring services and may hide a problem. A bot that respects robots.txt and loads assets is different from a script that hits the same search URL every second.
$bot = json_decode(file_get_contents(
'https://api2.globus.studio/v2/bot_ip?ip=' . urlencode($ip) . '&format=json'
), true);
if (!empty($bot['bot_detect'])) {
$limit = in_array($bot['bot_name'], ['google', 'bing'], true) ? 600 : 60;
}Limits
A detect bot traffic list covers known bots and changes over time. A new crawler may be missing. The endpoint is fast and rate limited. Cache the result per address for a few hours. Read the endpoint documentation for parameters and examples. If you detect bot traffic on every request, you add a network call to the critical path; cache the answer and re-check on a schedule.
ASN and hosting data
Two addresses can be in the same country and behave very differently. One belongs to a home connection, the other to a cloud provider. The ASN and the hosting flag separate them. A login from a hosting range is more suspicious than a login from a residential range in the same city.
A small score
To detect bot traffic, a score turns several weak signals into one decision. Start at zero. Add one for a bot hit, one for a hosting address and one for a mismatch between the claimed bot and the network. A score of zero or one can pass. A score of two can get a challenge. A score of three can be blocked or reviewed. The numbers are yours; the point is to make the decision visible and adjustable.
$score = 0;
$score += !empty($bot['bot_detect']) ? 1 : 0;
$score += ($ipdata['hosting'] ?? '0') === '1' ? 1 : 0;Monitoring services and small crawlers
When you detect bot traffic, remember that not every bot is hostile. An uptime monitor, a link checker and a search crawler all use automated clients. A hard block catches them along with the scrapers. A rate limit and a clear User-Agent policy separate the useful agents from the noise.
Keep an allow list for the monitors you run. Review it when a monitor changes its address range. An allow list that nobody maintains becomes a hole.
Related reading
The User-Agent guide covers the claimed identity. For anonymized traffic, see the Tor exit node check, and for the conceptual limits read why a string is not proof.
Further reading: Google, verifying Googlebot.