Parse User-Agent: 3 simple rules for a safe guess

To parse User-Agent strings, treat the value as a claim and look for evidence elsewhere. A User-Agent string is a claim about the client. It can tell you the browser family, the operating system and whether the client says it is a bot. It cannot prove any of that. The useful part is the comparison with the network and the behaviour.

The Bot detector (UA) endpoint matches a string against a maintained list of search, social and malicious agents.

curl "https://api2.globus.studio/v2/bot_ua?ua=Googlebot&format=json"

What you get

A parse User-Agent response has a bot flag and the matched signature. The match is case-insensitive and looks at the whole string. When ua is missing, the endpoint uses the request header.

What a parse User-Agent check misses

A new bot with an unknown string passes. A real browser with an unusual extension can look like a bot. A script can copy a known string exactly. The signature list is a filter for known agents, not a classifier for all traffic.

Combine it with the IP check

To parse User-Agent strings with context, call the UA endpoint and the IP endpoint. A Googlebot string from a Google range is a good sign. The same string from a residential connection is a mismatch. Log both results with the address and the decision.

Client hints

Modern browsers send fewer details in the User-Agent and more in client hints. A string that looks generic may come from a real browser. Check Sec-CH-UA headers when they are present, and do not treat a short string as a bot by itself.

A practical policy

Use the UA result to choose a rate limit, a challenge or a log level. Do not use it to ban an address. A false positive on a real customer costs more than a slow request from a scraper.

$ua = json_decode(file_get_contents(
    'https://api2.globus.studio/v2/bot_ua?ua=' . urlencode($agent) . '&format=json'
), true);
if (($ua['bot'] ?? '') === 'true') {
    // lower the rate limit, do not block
}

Limits

Parse User-Agent strings with the list that ships with the service; it is updated with the release, not in real time. The endpoint is fast and rate limited. Read the endpoint documentation for parameters and examples.

Client hints and short strings

A modern browser can send a short User-Agent and put the details in Sec-CH-UA headers. To parse User-Agent strings from a modern browser, read both places. A short string is not a bot signal by itself. Check the hints when they are present, and keep the UA check for the claimed identity.

Two checks, one decision

Call the UA endpoint for the string and the IP endpoint for the network. If both agree that the client is a crawler, allow a higher rate. If they disagree, lower the rate and log the pair. A single check gives a flag; the pair gives a reason.

# Compare the two answers.
curl "https://api2.globus.studio/v2/bot_ua?ua=Googlebot&format=json"
curl "https://api2.globus.studio/v2/bot_ip?ip=66.249.66.1&format=json"

Privacy and logs

A parse User-Agent result can contain a device model, an app version and a language. The parsed result is still personal data when it is tied to an address. Treat it as personal data when you store it. Keep the raw string for a short period and store the parsed category longer. A support request can be answered with the category and the timestamp.

Parsing User-Agent strings with an API