robots.txt looks like a configuration file and behaves like a small matching language. The syntax is loose, the file is read by many crawlers with slightly different habits, and a rule written for one path can block a whole section. The robots endpoint parses the file and answers one question against it: is this URL allowed for this crawler.
curl "https://api2.globus.studio/v2/robots?url=https%3A%2F%2Fglobus.studio%2F&check=https%3A%2F%2Fglobus.studio%2Fcart%2F"Groups and user agents
The parser returns each user-agent group with its rules. Consecutive user-agent lines are treated as one group, which is what Google documents: a block that starts with User-agent: Googlebot followed by User-agent: Bingbot applies to both.
A rule placed before any user-agent is ignored by crawlers. The endpoint keeps it out of the groups and adds a warning, because the line usually means the file was edited by hand and lost its header.
Matching follows the longest path
Three details decide the answer for a URL. The most specific user-agent group wins, so a Googlebot block overrides the wildcard block for Googlebot. Inside the group, the longest matching path wins. When two rules have the same length, Allow beats Disallow.
Wildcards and the end anchor are part of the language. An asterisk matches any sequence, and a trailing dollar sign anchors the pattern to the end of the path. A rule like /*.pdf$ blocks PDF files, while /*.pdf blocks any path that contains that text.
The response names the rule that matched, with the user-agent group it came from. A surprise is traceable to a line in the file instead of being a yes or no answer.
Missing files and errors
A crawler that receives a 404 for robots.txt assumes full permission. The endpoint reports status: 404 with allowed: true, which is the correct reading and different from a server error.
A 5xx answer is another case. Some crawlers treat an unreachable robots.txt as a reason to slow down or stop, and Google documents a cautious behaviour. The endpoint keeps the status and adds a warning instead of pretending the file does not exist.
What the parser does not do
The file is read as text up to 500 KB. The endpoint does not download the sitemaps that the file mentions, does not fetch the tested URL, and does not check whether the page is indexed. It answers one question about the rules, and the sitemap endpoint covers the map side.
A Sitemap: line is returned as a list, wherever it appears in the file. The directive is global: it is not scoped to a user-agent group, and a file with several lines publishes several maps. Crawl-delay is reported as an unknown directive, because Google ignores it and the matching here follows the documented rules rather than a per-crawler extension.
The endpoint also helps with a hosting migration. A robots.txt copied from a staging server often keeps a blanket Disallow, and that file is the last thing anyone reads before launch.
Where it helps
Before a launch, test the pages that must stay out of the index, such as a cart, a search result page or an internal filter. After a migration, test the addresses that must stay in. A rule that accidentally blocks /blog/ instead of /blog/draft/ costs weeks of traffic and is invisible until a report drops.
The reverse question matters too. When a page never appears in search, test whether the rules allow it before rewriting the content.
The endpoint caches the file for an hour, so a corrected robots.txt needs a cache window before the answer changes.
The catalogue page has the parameter table and a live example: robots.txt API.