Reference
Parameters
| Name | Type | Required | Description |
|---|---|---|---|
url |
string | yes | Site URL or the full robots.txt URL. |
check |
string | no | URL to test against the rules. Without it check is null. |
user_agent |
string | no | User agent for the test. Default: Googlebot. |
Example request
curl -s "https://api2.globus.studio/v2/robots?url=https://globus.studio/"
robots.txt is a plain text file with a loose syntax and strict consequences. A rule that looks harmless can hide a whole catalogue from search, and an emptied file can do the opposite. This endpoint parses the file, groups the rules and answers whether one URL is allowed for a chosen crawler.
Groups, sitemaps and warnings
The parser returns each user-agent group with its allow and disallow rules, the sitemap addresses, and warnings for lines without a colon, rules placed before any user-agent, and groups with no rules at all. Consecutive user-agent lines are treated as one group, which is what Google documents.
curl -s "https://api2.globus.studio/v2/robots?url=https%3A%2F%2Fglobus.studio%2F&check=https%3A%2F%2Fglobus.studio%2Fcart%2F"Testing one URL
The check parameter runs the documented matching: the most specific user-agent group wins, the longest matching path wins, and Allow beats Disallow on a tie. Wildcards * and the trailing $ anchor are supported. The response names the rule that decided the answer, so a surprise is traceable.
A missing file is not an error. The endpoint returns status: 404 with allowed: true, because crawlers treat an absent robots.txt as full permission.
Limits
The file is read up to 500 KB, redirects are followed, and answers are cached for an hour. Only the supplied host is fetched.
Related
The full parameter table, every error code and a live request are on the documentation page. It opens in the same language as this page and needs no account.
A longer walkthrough, with the checks that matter and the limits worth planning for: robots.txt rules: what Google actually matches.