thatAPI - Robots.txt Analyzer
The Robots.txt Analyzer API parses and analyzes any website's robots.txt file, returning structured data about crawl rules, AI bot permissions, sitemaps, and overall quality. Key Features: - Analyze any URL (auto-fetches /robots.txt) or raw robots.txt content - AI bot coverage: checks rules for GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot, Google-Extended, and 15+ other AI crawlers - Returns…
8.9/10 popularity
135 ms avg latency
100% success rate
1 endpoints
The in-depth APIMemo review for this API hasn't been published yet —
the data below comes straight from the public marketplace listing.
thatAPI - Robots.txt Analyzer endpoints
| Method | Endpoint | Description |
|---|---|---|
| POST |
Analyze Robots.txt /v1/robots-txt/analyze |
Parse and analyze a website's robots.txt file. Returns structured rules, AI bot blocking status for 19 crawlers, sitemap references, crawl delays, and a quality score (0-100).… |
