website-tech-seo-ai-readiness

# Website Tech, SEO & llms.txt Check for Agents

**Returns one row per website (tech stack, SEO score, robots.txt AI-crawler rules, sitemap and llms.txt) for $1.50 per 1,000 sites.** You pay $0.0015 per site row, and error rows are free. Example: auditing 250 lead websites costs 250 × $0.0015 = **$0.38**, plus Apify's $0.00005 Actor-start fee.

**Fairquery** Apify Actor that returns **one structured row per website**: tech stack, title/meta/Open Graph/JSON-LD, an SEO issue list with a 0–100 score, **robots.txt rules for AI crawlers** (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …), XML sitemap, **llms.txt / llms-full.txt**, security headers, and a 0–100 **AI-readiness score**. Plain HTTP (no browser), so it's fast and cheap.

## When to use this Actor (LLM / tool selection)

Use it when an agent needs to:

- Enrich a lead/company list with "what is this site built on?" (CMS, framework, analytics, chat, payments, CDN, hosting)
- Run a quick on-page SEO check of homepages or landing pages
- Answer "does this site have an llms.txt?" / "does it block GPTBot or ClaudeBot?"
- Get a site's title, description, social links, feeds and structured-data types before summarizing it

Do **not** use it for full-site crawls, JS-rendered content (single-page apps may show `THIN_CONTENT`), Core Web Vitals, or backlink data.

## How it works

Per site (max 3 in parallel): `GET /robots.txt` → `GET <url>` → sitemap (from robots `Sitemap:` or `/sitemap.xml`, `/sitemap_index.xml`) → `/llms.txt` and `/llms-full.txt`. Identifies as `FairqueryBot/1.0`; by default skips pages that robots.txt disallows for generic crawlers. Tech detection uses ~75 HTML/header/cookie/meta-generator signatures from a single response.

## Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `urls` | `string[]` | – (required; 3 sample sites are prefilled in the Console form) | Domains or URLs |
| `checkRobots` | `bool` | `true` | robots.txt + AI-bot access map |
| `checkSitemap` | `bool` | `true` | Find sitemap and count URLs |
| `checkLlmsTxt` | `bool` | `true` | Check `/llms.txt` and `/llms-full.txt` |
| `respectRobotsTxt` | `bool` | `true` | Skip URLs disallowed for generic crawlers |
| `maxSites` | `int` | `500` | Safety cap per run |

```json
{ "urls": ["stripe.com", "https://www.example.org/pricing"], "checkSitemap": false }
```

## Output

Every row has `schema_version` (`1.0.0`), `type` (`site` | `error`), `source_url` (final fetched URL), `checked_at`.

```json
{
  "type": "site", "domain": "vercel.com", "status_code": 200, "https": true, "response_ms": 310,
  "title": "Agentic Infrastructure - Vercel", "meta_description": "…", "canonical": "https://vercel.com/", "lang": "en",
  "h1": ["…"], "word_count": 256, "open_graph": { "title": "…", "image": "…" }, "json_ld_types": ["Organization", "WebSite"],
  "technologies": [{ "name": "Next.js", "category": "JS framework", "evidence": "html \"/_next/static\"" }, { "name": "Vercel", "category": "Hosting", "evidence": "header x-vercel-id" }],
  "security_headers": { "strict_transport_security": true, "content_security_policy": true },
  "robots_txt_found": true, "ai_bots": { "GPTBot": "partial", "ClaudeBot": "partial" },
  "sitemap_found": true, "sitemap_url_count": 8787, "llms_txt_found": true, "llms_txt_title": "Vercel",
  "seo_score": 100, "seo_issues": [], "ai_readiness_score": 85
}
```

- `ai_bots[bot]`: `allowed` (no rules), `partial` (some paths disallowed), `blocked` (root disallowed).
- `seo_issues[]`: `{code, severity, message}`; codes include `NO_HTTPS`, `MISSING_TITLE`, `TITLE_TOO_LONG`, `MISSING_META_DESCRIPTION`, `MISSING_H1`, `MULTIPLE_H1`, `MISSING_CANONICAL`, `MISSING_LANG`, `MISSING_VIEWPORT`, `NOINDEX`, `MISSING_OPEN_GRAPH`, `MISSING_STRUCTURED_DATA`, `IMAGES_MISSING_ALT`, `THIN_CONTENT`, `MISSING_ROBOTS_TXT`, `MISSING_SITEMAP`. Score = 100 − 15/8/3 per high/medium/low issue.
- `ai_readiness_score`: llms.txt 25, llms-full.txt 5, AI bots allowed up to 20, JSON-LD 15, ≥150 words of server-rendered text 15, sitemap 10, meta description 10; −30 if `noindex`.

### Error rows (machine-readable, never charged)

```json
{ "type": "error", "code": "BLOCKED_OR_RATE_LIMITED", "message": "GET https://x.com/ returned HTTP 403.", "retryable": false, "status_code": 403, "input": "x.com" }
```

Codes: `INPUT_EMPTY`, `INVALID_URL`, `MAX_SITES_EXCEEDED`, `ROBOTS_DISALLOWED`, `FETCH_FAILED`, `HTTP_ERROR`, `BLOCKED_OR_RATE_LIMITED`.

### Run summary

Key-value store record **`RUN_SUMMARY`**: sites checked, errors, HTTP request count, average SEO and AI-readiness scores, llms.txt count, echoed input.

## Pricing

| Event | Price | When |
| --- | --- | --- |
| `site-checked` | $0.0015 | Each website row returned |
| `apify-actor-start` | $0.00005 | Once per run, per GB of memory (Apify default event) |

Error rows (`FETCH_FAILED`, `HTTP_ERROR`, `BLOCKED_OR_RATE_LIMITED`, `ROBOTS_DISALLOWED`, …) are **never charged**.

**Worked example:** 1,000 domains from a lead list where 60 cannot be fetched: 940 × $0.0015 = **$1.41**, plus $0.00005.

To cap spend, set a maximum cost per run in the run options; the Actor stops cleanly at the limit. `maxSites` caps how many sites one run checks. Prices are from the Store pricing tab (October 2026).

## Memory / runtime

Runs at a fixed **256 MB** (default and maximum; the check is network-bound, so more memory does not help), no browser, no proxy. Typically 1–3 s per site, 3 sites in parallel. Slow or rate-limiting sites take longer.

## MCP / Apify tools

Call from agents via [Apify MCP](https://mcp.apify.com) (`https://mcp.apify.com`, authenticated with your Apify token), Actor `fairquery/website-tech-seo-ai-readiness`:

```json
{ "actor": "fairquery/website-tech-seo-ai-readiness", "input": { "urls": ["stripe.com"] } }
```

Read dataset items (`type == "site"`) and `RUN_SUMMARY`. A standalone stdio MCP server lives in the GitHub repo under `mcp/`.

## Local stdio MCP server

This repo ships a **stdio MCP server** (TypeScript, `@modelcontextprotocol/sdk`) under [`mcp/`](./mcp) with single-purpose tools. Seven are **free** and run locally (no token); one optional bulk tool runs this Actor through the Apify API. `initialize` and `tools/list` work **without** a token.

| Tool | What it does | Paging | Cost |
|---|---|---|---|
| `web_quick_check` | 1-20 URLs: status, redirect chain, response time, content type, title | – | free |
| `web_seo_audit` | One URL: meta/OG/JSON-LD/headings/images, `seo_issues`, `seo_score` and `ai_readiness_score` (0-100); output is a reusable snapshot | – | free |
| `web_ai_crawler_access` | robots.txt rules per AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ...), `llms.txt` / `llms-full.txt` | – | free |
| `web_tech_stack` | Frameworks, CMS, hosting/CDN, analytics with evidence; `category` filter | `limit`/`offset` | free |
| `web_sitemap_info` | Sitemap discovery, index/children, URL list with lastmod; `contains`, `sort` | `limit`/`offset` | free |
| `web_security_headers` | HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-/Permissions-Policy, COOP; score + grade A-F | – | free |
| `web_compare_snapshots` | Diff two saved snapshots, a saved snapshot vs a fresh re-audit (follow-up), or two live URLs | – | free |
| `web_batch_audit` | Bulk audit of up to thousands of sites via the Apify Actor (formerly `check_websites`) | – | pay-per-event, needs `APIFY_TOKEN` |

Errors are named codes: `INVALID_URL`, `BLOCKED_PRIVATE_ADDRESS`, `DNS_FAILED`, `FETCH_FAILED`, `TIMEOUT`, `TOO_MANY_REDIRECTS`, `SITEMAP_NOT_FOUND`, `INVALID_ARGUMENTS`, `APIFY_TOKEN_MISSING`, `APIFY_ACTOR_RUN_FAILED`.

**Respectful fetching:** User-Agent `FairqueryBot ... Fairquery-MCP`, 15 s timeout, 1.5 MB HTML / 10 MB sitemap cap, max 5 redirects, and an SSRF guard that refuses hosts (including redirect targets) resolving to private, loopback, link-local or reserved addresses. If you run behind a fake-IP DNS proxy that maps all hosts into 198.18.0.0/15, set `FAIRQUERY_ALLOW_FAKE_IP_DNS=1`.

**Breaking change in v1.2.0:** `check_websites` → `web_batch_audit` (same input, same billing); `check_llms_txt` was removed, use `web_ai_crawler_access`.

```bash
cd mcp && npm install && npm run build
npm run test:live                                    # live test of every free tool, no token
node scripts/smoke-test.mjs                          # initialize + tools/list, no token
APIFY_TOKEN=... node scripts/smoke-test.mjs --call   # real web_batch_audit call
```

```json
{ "mcpServers": { "fairquery-web": { "command": "node", "args": ["/path/to/website-tech-seo-ai-readiness/mcp/dist/index.js"], "env": { "APIFY_TOKEN": "<optional, only for web_batch_audit>" } } } }
```

Docker: `docker build -t fairquery-web-mcp .` (the repo-root `Dockerfile` builds this MCP server and is what Glama uses; the Apify Actor image is built from `.actor/Dockerfile`). `glama.json` lists `fairquery` as maintainer.

## Works well after

- [Google Maps Scraper](https://apify.com/compass/crawler-google-places) or [Google Search Results Scraper](https://apify.com/apify/google-search-scraper): pass the website URLs they return into `urls` to see each business's tech stack, SEO basics and AI-crawler rules.
- Pair it with the [Website Contact & Company Enricher](https://apify.com/fairquery/website-contact-company-enricher) on the same domain list to get contacts and tech in two rows per company.

## Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| Run fails at once with `INPUT_EMPTY` | `urls` was empty. The three sample sites in the Console form are prefill only and are not sent with API calls. |
| `BLOCKED_OR_RATE_LIMITED` (HTTP 403/429) | The site blocks datacenter traffic or bots. These rows are free; the Actor does not use proxies or browsers to get around blocks. |
| `ROBOTS_DISALLOWED` | robots.txt disallows the page for generic crawlers. Set `respectRobotsTxt` to `false` only if you have the right to fetch the page. |
| Low SEO score or `THIN_CONTE