bulkurlchecker-python

# bulkurlchecker

[![PyPI version](https://img.shields.io/pypi/v/bulkurlchecker.svg)](https://pypi.org/project/bulkurlchecker/)
[![Python versions](https://img.shields.io/pypi/pyversions/bulkurlchecker.svg)](https://pypi.org/project/bulkurlchecker/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

Python client for the [Bulk URL Checker](https://bulkurlchecker.com) API.

**Skip the proxy-rotation, rate-limiter, soft-404 detector, and retry classifier you would otherwise spend two weeks building.** Submit thousands of URLs, get status codes, redirect chains, and broken-link detection back as plain Python objects. Backed by a managed cloud service with residential proxies and per-domain throttling.

## Install

```bash
pip install bulkurlchecker
```

## 5-line example

```python
from bulkurlchecker import Client

client = Client(api_key="uck_live_...")
results = client.check_urls(["https://example.com", "https://example.org"])
for r in results.results:
    print(r.url, r.status_code, "BROKEN" if r.is_broken else "ok")
```

Get an API key at https://app.bulkurlchecker.com/dashboard/api-keys. First 300 URLs are free, no card required.

## What you get back

```python
results = client.check_urls(urls)

results.status            # 'completed' | 'paused' | 'failed' | 'cancelled'
results.timed_out         # True if the wait deadline passed (job still running)
results.total_urls        # how many URLs the engine accepted
results.completed_urls    # how many it finished checking
results.duplicates_removed
results.invalid_urls_rejected

for r in results.results:
    r.url                # the original URL you submitted
    r.final_url          # after redirects
    r.status_code        # 200, 301, 404, 429, 500, ...
    r.redirect_chain     # list of intermediate URLs
    r.is_broken          # True if the engine flagged this as broken
    r.is_soft_404        # True if 200 OK but page content says "not found"
    r.response_time_ms

# Convenience properties:
results.broken           # list of URLResult where is_broken == True
results.soft_404s        # list where is_soft_404 == True
```

## Command-line usage

Install with the `[cli]` extra to get the `bulkurlchecker` command:

```bash
pip install 'bulkurlchecker[cli]'
export BULKURLCHECKER_API_KEY=uck_live_...

# One URL per line in urls.txt → CSV on stdout
bulkurlchecker check urls.txt > report.csv

# Pipe from anywhere
sitemap-extractor mydomain.com | bulkurlchecker check - --output jsonl > report.jsonl

# Only show broken URLs, ad-hoc input
bulkurlchecker check --urls "https://example.com,https://example.org" --only-broken

# Two-step (submit, then poll & fetch later)
JOB=$(bulkurlchecker submit urls.txt)
bulkurlchecker status "$JOB"
bulkurlchecker results "$JOB" --output csv > report.csv
```

Output formats: `csv` (default), `json`, `jsonl`. Run `bulkurlchecker check --help` for the full flag list.

## Larger jobs: submit and poll

`check_urls()` blocks for up to 15 minutes server-side. For lists where the wait would time out, use the two-step pattern:

```python
job = client.submit(my_500k_urls)
print(f"Submitted {job.job_id}, {job.total_urls} URLs queued")

# Poll explicitly, or use the convenience method
done = client.wait_until_done(job.job_id, timeout=3600)

# Stream results in pages
for batch in client.iter_results(job.job_id, page_size=1000):
    for r in batch:
        if r.is_broken:
            print(r.url, r.status_code)
```

## Safe retries with `idempotency_key`

Pass an idempotency key on `submit()` or `check_urls()` to make retries safe under network failures. The server caches the response for 24 hours; a retry with the same key + same body returns the original result without creating a duplicate job.

```python
import uuid

key = str(uuid.uuid4())  # generate once per logical request

# First call: creates a new job.
job = client.submit(urls, idempotency_key=key)

# Network blip, no clean response received? Just retry with the same
# key -- you'll get the SAME job summary back, no duplicate submission.
same_job = client.submit(urls, idempotency_key=key)
# job.job_id == same_job.job_id
```

Same `idempotency_key` + different `urls` returns `409 Conflict` (raised as `ValidationError`) so client bugs that reuse a key against a new payload are caught loudly instead of silently mapping to the wrong cached response.

## Receiving webhooks

When a job finishes, we POST to your registered endpoint with a signed payload. Verify the signature before trusting the body — anyone who knows your endpoint URL can otherwise send fake events.

```python
from flask import Flask, request
from bulkurlchecker import verify_signature, InvalidSignatureError

SECRET = os.environ["MY_WEBHOOK_SECRET"]  # the signing_secret we showed once

app = Flask(__name__)

@app.route("/webhook/bulkurlchecker", methods=["POST"])
def webhook():
    try:
        verify_signature(
            request.get_data(),  # RAW bytes — not request.get_json()
            request.headers.get("Bulkurlchecker-Signature", ""),
            SECRET,
        )
    except InvalidSignatureError:
        return "", 401
    event = request.get_json()
    if event["type"] == "job.completed":
        job_id = event["data"]["job_id"]
        # ... fetch results, update your DB, ping Slack, etc ...
    return "", 200
```

Register endpoints + get secrets at https://app.bulkurlchecker.com/dashboard/webhooks (or via `POST /api/v2/webhooks/endpoints`).

`verify_signature()` enforces a 5-minute timestamp tolerance by default to defeat replays. Tune via `tolerance_seconds=`.

## Error handling

All errors derive from `BulkURLCheckerError`. Catch specific subclasses when you want to branch on the failure mode:

```python
from bulkurlchecker import (
    Client,
    BulkURLCheckerError,
    AuthenticationError,
    RateLimitError,
    QuotaError,
    ValidationError,
)

try:
    results = client.check_urls(urls)
except QuotaError as e:
    print(f"Out of credits. Top up at https://app.bulkurlchecker.com/billing")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s.")
except AuthenticationError:
    print("API key rejected — check it's not revoked.")
except ValidationError as e:
    print(f"Bad request: {e}")  # bad URLs, too many URLs, etc.
except BulkURLCheckerError as e:
    print(f"Other error: {e} (request_id={e.request_id})")
```

Every error carries `status_code`, `code` (server's machine-readable string), `request_id` (for support), and `details` (when the server provides them).

## Why use this instead of writing your own checker with httpx + asyncio?

Honest answer: for ≤500 URLs you don't need this. The standard `requests`/`httpx` toolchain handles it fine.

The wall hits at scale:

| Problem | Rolling your own | This SDK |
|---|---|---|
| Concurrency | `asyncio` + careful semaphores | done |
| Proxy rotation across residential IPs | $90+/mo Webshare / Bright Data subscription + custom code | done |
| Per-domain rate limiting (so you don't hammer one host) | wire it yourself | done |
| Distinguishing real 403 from "you got blocked" 403 | guess and check | done |
| Detecting soft 404s (200 OK + "not found" body) | regex / heuristic per template | done |
| Retry classification (transient vs permanent) | tune for weeks | done |
| Long-running job state (resume after crash) | Redis + queue + worker infra | done |
| Engineer time, weeks 1-4 | $$$ | nothing, ship today |

If you've already lost a weekend to httpx + proxy rotation, you know what we're talking about.

## Pricing

- **First month on us:** 15,000 URL checks on the Starter plan, no credit card.
  The first time an API or SDK integration runs successfully, the trial grows
  to a month of Pro: 50,000 URL checks, once per account.
- **After that:** 300 URL checks a month are included on every account.
- **Starter:** $9/month or $90/year, 15,000 URLs/month
- **Pro:** $29/month or $290/year, 50,000 URLs/month
- **Agency:** $99/month or $990/year, 200,000 URLs/month

Scheduled checks, with email, Slack and webhook alerts, are included on every
plan. One-time credit packs start at $5 for 3,000 checks, and pack credits
roll over. Current prices: https://bulkurlchecker.com/#pricing

## Links

- [Web app](https://app.bulkurlchecker.com)
- [REST API reference](https://bulkurlchecker.com/developers)
- [OpenAPI spec](https://api.bulkurlchecker.com/openapi.json)
- [GitHub](https://github.com/carlosofscience/bulkurlchecker-python)
- [Changelog](CHANGELOG.md)

## Stability

The SDK follows semver. While we're at 0.x, breaking changes can land in minor releases (we'll always note them in `CHANGELOG.md`). Once we hit 1.0 you can pin major versions safely.

## License

MIT. See [LICENSE](LICENSE).