Bulk classification

Classify from the dashboard

Enter a single Page URL, or use the upload button beside it to select a CSV or TSV file (up to 1 MB). Use one column headed URL, Website, or Page URL; a headerless list of full http:// or https:// URLs also works. Other columns are ignored. Files with multiple URL columns must be reduced to one URL column before import.

The browser reads the file locally and shows the imported URL count. Duplicate URLs are removed, preserving their first position. Invalid or missing URLs and files with more than 100 unique URLs must be corrected before running. Selecting a file does not submit requests: choose Run classification when ready. Remove the file to return to your single URL. The API itself still accepts a JSON urls array, not file uploads.

Create a classification batch

POST /api/v1/scraping/classifications with your customer API key, Content-Type: application/json, and a saved UUID v4 Idempotency-Key. That UUID becomes batch.id. Save the exact body and key before sending. HTTP 202 means accepted, not completed.

Provide 1-100 public URLs, a question (1-2,000 characters), and 2-20 unique answer labels (1-80 characters each). Duplicate normalized URLs are collected once, preserving their first position. The body limit is 256 KiB. Source collection uses Markdown. Cookies, login and CAPTCHA bypass are not supported.

{
  "urls": [
    "https://example.com/about",
    "https://another-company.com"
  ],
  "question": "Does this company offer managed web scraping services?",
  "choices": [
    "Yes",
    "No",
    "Maybe"
  ]
}

Send the saved request once

Use your ScrapingBuddy API key, never an upstream provider key. Start with at least $1 available balance. Generate SCRAPINGBUDDY_BATCH_KEY once, save it with request.json, and reuse it on an uncertain retry. If submission is uncertain, GET the original ID first; only replay the same POST and key if it is not found. Replaying never restarts completed or failed work.

Create a UUID v4 with node -p 'crypto.randomUUID()', then set SCRAPINGBUDDY_BATCH_KEY to that saved value. Set SCRAPINGBUDDY_TARGET_URL for JavaScript and Python; their examples use one URL, and the urls array can contain up to 100. cURL reads your full request.json.

curl --fail-with-body -X POST "https://www.scrapingbuddy.com/api/v1/scraping/classifications" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY" \
  -H "Idempotency-Key: $SCRAPINGBUDDY_BATCH_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const batchKey = process.env.SCRAPINGBUDDY_BATCH_KEY;
const targetUrl = process.env.SCRAPINGBUDDY_TARGET_URL;
if (!apiKey || !batchKey || !targetUrl) {
  throw new Error("Set the required environment variables before running.");
}

const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/classifications`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Idempotency-Key": batchKey,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ urls: [targetUrl], question: "Does this company offer managed web scraping services?", choices: ["Yes","No","Maybe"] })
});
const data = await response.json();
if (!response.ok) {
  throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);

import os
import requests

api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
batch_key = os.environ["SCRAPINGBUDDY_BATCH_KEY"]
target_url = os.environ["SCRAPINGBUDDY_TARGET_URL"]
response = requests.post(
    "https://www.scrapingbuddy.com/api/v1/scraping/classifications",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Idempotency-Key": batch_key,
        "Content-Type": "application/json"
    },
    json={"urls": [target_url], "question": "Does this company offer managed web scraping services?", "choices": ["Yes","No","Maybe"]},
    timeout=30,
)
data = response.json()
if not response.ok:
    raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)

Examples use Bash, Node.js 20+, or Python with requests. View request and response fields.

Page scope and company context

By default, only the starting page is collected for each URL. Add depth: "deep" and maxPages: 2-30 (default 5) to discover relevant website pages. Add webSearch: true to search for independent public sources too. The budget includes the starting page. This is bounded link discovery, not an exhaustive website crawl; inaccessible or undiscovered pages can leave evidence incomplete.

companyContext is optional (1-3,000 characters). For a question such as "Is this our competitor?", describe your business and what counts as competition, either in the question or context. It is supplied separately from the target website content. Context is stored with this batch, not automatically reused in later requests.

Poll and handle each result separately

GET /api/v1/scraping/classifications/{id} with the same Authorization header. Poll at least 10 seconds apart while batch.status is queued or running. HTTP 202 indicates pending work; 200 indicates a terminal batch. A completed batch can contain failed rows, so inspect each item.status.

Each item has id, position, url, status, label, confidence, probabilities, insufficientEvidenceProbability, model, sourceUrls, warning and error. A classified label is exactly one of the supplied choices. sourceUrls identifies collected pages; it is not an independently verified citation for the decision.

insufficient_evidence, refused, failed, review, expired and cancelled have no answer label. Never turn null into No. queued and processing are pending. Confidence and probabilities are model estimates, not guarantees. A user-provided Maybe is distinct from the system insufficient-evidence outcome.

Batch status review means at least one upstream submission could not be confirmed. Model-call failures also remain per-row failures; neither is automatically retried. Contact support before creating replacement requests.

{
  "url": "https://example.com/about",
  "status": "classified",
  "label": "Yes",
  "confidence": 0.92,
  "sourceUrls": [
    "https://example.com/about"
  ],
  "error": null
}

History, cancellation and export

GET /api/v1/scraping/classifications?offset=0 returns up to 10 batch summaries and hasMore. Advance offset by 10. Fetch an individual batch for all results. Classification history is separate from /scraping/tasks and the ordinary Usage & history view.

POST /api/v1/scraping/classifications/{id}/cancel stops only URLs that have not started. In-flight collection and model calls are allowed to finish. Repeated cancellation is safe. Stopped entries remain in usage accounting; cancellation does not replenish the reserved allowance.

The dashboard provides JSON and CSV downloads. JSON preserves exact labels and types. CSV protects spreadsheet formula-like cell values with an apostrophe; use JSON when exact round-tripping matters. Results expire 24 hours after each starting page is collected. Export before expiry; metadata remains in history.

Retrieve the saved batch

Set SCRAPINGBUDDY_BATCH_ID to the UUID used as SCRAPINGBUDDY_BATCH_KEY. These examples perform one check. Poll at least 10 seconds apart only while queued or running, and honor Retry-After on 429. Stop on completed or review. Inspect every item rather than treating the batch status as an answer.

curl --fail-with-body -X GET "https://www.scrapingbuddy.com/api/v1/scraping/classifications/$SCRAPINGBUDDY_BATCH_ID" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY"

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const batchId = process.env.SCRAPINGBUDDY_BATCH_ID;
if (!apiKey || !batchId) {
  throw new Error("Set the required environment variables before running.");
}

const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/classifications/${encodeURIComponent(batchId)}`, {
  method: "GET",
  headers: {
    Authorization: `Bearer ${apiKey}`
  }
});
const data = await response.json();
if (!response.ok) {
  throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);

import os
import requests
from urllib.parse import quote

api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
batch_id = quote(os.environ["SCRAPINGBUDDY_BATCH_ID"], safe="")
response = requests.get(
    f"https://www.scrapingbuddy.com/api/v1/scraping/classifications/{batch_id}",
    headers={
        "Authorization": f"Bearer {api_key}"
    },
    timeout=30,
)
data = response.json()
if not response.ok:
    raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)

Examples use Bash, Node.js 20+, or Python with requests. View request and response fields.

Classification in an ordinary task

For one URL, POST /scraping/tasks also accepts analysis.choices with format: markdown. Use analysis.instruction for the question. Read task.analysis.decision only when task.status and task.analysis.status are ready. Its status is classified, insufficient_evidence or refused; only classified has a usable label.

Ordinary classification tasks use task IDs, six-second polling and ordinary task history. The /scraping/classifications endpoints instead use batch IDs, ten-second polling and per-item outcomes. Do not mix the two response shapes.

{
  "url": "https://example.com/about",
  "format": "markdown",
  "analysis": {
    "instruction": "Does this company offer managed web scraping services?",
    "choices": [
      "Yes",
      "No",
      "Maybe"
    ]
  }
}

Testing limits and errors

Classification has its own allowance of 100 URLs per account in a rolling 24-hour window, with one active batch at a time. Ordinary single-page tasks keep their separate allowance. Work proceeds through a server queue, so closing the browser does not cancel it. At most three root tasks run concurrently across the shared worker.

Deep tasks reserve their entire capacity budget, even if fewer pages are collected. This shared capacity limit is separate from wallet billing; unvisited pages are not charged. A shared server page cap can reject a batch with HTTP 429; reduce its size or return later. Failed and stopped entries also count. API keys retain the shared 60 requests/minute account limit; honor Retry-After.

400/413: invalid fields or body size. 401/403: authentication or access. 402: available balance below the $1 starting minimum. 404: missing batch or another account owns it. 409: changed idempotency payload or another active batch. 429: rate or usage cap. 503: service unavailable. DNS failures and inaccessible pages appear as per-URL failures after admission.

New runs charge provider costs multiplied by 1.5, per completed step. Collection and AI have separate holds. Content-only mode never calls AI. Classification uses input-only OpenAI Decisions billing on the server; keys never reach the browser. Earlier pilot tasks remain free.