Create a task

Collect a public page, with optional single-page or deep analysis.

01 / Request

Make a request

POST/api/v1/scraping/tasks

During the current testing stage, a valid customer API key is enough to create a public-page task; separate account and hostname approval is not required. Send one starting URL per task, with a JSON body of at most 8,192 bytes, including any analysis instruction and company context.

Persist a UUID v4 Idempotency-Key and your request before sending. An identical retry with the same key returns the existing task; changing the request with that key returns 409. A different key creates a new task.

Returns 202 for queued, submitting, processing, researching or analyzing tasks, otherwise 200. The response includes pollAfterSeconds: 6 and a Location header for the task.

Required headers

AuthorizationstringRequired: Bearer followed by your ScrapingBuddy customer API key.
Content-TypestringRequired: application/json.
Idempotency-KeyUUID v4Required: Generate once per new task. Save with the request and reuse on uncertain retries. This becomes its ID.

JSON body

urlstringRequired: Complete http:// or https:// URL, at most 2,048 characters. Use a public hostname on standard ports. IP literals, private addresses, embedded credentials and credential query parameters are rejected.
formathtml | markdownOptional: Default: html. Choose markdown for text-oriented workflows or html for your own parsing.
waitForobjectOptional: Optional. Use exactly one condition: {"text":"About us"}, {"element":"main","checkVisible":true}, or {"state":"domcontentloaded"}. state also accepts load. text and element are non-empty strings up to 300 characters. checkVisible defaults to false.
analysisobjectOptional: Optional. Omit to collect content without AI. Include instruction (1-2,000 characters) and optionally companyContext (1-3,000 characters). For multi-page research, add depth: "deep", maxPages (integer 2-30, default 5), and optional webSearch (boolean, default false), with format: "markdown". To classify this one URL, include choices (2-20 labels, 1-80 characters each, unique ignoring case) and use Markdown; read analysis.decision. Models and API keys are server-managed.

POST /scraping/tasks

# Save request.json with your public URL and format first.
# Reuse this file and the saved key for an uncertain retry.
curl --fail-with-body -X POST "https://www.scrapingbuddy.com/api/v1/scraping/tasks" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY" \
  -H "Idempotency-Key: $SCRAPINGBUDDY_TASK_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

POST /scraping/tasks

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const taskKey = process.env.SCRAPINGBUDDY_TASK_KEY;
const targetUrl = process.env.SCRAPINGBUDDY_TARGET_URL;
if (!apiKey || !taskKey || !targetUrl) {
  throw new Error("Set the required environment variables before running.");
}

const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/tasks`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Idempotency-Key": taskKey,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ url: targetUrl, format: "markdown" })
});
const data = await response.json();
if (!response.ok) {
  throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);

POST /scraping/tasks

import os
import requests

api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
task_key = os.environ["SCRAPINGBUDDY_TASK_KEY"]
target_url = os.environ["SCRAPINGBUDDY_TARGET_URL"]
response = requests.post(
    "https://www.scrapingbuddy.com/api/v1/scraping/tasks",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Idempotency-Key": task_key,
        "Content-Type": "application/json"
    },
    json={"url": target_url, "format": "markdown"},
    timeout=30,
)
data = response.json()
if not response.ok:
    raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)

Base URL https://www.scrapingbuddy.com/api/v1

Set SCRAPINGBUDDY_API_KEY, SCRAPINGBUDDY_TASK_KEY and SCRAPINGBUDDY_TARGET_URL in your server environment. JavaScript uses Node.js 20+; Python requires requests. These examples call the live, billed API.

request.json / replace the URL

{
  "url": "https://example.com/about",
  "format": "markdown"
}

02 / Response

Understand the response

Success returns HTTP 202 while work is active, otherwise 200 with JSON. A successful API response is not proof of successful page collection.

taskobject: The task object described below. Inspect task.status before reading content.
pollAfterSecondsinteger: 6. Wait at least this long before checking progress.

Task object

idUUID v4: The task ID is the Idempotency-Key you supplied when creating this task. Save it for result retrieval and support.
statusstring: queued, submitting, processing, researching, analyzing, ready, failed, review, or expired. An HTTP 200 alone does not mean the content is ready.
urlstring: The normalized source URL. URL fragments are removed.
formathtml | markdown: The requested content format. Defaults to html.
httpStatusinteger | null: The source website HTTP status, when available. This is separate from the status of your API request.
warningstring | null: A warning can accompany ready content when the wait condition was not met. Check whether the returned page is complete.
contentstring (conditional): Present only for a ready, unexpired task in create/get responses. Always omitted from task history. Save it before contentExpiresAt; do not execute returned HTML.
analysisobject (conditional): Present only when requested. status is pending, processing, ready, failed, skipped, or expired. A ready analysis contains result (summary, findings, limitations, sourceUrl), model, usage (inputTokens/outputTokens), and inputTruncated. With analysis.choices, it also contains decision: status, label, confidence, probabilities and insufficientEvidenceProbability. A failed analysis contains error.code and error.message; collected content is still returned. History exposes only analysis.status.
researchobject (conditional): Deep analysis only: maxPages, pagesCollected, pagesAttempted, stopReason and sources. stopReason is null, page_limit, links_exhausted, time_limit, unconfirmed_page, budget_exhausted or billing_review. Sources include URL, title, status, errorCode and warning; history returns sources: []. When available, webSearch reports status, sourcesFound and warning. content contains only the starting page, not all source pages.
errorobject | null: A task failure or review has code and message fields. This differs from the top-level error string returned for an unsuccessful API request.
billingobject: New paid tasks use mode: "usage". customerChargeUsd is the settled USD charge; heldUsd is a temporary wallet hold, not an additional charge. Provider costs are multiplied by 1.5. Earlier pilot tasks remain free.
createdAtISO timestamp: When the task was created.
completedAtISO timestamp | null: When processing finished, if available.
contentExpiresAtISO timestamp | null: Content and analysis expire 24 hours after page collection. After expiry, both results are omitted and status becomes expired.

Example response / illustrative values

{
  "task": {
    "id": "f824e030-0e09-4d70-a372-fdfd22e84721",
    "status": "queued",
    "url": "https://public-domain.example/about",
    "format": "markdown",
    "httpStatus": null,
    "warning": null,
    "error": null,
    "billing": {
      "mode": "usage",
      "customerChargeUsd": "0.000000000",
      "heldUsd": "0.000000000"
    },
    "createdAt": "2026-10-08T10:00:00.000Z",
    "completedAt": null,
    "contentExpiresAt": null
  },
  "pollAfterSeconds": 6
}

URLs, IDs, costs, content and timestamps are illustrative. Use your own public hostname.

Errors & next steps

Request errors return an HTTP error status and a top-level error string. Processing failures appear in task.error even when fetching the result succeeds.

New tasks and batches require $1 available balance; insufficient starting credit returns 402. All customer keys share 60 requests per minute per account. For 429, honor Retry-After; active-task and daily limits may require a longer wait.

Errors & limits Safe retries Task lifecycle