Your first scrape

1. Sign in, fund your wallet and create a key

During the current testing stage, sign in and use a public URL without separate account or hostname approval. Start with at least $1 available wallet balance. A provider-completed page costs $0.00045, with AI and web search billed separately when requested; temporary holds reduce available credit.

For API integration, create a customer API key in Profile > API access. Keep it on your server. These examples create live, billed tasks, not sandbox requests. The initial example omits analysis and does not call AI.

export SCRAPINGBUDDY_API_KEY="YOUR_API_KEY"
export SCRAPINGBUDDY_TARGET_URL="https://YOUR_PUBLIC_HOST/about"
# Generate once; save this key and request before sending.
export SCRAPINGBUDDY_TASK_KEY="$(node -p 'crypto.randomUUID()')"

2. Submit one page

Replace the placeholder below with your public URL and save the JSON as request.json for the cURL example. JavaScript and Python read SCRAPINGBUDDY_TARGET_URL instead.

Send the request once and save task.id from the response. A 202 means accepted for processing, not finished. The task ID is the same UUID v4 you supplied as Idempotency-Key.

{
  "url": "https://YOUR_PUBLIC_HOST/about",
  "format": "markdown"
}

# Save request.json with your public URL and format first.
# Reuse this file and the saved key for an uncertain retry.
curl --fail-with-body -X POST "https://www.scrapingbuddy.com/api/v1/scraping/tasks" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY" \
  -H "Idempotency-Key: $SCRAPINGBUDDY_TASK_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const taskKey = process.env.SCRAPINGBUDDY_TASK_KEY;
const targetUrl = process.env.SCRAPINGBUDDY_TARGET_URL;
if (!apiKey || !taskKey || !targetUrl) {
  throw new Error("Set the required environment variables before running.");
}

const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/tasks`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Idempotency-Key": taskKey,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ url: targetUrl, format: "markdown" })
});
const data = await response.json();
if (!response.ok) {
  throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);

import os
import requests

api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
task_key = os.environ["SCRAPINGBUDDY_TASK_KEY"]
target_url = os.environ["SCRAPINGBUDDY_TARGET_URL"]
response = requests.post(
    "https://www.scrapingbuddy.com/api/v1/scraping/tasks",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Idempotency-Key": task_key,
        "Content-Type": "application/json"
    },
    json={"url": target_url, "format": "markdown"},
    timeout=30,
)
data = response.json()
if not response.ok:
    raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)

Examples use Bash, Node.js 20+, or Python with requests. View request and response fields.

3. Wait, then check your task

Wait at least 6 seconds, then fetch the saved task ID. Continue checking queued, submitting, processing, researching or analyzing tasks at that interval. GET reads progress; do not POST a new task to check progress.

The example checks once. In your application, use a bounded polling loop and preserve the task ID if you stop waiting. For 429, honor Retry-After. Stop on ready, failed, review or expired.

export SCRAPINGBUDDY_TASK_ID="$SCRAPINGBUDDY_TASK_KEY"
sleep 6
curl --fail-with-body \
  "https://www.scrapingbuddy.com/api/v1/scraping/tasks/$SCRAPINGBUDDY_TASK_ID" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY"

4. Read and save the result

When task.status is ready, task.content contains your HTML or Markdown. Inspect task.warning: a wait condition may not have been satisfied even when content is returned.

Save content to your own storage before contentExpiresAt, 24 hours after page collection. Treat source content as untrusted text. Sanitize HTML before displaying it; do not run scripts from scraped pages.

If task.status is failed, inspect task.error.code and task.error.message. For review, contact support with task.id and do not create a replacement request.

A bounded polling loop

This Node.js example only reads a saved task. It never creates a replacement or charges for another scrape. Stop locally after ten minutes and keep the task ID; stopping polling does not cancel queued or in-flight server work. The thirty-second fetch timeout limits one network exchange, not page rendering.

ready means page content is available, not that optional AI succeeded. Inspect task.error and task.analysis.status too: insufficient funds or a failed model call can leave usable content with an incomplete analysis.

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const taskId = process.env.SCRAPINGBUDDY_TASK_ID;
if (!apiKey || !taskId) throw new Error("Set your API key and saved task ID.");
const active = new Set(["queued", "submitting", "processing", "researching", "analyzing"]);
const deadline = Date.now() + 10 * 60_000;
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
let result;
while (Date.now() < deadline) {
  const response = await fetch(
    "https://www.scrapingbuddy.com/api/v1/scraping/tasks/" + encodeURIComponent(taskId),
    { headers: { Authorization: "Bearer " + apiKey }, signal: AbortSignal.timeout(30_000) }
  );
  if (response.status === 429 || response.status >= 500) {
    const seconds = Math.max(6, Number(response.headers.get("Retry-After")) || 60);
    await sleep(Math.min(seconds * 1000, Math.max(0, deadline - Date.now())));
    continue;
  }
  const data = await response.json();
  if (!response.ok) throw new Error(response.status + ": " + data.error);
  if (!active.has(data.task.status)) { result = data.task; break; }
  await sleep(Math.min(6000, Math.max(0, deadline - Date.now())));
}
if (!result) throw new Error("Stopped waiting. Resume later with task ID " + taskId);
if (result.status !== "ready") {
  throw new Error(JSON.stringify({ id: taskId, status: result.status, error: result.error }));
}
console.log(result);