Task lifecycle & retention
Know when to poll, when to save content and when to stop.
Collection, research and analysis
queued means accepted and waiting to start. submitting means the task is being handed off. processing means the starting page is not ready yet. researching means linked pages are being collected for deep analysis. analyzing means the collected sources are being analyzed. Poll the same task at intervals of at least 6 seconds.
These statuses count toward your limit of 3 active tasks. review also counts toward that limit. A starting page exceeding the 30-minute processing window can fail with RESULT_TIMEOUT; it is not automatically resubmitted.
Ready: save your content
ready means content is available in the individual task response. Check warning and httpStatus, then save content before contentExpiresAt. Content and analysis are retained for 24 hours after page collection. If analysis was requested, also inspect analysis.status; a failed analysis does not discard the page.
expired means content is no longer available. Metadata can still appear in history. Fetching or replaying the original task does not renew retention or run it again.
Failed or review: stop polling
failed includes a task.error with a code and message explaining the failure. Correct the underlying cause before deliberately creating a new task with a new key.
review means submission could not be confirmed. Do not automatically submit a replacement; contact support with the existing task ID. Reusing that ID reads or returns the existing task, without clearing review.
Find previous tasks
GET /scraping/tasks?offset=0 returns up to 10 of your tasks, newest first, and hasMore. Increase offset by 10 while hasMore is true. Content is omitted from every history item; retrieve individual tasks to read it.
curl --fail-with-body -X GET "https://www.scrapingbuddy.com/api/v1/scraping/tasks?offset=0" \
-H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY"const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
if (!apiKey) {
throw new Error("Set the required environment variables before running.");
}
const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/tasks?offset=0`, {
method: "GET",
headers: {
Authorization: `Bearer ${apiKey}`
}
});
const data = await response.json();
if (!response.ok) {
throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);import os
import requests
api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
response = requests.get(
"https://www.scrapingbuddy.com/api/v1/scraping/tasks?offset=0",
headers={
"Authorization": f"Bearer {api_key}"
},
timeout=30,
)
data = response.json()
if not response.ok:
raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)Examples use Bash, Node.js 20+, or Python with requests. View request and response fields.