AI analysis & company context

Keep content collection separate from AI

Omit analysis for HTML or Markdown only. Add an analysis object to ask a question about the collected page. Analysis runs after successful page collection, using OpenAI on the server. Your customer API key stays the same; you do not supply an OpenAI key.

Only collected sources and the instruction and company context you provide are sent to OpenAI. Do not put secrets in either field. Final analysis has no tools. Deep analysis can optionally use OpenAI web search to discover additional public sources before our bounded collection workflow retrieves them.

Describe the information you need

analysis.instruction is required when analysis is present and accepts 1-2,000 characters. Be specific about fields, summary length or classification criteria. analysis.companyContext is optional and accepts 1-3,000 characters.

The complete JSON body, including URL and all analysis fields, must fit within 8,192 bytes. Save it with your UUID v4 task key before submitting. For cURL, use this JSON as request.json; JavaScript and Python use the target URL environment variable.

{
  "url": "https://YOUR_PUBLIC_HOST/about",
  "format": "markdown",
  "analysis": {
    "instruction": "Extract company name, location and public email.",
    "companyContext": "We provide sustainable packaging research."
  }
}

# Save request.json with your public URL and format first.
# Reuse this file and the saved key for an uncertain retry.
curl --fail-with-body -X POST "https://www.scrapingbuddy.com/api/v1/scraping/tasks" \
  -H "Authorization: Bearer $SCRAPINGBUDDY_API_KEY" \
  -H "Idempotency-Key: $SCRAPINGBUDDY_TASK_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

const apiKey = process.env.SCRAPINGBUDDY_API_KEY;
const taskKey = process.env.SCRAPINGBUDDY_TASK_KEY;
const targetUrl = process.env.SCRAPINGBUDDY_TARGET_URL;
if (!apiKey || !taskKey || !targetUrl) {
  throw new Error("Set the required environment variables before running.");
}

const response = await fetch(`https://www.scrapingbuddy.com/api/v1/scraping/tasks`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Idempotency-Key": taskKey,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ url: targetUrl, format: "markdown", analysis: { instruction: "Extract company name, location and public email.", companyContext: "We provide sustainable packaging research." } })
});
const data = await response.json();
if (!response.ok) {
  throw new Error(`${response.status}: ${data.error}`);
}
console.log(data);

import os
import requests

api_key = os.environ["SCRAPINGBUDDY_API_KEY"]
task_key = os.environ["SCRAPINGBUDDY_TASK_KEY"]
target_url = os.environ["SCRAPINGBUDDY_TARGET_URL"]
response = requests.post(
    "https://www.scrapingbuddy.com/api/v1/scraping/tasks",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Idempotency-Key": task_key,
        "Content-Type": "application/json"
    },
    json={"url": target_url, "format": "markdown", "analysis": {"instruction": "Extract company name, location and public email.", "companyContext": "We provide sustainable packaging research."}},
    timeout=30,
)
data = response.json()
if not response.ok:
    raise RuntimeError(f"{response.status_code}: {data.get('error')}")
print(data)

Examples use Bash, Node.js 20+, or Python with requests. View request and response fields.

Add company context only when it helps

For relevance analysis, describe your services and target customers in companyContext, then ask the instruction to assess fit. For example: "Assess whether the company could benefit from our packaging research. Separate observed facts from inferences and list unknowns."

Context belongs to this request. It is not a saved company profile and is not automatically included in later requests. Website-based profile generation is not part of this API.

Handle partial research and budget stops

Deep research can stop with page_limit, links_exhausted, time_limit, unconfirmed_page, budget_exhausted or billing_review. A null stopReason means no stopping reason has been recorded yet. Inspect research.sources, including failed sources and warnings, before treating findings as complete.

research.webSearch, when available, reports status, sourcesFound and warning. The source count describes search discovery, not how many pages were successfully collected. Search failure normally falls back to website links, but INSUFFICIENT_BALANCE or BILLING_REVIEW stops further paid work instead.

A ready task can retain its starting-page content while task.error reports a budget or billing issue and task.analysis.status is failed. Check all three fields; do not interpret ready as a complete research report. Adding funds does not automatically resume stopped work.

Research beyond the starting page

Set analysis.depth to deep with format: markdown. maxPages is an integer from 2 to 30, defaults to 5, and includes the starting page. Company context is still optional. The starting page is collected first; public links from navigation and page content are ranked using your instruction, URL and link text. New links from collected pages can be followed up to three link levels. This is bounded link discovery, not autonomous browsing or exhaustive crawling.

For example, start from a careers page and ask for named customers. The collector can follow customer stories or case studies on the same origin before one combined analysis. It may also visit the homepage. Set analysis.webSearch: true to also discover independent sources through OpenAI web search, then collect relevant cited pages within the same page budget. The dashboard enables this by default; the API defaults to website-only discovery when omitted. It does not log in, bypass CAPTCHA, submit forms, or visit query-based pages, downloads and account pages. It may miss relevant links or information outside its limits.

Poll through researching and analyzing. research.sources shows selected URLs and collection outcomes; stopReason explains page_limit, links_exhausted, time_limit or unconfirmed_page. Each supported deep finding includes sourceUrl and a quote verified against that specific page. A logo, partner listing or integration is not automatically evidence of a current customer relationship.

waitFor applies only to the starting page. Linked pages use default readiness. Each extra page consumes scraping usage. Optional web search adds model and search-tool usage, with at most three search-tool calls. Private company context is not included in search; your question and a public starting-page excerpt are sent to the search model. A failed or interrupted search is not automatically retried; research continues using website links and reports a warning. Deep analysis sends up to the first 12,000 characters per collected source, at most 120,000 characters total, shared across all collected sources, with a 3,000-token output cap. Individual linked pages time out after five minutes; research stops after a 60-minute task window. Failed pages produce partial results, not invented details.

content remains the starting-page content; the analysis combines the successful sources. An uncertain linked-page submission stops further discovery and is not resubmitted automatically. Source content and analysis expire together 24 hours after the starting page is collected.

{
  "url": "https://YOUR_PUBLIC_HOST/career",
  "format": "markdown",
  "analysis": {
    "instruction": "Find named customers and public case studies. Cite evidence and distinguish customers from partners.",
    "depth": "deep",
    "maxPages": 30,
    "webSearch": true
  }
}

Read the analysis result

Continue polling while task.status is analyzing. When it is ready, inspect task.analysis.status. A ready analysis includes result.summary, result.findings, result.limitations and result.sourceUrl.

Each finding has field, value and evidence. evidence is an exact quote from the supplied page; values without a matching quote are left null. A matching quote is not a guarantee that the interpretation is correct. Review summaries and business-fit inferences before using them.

The response also includes model, usage.inputTokens, usage.outputTokens and inputTruncated. Usage describes tokens, not a customer invoice. History only includes analysis.status; fetch the individual task for the result.

{
  "status": "ready",
  "result": {
    "summary": "Northstar Packaging is based in Bristol.",
    "findings": [
      {
        "field": "location",
        "value": "Bristol",
        "evidence": "Based in Bristol, United Kingdom."
      }
    ],
    "limitations": [
      "Buying intent is not stated on the page."
    ],
    "sourceUrl": "https://northstar.example/about"
  },
  "model": "gpt-4.1-mini",
  "inputTruncated": false,
  "usage": {
    "inputTokens": 420,
    "outputTokens": 130
  }
}

Input, output and retention limits

Single-page analysis reads at most the first 40,000 characters of collected content and has a maximum output of 1,800 tokens. Deep analysis instead uses up to 12,000 characters per page and 3,000 output tokens. Truncation is reported with inputTruncated and a limitation. Prefer Markdown when raw HTML contains substantial markup or scripts.

Collected content and its analysis share contentExpiresAt, 24 hours after starting-page collection. Save both before expiry. New tasks debit confirmed provider usage at 1.5 times cost. Earlier pilot tasks remain free, and billing records survive content expiry.

Keep the content when analysis fails

If AI analysis fails, task.status becomes ready so you can retrieve the collected page. task.analysis.status is failed, with an error.code and error.message. A failed scrape skips analysis entirely.

ANALYSIS_UNAVAILABLE means analysis is disabled or not configured. ANALYSIS_RATE_LIMITED or ANALYSIS_PROVIDER_ERROR means the model service could not complete the request. ANALYSIS_INCOMPLETE means the output did not finish. ANALYSIS_REFUSED means the requested analysis was declined.

ANALYSIS_UNCONFIRMED means a timeout, interruption or unverified response. No automatic model retry occurs, because the original call may already have incurred cost. Replaying the same task returns its saved outcome. Contact support with the task ID before deliberately submitting another task.

A new analysis request returns HTTP 503 before collecting a page if analysis is disabled. Remove analysis for a content-only task, using a new task key for the changed request.