Formats & page readiness

HTML or Markdown

format defaults to html. Choose html when your own parser needs markup, or markdown when you need readable text for research, indexing or another processing step. Deep analysis requires markdown.

Submit one starting URL per task. Normal analysis uses that page only; analysis.depth: deep can collect a bounded set of linked website pages and, when webSearch is enabled, relevant public sources from other websites. For bulk classification, use the separate /scraping/classifications endpoint. Single-task requests do not accept URL arrays, cookies, credentials or custom browser controls.

JavaScript content and timeout boundaries

Do not assume HTML is a serialized copy of the final rendered DOM: it can contain the original page source. Markdown is preferable for text research, but neither format guarantees that every dynamic section was loaded. Pick an appropriate text or element wait condition and verify the returned content.

The provider wait condition is capped at 30 seconds. Reaching that cap can return content with a warning instead of a task failure. ScrapingBuddy accepts waitFor, not a timeout or renderJavaScript field. An SDK or HTTP-client timeout only controls how long your client waits for a response; it does not cancel a server task. Use the saved task ID to check progress after an interrupted request.

Choose one wait condition

waitFor is optional. Choose one of the following shapes, not a combination. text waits for text; element waits for a selector, with optional checkVisible; state accepts load or domcontentloaded.

text and element must be non-empty strings no longer than 300 characters. checkVisible defaults to false. There is no custom timeout field.

[
  { "text": "About us" },
  { "element": "main", "checkVisible": true },
  { "state": "domcontentloaded" }
]

Request a visible element

Replace the URL with a public page. Choose a selector that exists on that page.

{
  "url": "https://YOUR_PUBLIC_HOST/about",
  "format": "markdown",
  "waitFor": { "element": "main", "checkVisible": true }
}

Check warnings and source responses

A ready task can contain a warning if the wait condition was not met. Check the content for completeness before using it. HTTP 200 from the API is not proof that the source page was collected successfully; inspect task.status and task.httpStatus.

Login pages, access challenges and unsuccessful source responses can fail the task. This integration does not offer login or CAPTCHA bypass.