Pending is not failed: asynchronous scraping without duplicate jobs
Separate submitting a task from checking its result, and handle ambiguous timeouts without creating duplicate work.
In this article
THE KEY IDEA
Persist the task ID, poll the documented result endpoint, and distinguish a failed status check from a failed collection task.
01Submission and collection are separate operations
An asynchronous service can accept your request before it has finished collecting anything. HTTP 202 indicates acceptance, not successful completion. The response normally gives you a way to inspect the task later. Save that identifier together with your input row, submission time and relevant configuration before starting another job.
Do not assume that every endpoint uses the same host or authentication rules. Submission and result retrieval can have different contracts. Build both from the provider's documentation rather than constructing a result URL from memory. Keep API keys out of logs and browser-visible URLs.
02Use a small, explicit state machine
The names below are an application design example, not a promise about any provider's exact status strings. Map the provider's documented states into your own model and preserve the original response for debugging after removing secrets.
| Local state | Meaning | Next action |
|---|---|---|
| Submitted | Task ID saved | Check that task |
| Pending | No terminal result yet | Poll after a bounded delay |
| Completed | A result is available | Validate and save once |
| Failed | Provider reports terminal failure | Record the error and evaluate a new attempt |
| Unknown | Submission outcome is ambiguous | Reconcile before resubmitting |
03A timeout does not prove that nothing happened
If the connection drops after submission, the provider might still have created a task. Blindly submitting again can create duplicate collection and duplicate cost. Use a supported idempotency mechanism when available, with the same key for the same logical request. Do not invent an idempotency header and assume the server honours it.
A failed polling request is different. The task can continue running even when your status check times out. Retry the status check within a limited budget; do not create a replacement task merely because one check failed. Keep polling ownership in a durable worker when a browser tab may close.
04Back off without hiding permanent errors
Space status checks and use the provider's recommended interval. For rate limiting, respect Retry-After when supplied. Bound both attempts and elapsed time. A permanent validation error, invalid credential or disallowed target needs correction, not a loop of identical requests.
When a provider returns a terminal error, retain the code and the complete sanitised response. The task ID helps support find a request, but usually does not explain why it failed. A useful incident report includes the endpoint, HTTP status, UTC time, request options and result body, with credentials removed.
05Make completion idempotent too
Two workers can observe the same completed task. Use the task ID as a unique completion key so processing the result twice does not insert duplicate records or trigger duplicate downstream actions. A successful write should be atomic with the local completion transition. This protects the workflow even when a worker restarts between receiving and saving a result.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.