Prepare a URL file that will not break your batch

01Give each row an identity before making requests

A useful input file contains a stable row identifier and a URL. Optional context can live in separate columns. The identifier lets you join results back to the original list even if tasks finish out of order, two rows share a domain or one URL redirects. Never use completion order as the relationship between an input and its answer.

Keep your question and answer labels consistent for a classification batch. If some rows use a different definition of competitor, separate those into another job or record the definition per row. Otherwise a single result column appears comparable when the underlying questions were different.

record_id,url,segment
company-001,https://example.com/about,software
company-002,https://example.org/company,services

02Use a CSV parser, not string splitting

Quoted cells can contain commas, quotes and line breaks. TSV uses a different delimiter but still needs consistent parsing. A real parser should handle the file's delimiter, UTF-8 encoding, an optional byte-order mark and malformed rows explicitly. Show parsing errors before paid work begins, rather than silently skipping half a file.

For spreadsheet exports, check the header names and preview several rows. A visible hyperlink label is not necessarily the actual URL stored in a cell. Remove unrelated personal information and secrets from the file before upload; a URL batch rarely needs those columns.

03Validate and deduplicate conservatively

Accept only the URL schemes your service supports. Flag missing hosts, credentials embedded in URLs, private-network addresses and unsupported targets. Server-side validation is still required even if the browser already checks the file. Import validation must not become a way to make requests to internal infrastructure.

Trim accidental surrounding whitespace and identify exact duplicate URLs. Do not automatically remove every query parameter: a parameter may select a product, language, location or page number. Keep the original value alongside any normalised value so a user can understand how the request was constructed.

• Report valid, invalid and duplicate row counts separately.
• Retain the original row ID on every error and result.
• Ask for an explicit policy when duplicate rows carry different context.

04A batch needs progress and partial results

Queue accepted rows with bounded concurrency. Show pending, running, completed and failed counts without claiming that every accepted row will succeed. Preserve completed results when one source fails. Retrying only failed or unresolved rows is easier to audit and avoids paying again for work you already have.

Apply a job budget as well as a row limit. A hundred URLs can have very different costs if some require multiple pages or AI analysis. Estimate the work before starting and stop safely at the budget boundary, keeping the results already completed.

05Export results for the next system

Include row ID, submitted URL, final URL, status and any error code. Keep data fields separate from operational errors. For spreadsheet delivery, protect against formula execution in untrusted text cells; do not let a page title beginning with a formula character become executable spreadsheet content. Keep an unmodified machine-readable export when exact source text is required.