Evaluate a data provider with a representative pilot

01Give every candidate the same brief

Define source scope, required fields, context, refresh needs and delivery format before comparing providers. If one candidate returns a visible base price and another returns a tax-inclusive delivered price, their outputs are not directly comparable even if both columns are called price.

Provide a small example of the desired schema and identify required versus optional fields. Specify how missing values and failed collection should be represented. This makes it possible to evaluate usefulness rather than rewarding whichever demo contains the most rows.

02Select difficult and ordinary examples

Include common pages as well as variations: multiple layouts, different regions, missing fields, out-of-stock items and legitimate empty results. A sample made entirely of edge cases is not representative either. Choose a mixture that reflects the expected workload.

Reserve a holdout set that is not used to tune the implementation. After the initial collection works, evaluate those pages with the same acceptance rules. This helps reveal whether the solution depends on manually prepared examples rather than a repeatable process.

03Measure the dimensions separately

Do not compress the whole pilot into one success percentage. A provider can have excellent transport success and poor field accuracy, or good accuracy on a small subset but weak coverage. Ask for the denominator behind each reported metric.

04Test the handover, not only collection

Import the pilot output into the system that will actually use it. Check data types, stable IDs, encoding, nested fields and duplicate handling. A file that looks good in a browser can still fail a warehouse import or silently change numeric precision.

Ask how schema changes are versioned and communicated. Identify who investigates a missing field and what evidence will be available. Review data handling, access permissions and retention as part of the evaluation rather than waiting until the feed is operational.

05Make a decision with documented gaps

Record what passed, what failed and what was not tested. Agree whether unresolved gaps are acceptable, need remediation or make the project unsuitable. A clear limitation is not necessarily a reason to reject a provider; an undisclosed limitation is much harder to manage.

Keep the pilot fixtures and acceptance checks after selection. They become a regression set for future changes and a common reference when the source changes. The best outcome is not a perfect demonstration but a maintainable contract that both teams understand.