Turn job listings into a deduplicated hiring dataset
Preserve job identity, location scope, posting dates and salary disclosure while avoiding duplicate roles and unsupported hiring claims.
In this article
THE KEY IDEA
A job advertisement is evidence of a published role, not proof of a hire. Keep reposts, locations and collection dates distinct.
01Define what one job record means
A role can appear on an employer site, several job boards and multiple location pages. Decide whether your dataset counts source advertisements or deduplicated openings. Both can be useful, but mixing them inflates counts and makes trends difficult to interpret.
Keep each source's job ID and URL, then add a separate match key for a suspected shared role. Employer, title, location and description can support matching, but similar titles do not prove that two advertisements are the same opening.
02Preserve dates without manufacturing them
Store published date, update date, expiry date and observation time separately when available. A first-seen date in your dataset is not the same as the original posting date. If a source presents a relative date, preserve the collection context used to interpret it.
A repost may represent a refreshed advertisement, an additional opening or an unresolved earlier role. Do not automatically count it as a new hire or a new vacancy. Keep the underlying evidence and apply a documented deduplication window.
| Field | Important distinction |
|---|---|
| Employer | Direct employer versus recruiter or intermediary |
| Location | Office location versus applicant eligibility |
| Remote status | Remote work versus remote from any country |
| Salary | Published range versus absent information |
| Status | Explicitly closed versus no longer observed |
03Keep the source's employment context
Preserve employment type, seniority and location restrictions before normalising them. Remote roles can still have country, timezone or work-authorisation requirements. A city in the listing may describe an office rather than the only permitted work location.
Salary fields need currency, period and range semantics. Do not convert a missing salary into zero or infer a salary from the title without labelling it as a separate estimate. Keep source text available for reviewing ambiguous compensation descriptions.
04Validate discovery and extraction independently
A parser can accurately extract every page it sees while discovery misses most of a source. Track expected inputs, discovered pages and usable records separately. Include pagination, legitimate no-results pages and expired listings in your fixture set.
Review duplicate groups manually for a representative sample. Over-merging hides real openings; under-merging inflates counts. The appropriate balance depends on the research question and should be documented with the delivered data.
05Report labour-market signals with caveats
Published advertisements can support research into hiring activity and requested skills. They do not directly measure completed hires, headcount growth or the entire labour market. Keep coverage, source mix and observation windows visible alongside any trend.
Avoid collecting applicant data or unrelated personal information. A public job-posting dataset should remain focused on the role and employer information required for the stated purpose. Use a separate review for any more sensitive downstream decision.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.