Over the past few months, I've been working on a high-scale scraping pipeline to aggregate listings directly from company job boards and applicant tracking systems. Mapping over 100,000 distinct companies to their career pages turned out to be a massive engineering headache, but it's finally stable.

The result is a unified database of more than 2 million active job postings, which I'm opening up to everyone for free. I am running daily delta refreshes to keep it current.

Dataset Overview

• Scale: 2M+ active job listings across 100,000+ unique companies.
• Format: Parquet. (To keep storage costs to minimum)
• Core Fields: job_title, company_name, company_website, job_description, location, post_date, and the original tracking URL. For more detailed info check here.
• Update Cadence: Refreshed daily straight from the source.
• View the stats here. (Currently it contains only minimal stats, but I plan on improving it based on the comments)

Why I Built This

Finding a clean, scaled, and up-to-date job dataset is surprisingly difficult. Most available options are either heavily gatekept by expensive subscription APIs or restricted to a single job board like LinkedIn. By scraping the actual employer sites directly, this collection sidesteps the noise and captures a much cleaner cross-section of the live market.

How to Access It

I set up a dedicated project space where you can grab the data directly: Open Job data

Let me know what kind of analysis or projects you end up running with it. If you have questions about the engineering architecture behind handling this scale, or ideas for specific fields you'd like to see enriched next, let's discuss in the comments.