Meta Careers Scraper
A recursive Scrapy spider that follows listing pages and exports each role as a consistent JSON object.
Inspect the sourceCarry-on evidence
- Paginated
- Traversal
- 4 fields
- Schema
- JSON
- Output
01 / Context
Job pages are made for browsing, not comparing.
A careers page works well when someone is opening one role at a time. It becomes much less useful when the goal is to compare a large set of roles or reuse the information elsewhere.
This project treats the visible job board as an input surface and asks a simple question: what would the same information look like as data?
02 / Crawl design
Follow the cards, then follow the next page.
The spider reads each listing card with one selector pattern, yields the structured result, and continues through pagination until there is no next page.
Keeping traversal and extraction in the same crawl means the output does not depend on someone manually opening every page first.
03 / Structured output
Four fields make every role comparable.
Each record keeps the job title, location, area of work, and category. The same shape is applied to every result before the crawl is exported to JSON.
The value is not a dramatic dashboard. It is a small, predictable interface between a public web page and whatever analysis might come next.
{
"title": "Software Engineer",
"location": "Menlo Park, CA",
"area": "Engineering",
"category": "Software Engineering"
}04 / Reflection
A useful tool can be narrow on purpose.
The scraper does one job: it turns a browsing flow into reusable records. Its clarity comes from resisting extra features until the extraction path is dependable.
It is also a reminder that product work can begin with noticing friction in an existing interface, then building the smallest system that removes it.
{
"title": "Software Engineer",
"location": "Menlo Park, CA",
"area": "Engineering",
"category": "Software Engineering"
}A one- or two-player platformer where the whole team has to keep climbing.
Read next →