We crawl the web continuously and cannot keep all of it, refresh all of it, or spend a GPU on all of it. Something has to decide which pages are worth indexing, which links are worth following, which pages say the same thing as pages we already have, what has gone stale, and what we should never have picked up. These decisions are worth more than most ranking work. The cheapest way to improve search is to stop indexing pages nobody should ever retrieve, and to start indexing the ones we are missing. Today they run on a mix of learned signals and thresholds someone picked.
We are looking for a research engineer to own the signals behind those decisions, and the decisions themselves.
Desired Experience- You are comfortable owning a problem with no ground truth, where the first job is defining what the right answer means
- Hands-on ML experience with classifiers, rankers and calibration, plus strong data instincts at web scale
- You think in end-to-end impact. A signal only counts if a decision changes and search gets better
- You like working across teams. Crawling, indexing and retrieval all consume what you build
- You care about the problem of finding high quality knowledge and recognize how important this is for the world
Example Projects- Make parsing work on the pages where it currently does not, and prove the improvement rather than assert it
- Teach a model to judge page quality, and get everyone to agree on what quality means well enough to supervise it
- Work on credibility and misinformation as a modelling problem: what a page claims, whether it is a reliable source of it, and whether it was written for a reader or for a crawler
- Decide whether two documents are semantically the same or genuinely different, so we can deduplicate the web without collapsing pages that a user would want to see separately
