Case study · our own build

ATS Jobs Search is our keyword search over the public job boards that companies run on applicant tracking systems (ATS) such as Greenhouse, Lever and Ashby. A buyer sends a few keywords and gets the matching open jobs, each read live from the employer's own board during the run. This page shows how it was built, from our own records: where the boards come from, the limits it keeps, what pushed back, how we tested it and what it costs to run.

Describe your data task

Measured on 19 November 2026

The index after one refresh.

23,287
company job boards answered, on six applicant tracking systems
461,985
open jobs in the index
82
minutes for the whole pass, on one machine
9
boards failed after retries, all of them on Recruitee

From the index refresh of 19 November 2026. The 23,287 boards include 3,681 on Workable, read before Workable rate-limited us and stopped the rest of its lane (see below); that stop is not counted as failures. Our own product, not client work.

The brief

Search every board, read each job live.

Each applicant tracking system hosts the job boards of thousands of companies, and most publish a public feed for every board. Reading one company's feed is easy. The work is in the rest: no system publishes a list of its boards, a board address that worked last year may be gone, each host sets its own limits, and a search that read every board on every request would take over an hour.

The buyers are job boards, recruiting and sales tools, and AI agents. They want one call: keywords in, current jobs out, and nothing returned that has already closed. The spec, written before any code, set the rules the build had to keep, and one was tightened during the build:

  • Read only public job feeds, and follow each host's robots.txt. Tightened during the build to feeds that the vendor documents as public (see BambooHR below).
  • No proxies, no logins, no browser.
  • When a host pushes back, slow down or stop. Never route around a block.
  • Every coverage figure in the listing is the index's own count, with its date.

How it works

Find, check, index, then read live.

  1. A

    Find boards, then check each one

    Common Crawl publishes the addresses of the pages it crawls. A script reads its downloadable URL index files (not its index server, whose robots.txt disallows automated paths) for each system's board address patterns, across 30 of its crawls. A candidate becomes a board only when its system's public feed answers HTTP 200 with a job list; 404 or 410 means it is not a board. On 18 November 2026, 11,123 of 30,928 candidate addresses on five systems (36%) were not boards.

  2. B

    Keep the index lean

    One short row per open job: board, job id, title, location, remote flag and posting date. Descriptions stay out, so the whole index is 13.4 MB compressed, in six files (shards) of at most 8 MB each.

  3. C

    Read live at search time

    A search filters the index, then reads only the boards that hold matches, straight from each system's feed, eight boards at a time. This adds matching jobs posted since the refresh and drops jobs that have closed.

Per-host limits

One refresh, host by host.

At most 4 requests at a time to any system. Where the host asks for it or pushed back, 1 request at a time and at most 1 a second. Up to 4 retries with growing waits; a system is left alone for the rest of the run when 20 or more of its last 50 answers are 403 (forbidden) or 429 (too many requests).

Refresh of 19 November 2026 on one machine. The systems run side by side; minutes run from the start of the pass to the end of each system. Boards with a working feed but no public board page are left out: Lever 126, Ashby 21, Personio 15.
SystemPaceCheckedBoardsNot boardsFailedMinutes
Greenhouse4 at a time6,5006,494608
Ashby4 at a time3,7133,689305
Recruitee4 at a time3,1883,1718913
Personio1 at a time, at most 1 a second4,1094,0913070
Lever1 at a time, at most 1 a second (its robots.txt)2,2952,1618082
Workable1 at a time, at most 1 a second3,834 of 10,7853,681153stopped at a 42964

Greenhouse, Ashby and Recruitee finish within 13 minutes. Lever and Personio, read one request at a time, set the pace; Lever, the slowest, took the whole 82 minutes. It peaked at 1.1 GB of memory and ran on our own machine, so it used no cloud compute; only the finished 13.4 MB index is uploaded.

What went wrong

What pushed back, and what we did.

Each of these is in the build's own records, with the run that found it.

Workable rate-limited us before the first board

On 18 November 2026 the refresh's first request to Workable was answered HTTP 429 (too many requests) with Retry-After 84,362 seconds: the host asked us to wait about 23 hours. The lane stops at the first such answer, so none of the 10,785 Workable candidates was read and the listing does not name Workable. Reading it from another address would get past the limit; we count that as evasion and do not do it. On 19 November Workable answered 3,834 requests at one a second, then sent a 429 with Retry-After 86,400 seconds. The 6,951 boards still unread wait for a later refresh.

One gap, found while writing this page. An earlier attempt at the 19 November refresh, which did not finish, began about three and a half hours before that 23-hour wait had run out; Workable answered 3,840 requests before its lane stopped (the log does not say why). The pass reported here began 81 minutes after that stop. The refresh honours Retry-After within a run but does not carry it from one run to the next, so neither run knew of the earlier wait. The fix is to keep each host's wait between runs.

Fixed on 20 November 2026. Both 19 November runs were cut off after about 3,840 requests at one a second, so we now treat Workable's limit as a daily request count, not only a rate. The refresh now keeps each host's Retry-After between runs and holds Workable to 30 requests a minute and at most 1,800 requests in any 20 hours, so its boards are read on a six-day cycle, never-read boards first. The 20 November refresh read 1,800 Workable boards, and none of its eight lanes received an HTTP 429.

Personio pushed back at four at a time

On 18 November 2026, after 515 answers, 20 of the lane's last 50 were 403 or 429, and the lane stopped itself as designed. The rerun the same day, one request at a time and at most one a second, checked the remaining 6,682 boards in 2.8 hours with 0 failures. Personio has run at that pace since.

We removed a whole system for its terms

The first refresh, on 18 November 2026, included BambooHR: 10,187 boards holding 51,717 jobs. Its developer terms bar crawling its interfaces or content without written consent, and no documented public feed stands behind its careers pages. It was taken out of the index, the refresh and the search options. SmartRecruiters was never added: its feed's robots.txt allows only LinkedIn's crawler.

A few big boards crowded out everyone else

The first test search returned 500 jobs from only 21 companies, because a few large boards held hundreds of matching sales roles. We added a cap per company. With a cap of five, the test search returned 500 jobs from 135 companies.

A slow board was overtaken and dropped

Results were kept in the order the boards finished. A large Lever board, read at one request a second, could finish last and be cut off when the 500th job arrived: one test run had 0 Lever jobs although Lever held newer matches. Jobs now come out in the order of each board's newest match, and the next run had 17 Lever jobs.

Descriptions in the wrong language, and a dead link domain

Personio's feed carries descriptions in the account's main language only, so a job published only in English came back empty: 0 of 6 on one test board. The reader now asks for English when a description is missing: 6 of 6. A Recruitee job link pointed at a company's own careers domain, which no longer resolved; links now use the recruitee.com address, which still answered.

How we tested it

Counted against the source.

Each check has a pass line written before the run.

Counts equal to each system's own feed. On 7 November 2026 we ran Career Site Job Feed, our Actor that reads the boards a user lists, on 33 real boards: Greenhouse 10, Ashby 9, Lever 8, Workable 6. It returned 3,204 jobs in 35 seconds. A separate short script then counted each board's jobs straight from its system's public feed: all 33 counts were equal. The same input held 3 duplicate entries, which were skipped, and 1 address that does not exist, which was reported as failed.

A full search on Apify. Keywords "data engineer", "product designer" and "account executive", at most 500 jobs and a cap of five per company, on the index of 18 November 2026:

Done-check run on Apify, 1 GB of memory, 18 November 2026.
Matches in the index12,007 on 3,419 boards
Boards read live136, 0 failed
Jobs returned500 from 135 companies
With title, company, link and description500 of 500
Closed since the index was built, not returned4
New since the index was built, returned23
Duplicates removed72

Links that open. 35 job links, one per company, across the five systems the listing names (it does not name Workable): all answered HTTP 200.

Tests with no network. Every feed reader, the matching rules, de-duplication, the failure paths and the charge limit are tested on recorded answers, so a change cannot pass by luck of what a host returned that day.

Running cost

What it costs to run.

Measured, not estimated.

Index refresh, 19 November 202682 minutes on one machine, 1.1 GB peak memory, no cloud compute
Index uploaded13.4 MB compressed, six shards
One search of 500 jobs, on Apify108 seconds, 0.030 compute units (Apify's billing unit), USD 0.0039 of platform cost
Price to the buyerUSD 0.002 per job returned, on the Apify Store

The same method, on your task

When we build data collection or AI automation for a business, it runs the same way: the rules and the pass lines are written before the code, every run reports what it read, what failed and why, the cost is measured per item, and a host that says no gets a no from us too. Describe the task at [email protected], or read about our AI services and custom data work.

ATS Jobs Search itself is on the Apify Store; job feeds compares it with our other two job tools.

What this page does not show

  • Board addresses come from Common Crawl, so the index covers the boards found there, not every company on these systems.
  • The figures come from the refreshes of 18 and 19 November 2026. Two refreshes are not a trend.
  • Workable's figures cover only the boards it answered before its rate limit.
  • This is our own product. No client was involved, and no employer is named here.
  • MakeYeti is not affiliated with or endorsed by BambooHR, SmartRecruiters, LinkedIn or Common Crawl.