FrontlatchBot

The crawler behind the business index.

What it sends, how it reads robots.txt, how fast it asks, and how to keep your site out.

Last updated 10 October 2026.

01User agents

The exact strings we send.

The crawler that builds the business index:

FrontlatchBot/0.1 (builds the business index, read-only; +https://frontlatch.com/bot)

The jobs that seed the index from public datasets:

FrontlatchBot/0.1 (+https://frontlatch.com/bot; index seed from OpenStreetMap)
FrontlatchBot/0.1 (+https://frontlatch.com/bot; Common Crawl host graph)
FrontlatchBot/0.1 (+https://frontlatch.com/bot; AU index ingest)

The scanner, when someone asks for a scan or AI reads a page through the gateway, and the browser that maps a site’s forms (it never submits them):

FrontlatchScanner/0.1 (read-only agent-readiness scan; +contact: <address>)
<browser user agent> FrontlatchMapper/0.1 (renders pages, fills forms to trigger client-side validation, NEVER submits; +contact: <address>)

The monitors that check frontlatch.com itself. They only ever load our own site, and the tracker counts them as neither people nor agents:

<browser user agent> FrontlatchMonitor/1.0 (+https://frontlatch.com/bot)

02robots.txt

We read robots.txt before every site.

Each agent looks for a group naming it (FrontlatchBot, FrontlatchScanner or FrontlatchMapper) and otherwise follows the * group. A group named Frontlatch applies to all three. The longest matching rule wins, and an Allow beats a Disallow of the same length. If robots.txt disallows a page, the crawler does not fetch it. Crawl-delay is honoured.

03Crawl rate

How fast we ask any one host.

  • FrontlatchBot: at most one request per second to a host, or slower if your Crawl-delay says so, and at most 8 pages of a business’s site per visit.
  • FrontlatchScanner: at least half a second between requests to a host, or your Crawl-delay if longer, and at most 15 pages per scan.
  • FrontlatchMapper: at least 2.5 seconds between page loads on a host.
  • The crawler, scanner and mapper are read-only: they never sign in, buy or submit a form. The one exception is a sign-up the site's owner switched on and the user confirmed in their own chat; it never uses a password or card and stops at a CAPTCHA.

04Free tier & bulk bot toll

Free for businesses and everyday AI users.

  • For Local Businesses & Website Scanning: 100% Free to scan your website, turn on your AI Hot-Patch, and receive leads. You only pay the 8% fee when you accept and settle a real paid booking.
  • For Everyday AI Users (Claude, ChatGPT, Cursor): 100% Free to search local businesses and check calendars (up to 50 calls/day per client); pay for bookings with normal Apple Pay / Credit Card (Stripe), a $0.00 Free Quote Request fallback, or optional Mastercard Agent Pay.
  • For High-Volume Scraper Bots & B2B Aggregators: Metered via the $0.05 x402 AI Tollbooth above the free daily tier so local business websites are protected from free bulk scraping.

05Opting out

How to keep a site out.

To stop all three agents, add this to your site’s robots.txt:

User-agent: Frontlatch
Disallow: /

To remove a listing that is already in the index, use the remove link on the business’s own page in the business index: we email a code to an address at that domain, and once it is entered the listing is hidden and never added again. Every automated Escrow-to-Claim notification also includes a 1-click domain opt-out link (recorded in domain_opt_outs) that immediately blocks future automated dispatch. Or write to [email protected] with the domain. See Documentation for how com.frontlatch/gateway and Tier 1 read-only widget inspection work.