Bots

browser.bot.name is the name from bots.yml, or Generic Bot when another matcher caught it. The lists are in the repository.

Every week, script/sync_bots.rb rewrites the part of bots.yml below its marker line, and bot_categories.yml, from crawler-user-agents and ai.robots.txt, and a pull request shows the answers that change. Above the marker, the list inherited from browser stays as it was.

How many of the bots of crawler-user-agents and ai.robots.txt each gem calls a bot, per category: every user agent crawler-user-agents gives as an instance, and, for each name of ai.robots.txt, which gives none, the user agent an AI agent of 2026 sends. script/bot_coverage.rb counts them into docs/bot_coverage.json and the weekly sync regenerates it, with this table.

browser_automation is never bot? on purpose (below). ai_agent misses its plain-word names (Operator, Cursor, Devin, Codex): as substrings they would match human user agents.

browser.bot.category names what a bot declares it is, with the names of Cloudflare’s verified bot categories where one fits, so an app can compare it with what Cloudflare verified. It is nil for a browser, a bot exception, and a bot no list names (caught by a keyword).

browser = Browser.new("Mozilla/5.0 (compatible; Claude-User/1.0)")
browser.bot.category        # => :ai_assistant
browser.bot.ai?             # => true: ai_crawler, ai_search, ai_assistant or ai_agent
browser.bot.user_triggered? # => true: ai_assistant or ai_agent, acting for a person right now

To treat test drivers as bots in production: browser.bot? || browser.bot.category == :browser_automation.

browser.bot.why? returns the matcher that fired. A matcher is any callable taking the lowercased user agent and a browser:

Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }

To start from nothing, Browser::Bot.matchers.clear; to get the built-in ones back, Browser::Bot.matchers += Browser::Bot.default_matchers. To restore browser 2’s detection, remove the keyword and empty user agent matchers:

Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher)
Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)

Browser::Bot.bots.merge!(new_bots_hash)
Browser::Bot.bot_exceptions += new_exceptions
Browser::Bot.search_engines.merge!(new_search_engines_hash)

The prefix tries behind each list are rebuilt when the list grows or is replaced. Browser::Bot.category_by_keyword takes new categories the same way. To add a bot to Agentry itself, add its user agent to test/ua_bots.yml and its name to bots.yml; a search engine also goes in test/ua_search_engines.yml and search_engines.yml.

A trap. Public crawler lists call “bot” anything that is not a human browser: crawlers, but also test drivers (Playwright, Puppeteer, Selenium), scripts and scanners. Adding them all to bot? would flag the system tests of the apps using it, so test drivers get a category of their own and no bot?.

corpus/user_agents.txt holds 20,328 distinct user agents. A change that is not meant to change an answer proves it there: one script writes every answer of a gem as JSON lines, another diffs two snapshots.

ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
ruby -I lib script/corpus_snapshot.rb > /tmp/agentry.jsonl
ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/agentry.jsonl

path/to/upstream is a plain clone of fnando/browser at dd88933. A change meant to change answers, like a fresher bot list, lists the differences it expects in corpus/expected_differences.jsonl (corpus_diff.rb --write), and the diff must show exactly those. No browser of uap-core turns into a bot: the sync leaves out any keyword that would flag one.

The corpus gathers, deduplicated, the user agents of:

• the test cases of ua-parser/uap-core, Apache License 2.0;
• the instances of monperrus/crawler-user-agents, MIT;
• the test fixtures of fnando/browser, MIT.

The bot list and its categories are synced from permissively licensed sources only: crawler-user-agents and ai.robots.txt, both MIT. script/bot_coverage.rb counts, per category, the bots of both lists that browser and Agentry detect, into docs/bot_coverage.json. Details in corpus/SOURCES.md.