ThemeliorBot
The agent Themelior sends to read websites. If you found it in your logs, this page says what it requested, why, how often, and how to stop it.
Last updated .
ThemeliorBot/1.0 (+https://themelior.com/bot)- GET only
- No JavaScript
- 8 s timeout
- 2 MB cap
When it visits.
Only for one of these reasons. It does not crawl the web on its own.
When a person scans a page
On requestSomeone entered a URL on themelior.com or called the scan API. One scan is the eight requests below, and a redirect adds one per hop. A complete scan of the same URL under an hour old is reused, so nothing is requested again; and one host is scanned at most 20 times an hour, by everyone together.
A watched site, once a day
Daily, from 07:00 UTCA signed-in user can watch up to 4 sites. Each is scanned again once a day, with the same eight requests, to tell them when the score drops.
A verified site, for its owner
Daily, and on the owner’s requestOnly on a site whose owner proved it is theirs with a DNS record or a meta tag: that proof is checked on the home page once a day; up to 50 public pages are captured for its hosted MCP server, refreshed daily and at most once an hour on request.
A user’s own site, on their request
On request; agents twice a day at mostReal agents (Finish) run only when the user who marked the site as theirs asks. Each press first reads up to 4 of its pages, with no model, to see whether they changed since the last run; a user can press at most 10 times an hour. When the pages changed, agents run at most twice a day per site, once for its verified owner and once for anyone who marked it without proof: they write 3 questions, then browse it to answer them, at most 12 steps a question, and stay on the site’s host. Without proof, a user gets one run a day across all their sites, and none on a site whose owner asked Themelior to stop. When an agent never reached a question’s page, the site’s sitemap.xml (and up to two sitemaps it names) and llms.txt are read once more to say why.
One research run
2026-10-24The home pages of 100 recent Y Combinator companies: one scan each, one site at a time with two seconds between them, then one more GET of each home page.
What a scan requests.
Eight requests on the origin of the page given: the page first, then the other seven together.
| Request | To find |
|---|---|
| GET <the page given> | the page itself |
| GET /robots.txt | crawler rules and sitemaps |
| GET /sitemap.xml | the list of pages |
| GET /llms.txt | the summary written for models |
| GET /.well-known/mcp.json | an MCP server description |
| GET /openapi.json | an API description |
| GET /.well-known/openapi.json | an API description |
| GET http://<the page given> | whether plain HTTP moves to HTTPS |
What it never does.
- Runs JavaScript, sends cookies, signs in or submits a form: every request is a GET
- Solves or bypasses a bot challenge or a CAPTCHA: one is recorded as what an agent met
- Requests a private, loopback or reserved address
- Reads more than 2 MB of a response or waits more than 8 seconds for one
robots.txt, and how to stop it.
ThemeliorBot reads its own group in robots.txt, or the * group when it has none.
Captures follow robots.txt
The pages captured for a verified site’s hosted MCP server are only the pages your robots.txt allows. This keeps every page out:
User-agent: ThemeliorBot
Disallow: /A scan reads it, and still asks for the page
A scan is one page a person asked for, like a browser visit: robots.txt is one of the files it checks, not a reason to skip the page. To have scans of your site stopped, or a report about it removed, write to hello@themelior.com with the host. A person reads it; once control of the site is shown, the host goes on our opt-out list: no one can scan it through Themelior and its reports are unlisted.
Blocking the user agent at your server or CDN stops every request at once; a report then shows the block, which is what any agent would meet.