NorrelBot

Last updated 10 October 2026

NorrelBot is the crawler behind Norrel. It reads a small number of public pages on companies our customers track, so we can tell them what changed. This page says exactly what it does and how to turn it off.

How to recognise it

Every request carries this user agent:

We run on Heroku in Ireland and do not have a fixed IP address, so there is no range to allow or block. The user agent is the reliable way to identify us, and we will not change it without updating this page.

How to stop it

Add either of these to your robots.txt. We read robots.txt before every request, for the site that request goes to, and honour it — a page you disallow is a page we do not request. If your robots.txt cannot be reached, we treat that as a no and try again later.

We also answer to the tokens Norrel and norrel-bot. We never send more than one request a second to a site, and a Crawl-delay in your file, or in the wildcard group, slows us further; one longer than thirty seconds means we skip your site until its next check rather than hold a request open.

If you would rather not edit a file, write to hello@getnorrel.com and we will add the exclusion at our end the same day.

What it reads

• Your robots.txt, your home page, and the sitemaps you list — to see which pages exist. About once a week.
• Your pricing page, your careers page, your changelog and, if you publish one, your changelog for developers. About every three days each, usually as a conditional request that costs you a 304.
• When your sitemap lists new pages, at most ten of them a week, once each, to read their title and opening lines.

It finds those pages by trying the obvious addresses first, then the links on your home page, then your sitemap. The first time a customer adds you it makes a few dozen requests; after that, a handful a week.

It follows links you published, one step: from your careers page to the hiring platform your roles are listed on, from your home page to your developer site and from there to its changelog, from your home page to a changelog you host elsewhere. It never guesses an address on a platform you may not use, and it does not read job aggregators, because those are not your statement about your own hiring.

What it will not do

• It does not run JavaScript, so it never touches anything your pages load afterwards.
• It does not sign in, submit forms, or accept or send cookies.
• It does not spider your site. It uses your sitemap to see which pages exist, and reads only the pages listed above.
• It does not read anything behind a login, a paywall, or a Disallow rule.
• It stops and backs off when a server asks it to, and honours Retry-After.

What we keep

The readable text of the pages above, their addresses, when we read them, and what we found: prices, the titles, teams and locations of open roles, changelog entries, and the titles of new pages. That is so a change we report to a customer can be checked against what was actually published. We keep the text of a reading only when it differs from the one before, and delete it after 90 days except the latest reading. We do not keep your images, your scripts or your HTML, and we do not republish your pages.

The text is sent to OpenAI to be read into a structure, with a request that it not be retained. Nothing about our customers goes with it.

Contact

A person reads hello@getnorrel.com. If NorrelBot is doing something you did not expect, tell us and we will fix it rather than argue about it.

Back to getnorrel.com