Outpost's checks

Outpost watches websites for the people responsible for them: whether the site answers, when its domain and certificate expire, and what an attacker could see from outside. If you found Outpost in your logs, someone has added your site to Outpost, or asked for a one-off check of it. This page says exactly what Outpost asks your site for, and how to limit or allow it.

How to recognise it

Every request Outpost makes names it in the User-Agent header:

Outpost/0.1 (site registry and monitoring; +https://outpost.codeandvin.com/bot)

It comes from these addresses, and no others:

• 65.108.249.35
• 2a01:4f9:c012:74e4::1

Anyone can copy a User-Agent, so to allow Outpost through a firewall, allow these addresses rather than the name. If they change, this page changes first. The same list is at /bot/ips.json for tools that read one.

What it asks for

Outpost only reads public pages, the way a browser or a search engine does. It sends GET requests and nothing else.

• Uptime check. The homepage, every 5 minutes on paid plans and every 24 hours on the free plan. While a site is failing it is asked again every minute until the outcome is clear. If a firewall challenges the plain request, the homepage is loaded once in a headless browser instead.
• Daily scan, once a day per site, and never more than once in ten minutes per domain whoever asks: the homepage (over HTTP and HTTPS, to see the redirect), the certificate (and the www address’s, which can differ), robots.txt, sitemap.xml and sitemap_index.xml; on WordPress sites the REST API index, its public user list and readme.html; up to 8 sitemaps the site itself declares; the homepage once in a headless browser, which loads what the page loads; the status of up to 20 links on the homepage; and the homepages of up to 6 subdomains it links to.
• Cloaking check, only for a site whose owner asked for it: one whose control they proved to Outpost, or one Outpost’s operator watches on its owner’s behalf. The homepage and up to 5 pages from its sitemaps are fetched three ways: as Outpost, as Googlebot, and as a phone arriving from a Google search. Hacked sites often show spam only to Google, and this is the only way to see it. These are the only requests that do not name Outpost.
• Proof of control. When someone proves they control a domain by a file, Outpost reads /.well-known/outpost-verification.txt, then once a day to see it is still there.
• Public check. Anyone can ask Outpost to check any domain on its check page. That runs the daily scan above, without the cloaking check, at most once a day per domain. Accounts can also audit a domain once, which is the same scan.

Outpost never logs in, never submits a form, never guesses at addresses beyond the fixed list above, and never tries a password. Some of what it learns comes from elsewhere, not from your site: domain registration from the registry, DNS, Google’s blocklist, and WordPress.org’s plugin directory.

robots.txt

Outpost follows robots.txt (RFC 9309) for everything above except the homepage and the certificate, which every monitor reads, and the proof-of-control file, which a site’s owner puts there for Outpost. It answers to the name Outpost, and to * when no group names it. For example, to keep it out of your REST API:

User-agent: Outpost
Disallow: /wp-json/

Whatever robots.txt holds back is listed in the site’s registry as not checked, so its owner knows a check was not made. If robots.txt cannot be read because the server fails, Outpost reads only the homepage until it can.

If your firewall challenges Outpost

Uptime is still checked, through the headless browser, but parts of the daily scan cannot be made, so problems may go unseen. Outpost does not disguise itself to get past a challenge. To let it through:

• Cloudflare, Free plan. Bot Fight Mode cannot be switched off for one visitor by a custom rule. Add an IP Access rule instead (Security, then Security rules, then Create rule, then IP access rules) for each address above, with the action Allow, for this website only.
• Cloudflare, Pro and above. Add a custom rule (Security, then Security rules, then Create rule, then Custom rules) whose expression is (ip.src in {65.108.249.35 2a01:4f9:c012:74e4::1}), with the action Skip and “All Super Bot Fight Mode rules” ticked. This skips only the bot check.
• Sucuri, Imperva and others. Add the addresses to the firewall’s allowlist.

If you use Outpost, the site’s registry page has these steps filled in for your site, and a button to check again once you have made the change.

To stop it

To stop the daily scan reading anything past your homepage, disallow Outpost in robots.txt. To stop the uptime check, or if you did not expect Outpost to watch your site at all, write to Vincent@codeandvin.com with the domain. Blocking the addresses above at your firewall also works.

Outpost is run by Vincent Jacobs, trading as Code & Vin. The privacy policy says what is kept and for how long.