Feranor

FeranorBot is the web crawler operated by Feranor. This page explains what it does, why it may have visited your site, and how to control or block it. If you have a question this page doesn't answer, contact us.

User agent

Match on the token 'FeranorBot' when writing rules or filtering logs. The version number will change over time; the token will not.

Why you're seeing this bot

There are two reasons FeranorBot may appear in your logs.

A site check was requested. Feranor operates a public tool that reports on a domain's publicly observable configuration, such as TLS certificate, security headers, DNS setup, response time. Anyone can submit any domain, so a check may have been run by someone other than you.

A crawl was run. Feranor runs crawls to map site structure and track how it changes over time. A crawl fetches multiple pages and follows internal links.

What it does

• Requests

  HTTP GET only. Never other HTTP verbs.

• Rate

  One request at a time, with a minimum 200 ms gap. No parallel connections.

• Default budget

  Up to 200 URLs per crawl, up to 2 links deep from the entry point.

• JavaScript

  Not executed. Only the HTML your server returns is read.

• Forms and logins

  Never submitted. Only publicly reachable pages are fetched.

• Sitemaps

  Read if declared in robots.txt, and only when hosted on your own domain.

• robots.txt

  Fetched before every crawl and obeyed. See below.

Controlling FeranorBot

FeranorBot follows the Robots Exclusion Protocol, including * wildcards, $ end-of-path anchors, and Crawl-delay.

A group naming FeranorBot replaces the User-agent: * group rather than adding to it. If you write a FeranorBot group, restrictions in your wildcard group no longer apply to us. Changes take effect on the next crawl, since robots.txt is re-read each time.

Verifying a request really came from us

Any client can put our user-agent string in its requests. The string alone is not proof. We do not yet publish a list of source IP addresses. Until we do, treat a matching user-agent as an unverified claim. If you're seeing traffic that identifies as FeranorBot and the behaviour doesn't match what's described on this page. It is very likely not us, and we'd like to know about it.

Where FeranorBot shows up

• Run a check - Free, no account. Enter any domain and get its TLS, header, and infrastructure signals back in a few seconds.

• MCP server - The same resilience check as a tool for agents, priced per call via x402. Built for automation.
  Hosted on: https://www.feranor.com/mcp

Contact

For blocking help, complaints about crawl behaviour, or reports of traffic falsely identifying as FeranorBot. Include the relevant log lines with timestamps and source IPs where you can. Contact us.