Flavr

FlavrBot

FlavrBot is the web crawler run by Flavr. Flavr is building a directory of restaurants in the United States, and FlavrBot reads restaurant websites to check that each listing has the right name, address, phone number, hours and links.

How to recognise it

FlavrBot identifies itself in every request with this User-Agent:

Mozilla/5.0 (compatible; FlavrBot/0.1; +https://flavr.app/bot.html)

If a page requires JavaScript to show its content, FlavrBot loads it in a browser, and the request carries this one:

FlavrBot/0.1 (+https://flavr.app/bot.html)

The version number may change. The name FlavrBot will not.

What it reads

Only what a restaurant has published about itself:

• The name of the business and how it describes itself.
• The address, phone number and opening hours of each location.
• Links the site gives to its online ordering, reservations, maps and social pages.

How it behaves

• It reads very little. A visit covers the home page and, if the site links to them, its contact page and its locations page. Where a site gives each location a page of its own, it reads up to three of those. It does not follow other links or work its way through a site.
• It goes slowly. It waits at least one second between requests.
• It follows robots.txt. It fetches robots.txt before reading any page and obeys the rules written for FlavrBot or for all crawlers.
• It only reads pages. It requests the page itself: no images, stylesheets or downloads. It does not fill in forms, sign in to anything or try to get past a check meant to keep automated visitors out.
• It runs JavaScript only if a page requires it. If a page requires JavaScript to show its content, FlavrBot loads that one page in a browser, which fetches the scripts the page needs. It sends nothing to analytics or advertising trackers, and it follows the same robots.txt rules as every other request.
• It caches what it reads. FlavrBot holds a temporary cached copy of each page it has read, so that it makes no unnecessary calls to your site for a page it already has.

How to block it

To keep FlavrBot away from your whole site, add this to your robots.txt:

User-agent: FlavrBot
Disallow: /

To keep it away from part of a site, name that part instead of /.

Contact

If FlavrBot is causing a problem for your site, or you have a question about it, write to developers@flavr.app.