WebVitalsCollectorBot
If you found WebVitalsCollectorBot in your server logs, this page explains who we are and what it does.
WebVitalsCollectorBot is the website checker of Web Vitals Collector, run by Fusion Edge Development. We use it to check the websites of our own customers, once a week, so they can see what changed on their pages and what needs fixing. It does not crawl the web at large. It visits a website only in two cases: the owner (or their agency) has registered that website with us and switched site checking on, or a customer has named it as a competitor to compare with, in which case it reads only the few public files described below.
If you did not expect a visit from us, or you want it to stop, see how to stop it below or email [email protected].
How to recognise it
Requests from our crawler carry this User-Agent:
WebVitalsCollectorBot/1.0 (+https://fusionedgedevelopment.uk/crawler/)What it does
Once a week it checks the registered website of a customer:
- It reads
https://yoursite.com/robots.txtfirst and follows it. It also notes what that file says about common AI crawlers (see AI crawlers and your robots.txt). - It starts at the home page and the pages listed in
/sitemap.xml, then follows links between pages of the same website. - It reads the page’s HTML and notes a small set of facts about it (see below).
- It sends a small number of extra requests to check whether links it found still work.
What it fetches and notes
For each page it keeps a short list of facts, not the page itself:
- the HTTP status and any redirect,
- the page title and meta description,
- the canonical link and robots directives (for example
noindex), - the main heading (H1),
- the types of structured data (JSON-LD) present,
- a rough size bucket for the amount of text, and fingerprints (hashes) used only to tell whether a page changed,
- counts of internal links and images without alt text,
- whether the page needs JavaScript to show its content, and how many second-level headings (H2) it has,
- whether its structured data can be read, and whether article structured data names a modified date and an author,
- whether the page asks search tools not to show snippets of it.
It does not store the HTML of your pages, your images, or any visitor data. It does not run JavaScript, submit forms, log in, or follow links to other websites.
AI crawlers and your robots.txt
Many websites want to know which AI tools are allowed to read them. As part of the weekly check we read the rules in your robots.txt for a list of common AI and search crawlers, and tell the customer whether each one is allowed, asked to skip some pages, or asked to stay away. The list currently includes:
- OpenAI:
OAI-SearchBot,ChatGPT-User,GPTBot - Anthropic:
Claude-SearchBot,Claude-User,ClaudeBot - Perplexity:
PerplexityBot,Perplexity-User - Google and Apple controls for AI training:
Google-Extended,Applebot-Extended - Common Crawl:
CCBot - Search engines:
Googlebot,Bingbot
This is a common list, not every crawler there is, and it is reviewed regularly because vendors rename and add crawlers.
We never send requests as any of these crawlers. We only read what your robots.txt says about them, from the same file we already fetch, so no extra pages are requested. We do not fetch llms.txt.
What this cannot show. It reports what your robots.txt asks, not what your server or firewall does. A security service or CDN can block a crawler even when robots.txt allows it, and we cannot see that.
What we do not do. We do not use your pages to train AI, and we keep nothing but the short facts listed above. Whether to allow or block any crawler is your choice; allowing search crawlers while declining training crawlers is a common one, and we report the facts without recommending a policy.
Competitor sites that customers name
A customer can name up to three other websites, or single pages, to compare their own site with. Once a week, for each one, we read only the public files that describe how it is organised:
- Its
robots.txt, which we read first and follow. - Its sitemap (the one
robots.txtnames, otherwise/sitemap.xml; for a sitemap index, at most three of the sitemaps it lists). - Its home page.
That is at most six requests per site per week. If the customer named one specific page (for example a profile on a platform), we read robots.txt and that one page only: two requests, with no sitemap and no home page. We use the same User-Agent and the same pace as above, one request at a time, and we do not follow links to other pages.
If your robots.txt blocks WebVitalsCollectorBot, or cannot be read, we request nothing else and the customer is told the site could not be checked. We store only counts and labels: how many pages the sitemap lists, how many sections they fall into and how deep they go, how many pages the home page links to, and the wording of its main navigation. We do not store your page content or any visitor data. These facts are deleted after 90 days.
The comparison may also include Google’s PageSpeed Insights results for the page. For that we send Google only the page’s public address, and Google fetches the page itself; we make no request to your site for it.
The robots.txt rules below work the same way here. If you would rather we did not read your website for this purpose, email [email protected] with the address and we will stop.
How politely it behaves
| Pages per visit | At most 50 |
|---|---|
| Link depth | At most 3 clicks from the start |
| Pace | One request at a time, with at least 1 second between requests |
Crawl-delay in robots.txt | Honoured, up to 10 seconds |
| Page size | HTML only, up to 1 MB per page |
| Frequency | Weekly (a customer can also ask for an extra check, at most once every 24 hours) |
| Which sites | Only https pages on the customer’s registered website (the bare domain or www.). For a site a customer has named to compare with, only the files listed under competitor sites |
If your robots.txt blocks us, or cannot be read (for example your server returns an error or refuses the request), we do not crawl, and we record that the site could not be checked rather than assuming anything about it.
How long we keep the data
The page facts and any broken links are deleted after 90 days, together with the note of what your robots.txt said about AI crawlers. The counts we keep about a competitor site that a customer names are deleted after 90 days too. The record of what changed is kept for up to 365 days.
How to stop it
You can block or limit us with robots.txt, using the name WebVitalsCollectorBot.
Block the whole website:
User-agent: WebVitalsCollectorBot
Disallow: /Block only some sections:
User-agent: WebVitalsCollectorBot
Disallow: /private/
Disallow: /staging/Ask us to go slower (we honour values up to 10 seconds):
User-agent: WebVitalsCollectorBot
Crawl-delay: 10Changes take effect from our next visit, which is within a week. If you would rather not edit your robots file, or you are not sure which customer registered your site, email [email protected] with the website address and we will switch checking off for it.
Does it affect my website or my analytics?
The load is light: at most about 50 page requests plus a small number of link checks per week, spread out at roughly one request a second. Because it only reads HTML and does not run JavaScript, it does not trigger analytics scripts, so it should not appear as visitors in tools that depend on JavaScript.
What we do not claim
The checks describe your own pages and files: titles, links, headings, structured data and what your robots.txt says. For competitor sites they describe only how the site is organised (page counts, sections, navigation), never how good it is. They say nothing about search rankings, nothing about whether Google has indexed a page, and nothing about whether any AI tool reads, shows or recommends your website.
Contact us about the crawler
Questions, concerns or abuse reports: [email protected].