More people now ask ChatGPT, Claude, Perplexity or Google's AI features a question instead of searching. Some of those tools read websites to answer. That raises a fair question for any site owner: can they read mine?

This checklist covers the parts you control. It does not, and cannot, promise that any AI product will mention your business. Nobody can promise that, and anyone who does is guessing.

The short answer: make sure the tools you want can reach your pages, that your pages show their content without special software, and that it is clear who you are and when your content was written. Then check again every month, because a small change to a file or a plugin can quietly undo it.

1. Check what your robots.txt says about AI crawlers

Your robots.txt file, at yourwebsite.com/robots.txt, tells crawlers which parts of your site they may read. It is the first thing to look at, and it is easy to get wrong without noticing.

AI companies run more than one kind of crawler, and the difference matters:

Search crawlers (for example OAI-SearchBot, Claude-SearchBot, PerplexityBot) collect pages so a product can show or link to them in answers.
User-triggered crawlers (for example ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked about it. Some vendors say these may not follow robots.txt.
Training crawlers (for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect content to train AI models.

Whether to allow each kind is your decision. Many sites allow search crawlers and decline training crawlers; others allow everything or nothing. What matters is that the file says what you meant. A single line such as Disallow: / under User-agent: * blocks every one of them, often left over from a development site.

What to check: open your robots.txt, read any User-agent line that names an AI crawler, and confirm you would say the same thing out loud.

2. Make sure your pages show content without JavaScript

Some crawlers read the HTML your server sends and do not run JavaScript. If your page arrives as an empty shell that a script fills in later, those crawlers may see nothing.

What to check: view the page source (not the inspector) in your browser and look for your headline and main text. If they are missing, ask your developer about server-side rendering or pre-rendering for your key pages.

3. Add structured data that says who you are

Structured data (JSON-LD) is a small block of code that describes your business or article in a form machines read easily: a name, an address, a logo, an author, dates. An Organization or LocalBusiness block on your home page is the usual starting point.

It does not guarantee a mention anywhere. It removes guesswork about who you are, and a broken block (a missing comma is enough) is worse than none, because it may be ignored entirely.

What to check: run your home page through a structured data validator and fix any errors it reports.

4. Use clear headings

A page with one clear main heading (H1) and meaningful subheadings (H2) is easier for people and machines to follow, and it makes good sections for an answer to quote. A page made of styled paragraphs with no headings gives a crawler nothing to hold on to.

What to check: each important page should have exactly one H1 that says what the page is, and subheadings that match the questions a customer would ask.

5. Put dates and an author on articles

If you publish articles, mark the date published, the date last updated and who wrote them, both on the page and in the structured data. It helps a reader (and a machine) judge how current the content is.

What to check: open a recent article and look for a visible date and author. Then check its structured data has datePublished, dateModified and author.

6. Check your firewall and CDN

robots.txt is a polite request. Your hosting or a security service may enforce something different: a firewall can refuse a crawler that robots.txt allows, and sometimes refuses crawlers you want by mistake.

What to check: if you use a CDN or security plugin, look for a setting about bots or AI crawlers and make sure it matches your robots.txt. This is the one thing a robots.txt check cannot see, which is why our own checks say so.

7. Treat llms.txt as optional

You may have read about llms.txt, a proposed file that lists your key pages for AI tools. It is cheap to add, and we publish one ourselves. But as far as we know, no major AI search product has confirmed it reads the file, so do not expect it to change anything. Do the items above first.

What nobody can promise

No checklist can make an AI product mention, cite or recommend your website. Which pages these tools show depends on their own systems, which change often and are not public. What you can control is whether your site is readable and clear. That is worth doing for ordinary visitors and search engines too.

Checking it every week instead of once

These settings drift. A theme update rewrites robots.txt, a plugin starts blocking bots, a redesign hides text behind a script. Web Vitals Collector reads your robots.txt every week and tells you whether each of a list of common AI and search crawlers is allowed, asked to skip some pages, or asked to stay away. It also flags pages that show no content without JavaScript, structured data that cannot be read, articles without a date or author, and pages with no headings. We only report the facts; we do not claim any of it affects whether you are mentioned.

See how our website crawler works for exactly what it reads, or the FAQ for common questions.