Bot Appétit: Our Free AEO and SEO Scanner for a (Mostly) Dead Internet

Honestly, the most important visitor to your website this week probably wasn't a person.

It was a bot. Possibly several. One from Google, sure, but also one from OpenAI, one from Anthropic, one from Perplexity, and a rotating cast of scrapers with names you've never heard of. They read your homepage, squinted at your heading structure, checked whether you'd locked the front door in robots.txt, and then quietly decided whether your business exists.

That decision (whether the machines can read you, understand you, and cite you) is quickly becoming the difference between being found and being invisible. So we built a tool that tells you, in about 20 seconds, how your site tastes to the bots.

It's called Bot Appétit. It's completely free, every feature, for everyone, and it works on any site, not just Webflow. But before we get to the tool, let's talk about why we built it. Because the web has been changing under our feet, and the story starts with a conspiracy theory that turned out to be uncomfortably close to the truth.

The dead internet theory: from tinfoil hat to traffic report

Around 2021, a theory started doing the rounds on fringe forums (and later The Atlantic): the internet "died" years ago, and most of what you see online is generated by bots talking to bots, with a shrinking number of actual humans wandering around the ruins.

As originally stated (a coordinated plot to replace people with machines), it was nonsense. Fun nonsense, but nonsense.

Then generative AI happened, and the numbers started catching up with the paranoia:

• Bots now generate more traffic than humans. Imperva's 2025 Bad Bot Report found automated traffic crossed 51% of all web traffic in 2024, the first time in a decade that machines outnumbered us on our own network.
• A huge share of new content is machine-made. An Ahrefs study found 74% of newly published pages contain at least some AI-generated content, and a separate study found roughly 17.6% of new websites are completely AI-generated.
• Academics have stopped laughing. Computer scientists are now publishing stripped-down, conspiracy-free versions of the theory built on observable facts: algorithmic feeds, generative content, and humans who genuinely can't tell the difference anymore.

Nobody planned it. There's no lizard cabal. But functionally? A web where the majority of traffic is automated and a majority of new pages involve AI is a web where the primary reader is a machine.

Sit with that for a second, because it changes everything about how you should think about your website.

For twenty-odd years we've written web pages for humans, with a polite nod to Googlebot on the way out the door. That ratio has now flipped. The bots aren't the side audience anymore. They're the audience. The humans increasingly sit downstream of them, reading whatever summary the machines decided to serve.

The part that actually hurts: nobody's clicking

"Fine," you say, "bots read my site. Humans still click the blue links."

Do they, though?

• Roughly a quarter of Google searches now trigger an AI Overview, according to Conductor's 21.9-million-search study.
• When an AI Overview shows up, the #1 organic result loses about 58% of its clicks, per Ahrefs' 300k-keyword study.
• Pew found users click a traditional result on just 8% of searches with an AI summary, versus 15% without. Nearly half your clicks, gone, before anyone even scrolls.
• Meanwhile, AI search itself is exploding: visits to AI search platforms grew 42.8% year-on-year to 27.4 billion while classic Google search grew 2.4%. Basically flat.

And here's the twist that stops this from being purely depressing: the traffic that does arrive from AI answers is good. Really good. Semrush found an AI search visitor is worth about 4.4× the average organic visitor. Which makes sense: by the time someone clicks through from ChatGPT or Perplexity, the machine has already done the comparison shopping and served you up as the answer. They arrive pre-convinced.

So the game isn't "rank #1 and harvest clicks" anymore. The game is: be the source the machine quotes. Fewer clicks, but the ones you get are warm. That game has a name.

AEO: the two-year-old discipline eating a twenty-five-year-old industry

AEO (answer engine optimisation) is the practice of making your content easy for AI answer engines (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini) to read, understand, trust, and cite. SEO gets you ranked in a list of links. AEO gets you named inside the answer.

They overlap heavily (nearly everything that's good AEO is also good SEO), but AEO shifts the unit of optimisation from the page to the individual, independently-citable fact. Answer engines don't read your page top to bottom like a person. They parse it in chunks, lift the sections that cleanly answer a question, and cite whoever made that easiest.

What people miss: this discipline barely existed two years ago. Look at the timeline:

• May 2024: Google rolls out AI Overviews in the US. The blue links get a roommate.
• September 2024: llms.txt is proposed: a curated, markdown map of your site for AI. Brand new convention, still unconfirmed by the major engines, already showing up in audits everywhere.
• October 2024: OpenAI launches ChatGPT search. An answer engine with a browser, pointed at your website.
• Late 2024: Anthropic open-sources MCP, and suddenly "agents that act on your site" goes from sci-fi to spec.
• September 2025: Cloudflare launches its Content Signals Policy, letting sites declare how AI may use content, already switched on for over 3.8 million domains. Google eventually shrugged: in July 2026 John Mueller said it has no effect on any crawler. The standards are being argued about in production.
• July 2026: Cloudflare gave AI companies a September deadline to separate their search crawlers from their training crawlers or get blocked by default from any pages that host ads (that date has now passed).

That's not a mature field. That's a field where the ground moves roughly every two weeks. Best practice from six months ago is already quaint. Some of what's in this post will be quaint by Christmas (and we say that as the people who now maintain a scanner that has to keep up with it).

Which brings us to why we built the thing.

Why we built Bot Appétit

The honest origin story: we kept doing this by hand.

Every project that went out the door at Milk Moon Studio got the same ritual: check robots.txt isn't blocking GPTBot, check the schema validates, check there's exactly one H1, check the headings actually nest, check the FAQ markup matches the visible FAQ, check llms.txt, check the security headers, run PageSpeed, squint at fifteen browser tabs. Then a client would ask "but how do we look to AI?" and we'd do it all again on their site, manually, like it was 2009 and we were hand-checking meta keywords.

The tools that existed were either paywalled, demanded your email before showing you anything, checked classic SEO but ignored AI entirely, or checked "AI readiness" while ignoring the SEO fundamentals that still do most of the heavy lifting. Nothing looked at the whole plate.

So we built Bot Appétit. Here's what it is, answer-first, the way we'd want an answer engine to quote it:

Bot Appétit is a free website scanner by Milk Moon Studio that checks how ready a page is for both AI answer engines and classic search. Enter a URL and you get a score from 0 to 100 with a category-by-category breakdown in about 20 seconds. No email, no paywall, no paid tier. All of it is free. Think of it as an SEO scanner with AEO built in: security, SEO and AEO in one scan.

A few things we decided early and feel strongly about:

• Completely free, for everyone. Not free-trial free. Not "enter your email and a sales rep will circle you like a seagull" free. Free. You paste a URL, you get your score, no account needed. There is no paid version waiting behind the free one. We're a Webflow studio, but the scanner doesn't care what your site is built on: WordPress, Shopify, Squarespace, hand-rolled HTML from 2011, scan away.
• It checks the whole menu. 66 checks across security, classic SEO, structured data, AEO/content structure, AI access, AI signals, link health, and an Advanced course covering the emerging agent and MCP-readiness signals, plus your Core Web Vitals measured live through Google PageSpeed.
• It tells you why, not just what. Every check comes with why it matters and what good looks like, and every one of them now has its own explainer page. The full catalogue is public on the what we check page. It's the exact list the scanner runs, no mystery meat.
• We'll keep it current. This is the big one. AEO changes basically every two weeks: new crawlers, new directives, new well-known files, new arguments between Cloudflare and Google. Keeping up is our job now. When the ground moves, the scanner moves with it. That's the deal.

One honest limitation, stated plainly because we'd flag it in anyone else's tool: the quick scan on the homepage reads one page at a time. It also grabs the site-wide stuff around that page: robots.txt, sitemap, llms.txt, security.txt, the /.well-known/ agent files, an AI-discovery DNS lookup, and a missing-page probe, but it's not crawling your whole site. Since we first wrote this, we've added a whole-site scan that works through a representative sample of your pages, gives each one its own full report and rolls them up into a site score. It's free like everything else; you just sign in so your pages, sites and history can tie together in one place.

What's actually on the menu

The full list lives on the checks page, but here's the quick tour of the courses:

AEO / content. Can a machine parse you in chunks? Semantic landmarks instead of div soup, self-contained sections with descriptive headings, visible authorship, dates, llms.txt. Answer engines lift passages, not pages. This course measures how liftable you are.

AI access. The one that produces the most facepalms. Plenty of sites block GPTBot, ClaudeBot, and PerplexityBot in robots.txt, sometimes deliberately, often because a plugin did it in 2023 and nobody noticed. Blocked crawlers can't cite you. You've unplugged yourself from AI answers and called it a privacy strategy.

Structured data. Your JSON-LD: present, valid, and the right type for the page. FAQ schema, Organization schema with sameAs, the machine-readable version of "this is who we are," and the foundation of being confidently named by an AI rather than vaguely gestured at.

SEO. The classics, because they still carry the meal: titles, metas, one H1, heading order, canonicals, sitemaps, alt text, Open Graph. Answer engines were trained on a web organised by these rules. They still matter, arguably more, because now two kinds of machine are grading you on them.

Links, security, performance. Healthy anchors, HTTPS done properly, secure headers, no leaked server internals, and your Core Web Vitals pulled live from Google PageSpeed, both field data and a lab run.

Advanced. The frontier course: agent and MCP-readiness signals, the /.well-known/ files that let AI agents discover and act on your site. Early days, low stakes today, moving fast. This is the section we expect to rewrite most often.

What's new since launch

We said we'd keep it current. Since this post first went out in July, Bot Appétit has had a full redesign and grown well past a single-page scanner. Here's what you can do with it now:

• Scan a whole site. Give it your domain, and it works through a representative sample of your pages in the background, each with its own full report. Pages built from the same template are grouped, and where pages of one template scored differently, the report shows you the range instead of pretending one page speaks for all of them.
• Keep your sites in one place. Sign in (it's free, and it's only there so your scans can tie together) and every site you scan lives under Your sites, with its history, so you can see whether last week's fixes actually moved the needle.
• Pin the pages that matter. Your pricing page, your best blog post, the landing page your ads point at: pin them and they're rescanned with every site scan, and they count towards your site score.
• Page history. Rescan a page and you get a new line in its history, not an overwrite. Every scan also records which version of our scoring produced it, so when a score moves because of us rather than your site, the report says so.
• Share a scan. A site scan is private by default. Share it with a client, your developer or your boss, and they can open it without signing up for anything.
• Performance on demand. Core Web Vitals come from a fresh run through Google PageSpeed (field data and a lab run), never stitched onto an old result.
• Every check explained, on its own page. All 66 checks now have a page of their own: what it looks for, why it matters to a machine reading your site, and what good looks like. Start at what we check.
• Help for everything. A help centre with a short guide for everything you can do on the site, from reading your report to sharing a scan, plus a plain-English glossary and a directory of the AI crawlers out there and what each one does.
• Turned away? We tell you how to let us in. If a firewall or bot protection blocks our scanner, the report says what stopped us and what to do about it, and a site that blocks us gets no score at all rather than a made-up one. Our bot, BotAppetitBot, is named and signed, so you can check a request really came from us.
• A public changelog. Every change to how we check or score goes on the changelog, flagged when it could move your score, so you're never left wondering why this month's number differs from last month's.

Underneath all that, the scanner has been through months of accuracy work against real sites: a category we couldn't measure no longer shows up as a zero, site scans no longer count the same page twice, and we now look in more than one place before telling you something is missing. The goal hasn't changed. One honest answer to one question: can a machine read this page?

Why it's free, and how you can help

Bot Appétit is free. Not a free tier, not a trial: the whole product, every feature, for everyone. We built it for ourselves first, and we think every site owner deserves to know whether the machines can read them without paying or handing over an email address to find out. Scanning a page needs nothing at all. Signing in, which is also free, simply ties your site scans, pinned pages and history together so you can come back to them. That's a promise, not a launch offer.

There's a selfish reason too. Every site you scan teaches us something: an edge case we hadn't seen, a platform that does things its own way, a check that was too harsh or too kind. A lot of the fixes above came from exactly that. So scan your site, scan your clients' sites, scan the competition. And if the report gets something wrong, tell us. It makes the scanner better for everyone.

Scanned it. Now what?

You'll get a score, and unless you've been unusually diligent, some red.

Don't panic, fix things in order. High-impact first: unblock the AI crawlers you actually want, sort your titles and H1s, get valid schema on the page, add the FAQ. Then work down through the mediums (and if performance is where you're bleeding, our Cloudflare optimisation guide covers how we fix that on Webflow). If you want the full prioritised workflow, we (conveniently) maintain one: our free Webflow SEO & AEO Checklist, which has 52 steps across 7 phases, each rated by impact and effort, with the what, the how, and the why. It's Webflow-flavoured in its instructions, but the principles apply anywhere. Between the scanner telling you what's wrong and the checklist telling you how to fix it, you're most of the way there without paying anyone anything. And if you build pages at scale, our post on programmatic Webflow landing pages covers the SEO side of that, which the scanner will happily check afterwards.

And if you'd rather have a human walk through your results with you, that we do charge for, sort of, in the sense that you have to endure a video call with Jakes. Book a call, it's a plain-English chat, no pitch. And if you'd rather hand the fixing over, you can get in touch with the studio about your Webflow site.

The honest caveats

We ran our own checklist writing this post, so in the spirit of the "schema-content parity" check: some disclosure.

AEO is young, and the numbers move. Some of the stats above will drift; where the research is genuinely uncertain (llms.txt adoption, for instance, which no major engine has confirmed using), the scanner weights it accordingly and we say so rather than dressing a blog estimate up as gospel. And no scanner, ours included, can score the thing that still matters most: whether your content is actually worth citing. Structure gets you eligible. Authority, links, and genuinely useful content get you chosen. A perfectly-marked-up page of nothing is still nothing, just easier to parse.

The dead internet, it turns out, has a very discerning palate.

FAQ

What is Bot Appétit?

A free website scanner by Milk Moon Studio that checks how ready your site is for AI answer engines and classic search. Enter a URL and get a score from 0 to 100 across 66 checks (AEO, SEO, structured data, AI access, security, links, and live Core Web Vitals) in about 20 seconds. Think of it as security, SEO and AEO in one scan.

Is Bot Appétit really a free SEO scanner?

Yes, completely free. Every feature, no paid tier, no trial that ambushes you later. You can scan a page without an account; signing in (also free) just keeps your sites and history together. It works on any website on any platform, not just Webflow. We built it because we needed it for our own projects, and gating it felt silly.

What is AEO and how is it different from SEO?

AEO (answer engine optimisation) makes your content easy for AI engines like ChatGPT, Perplexity, and Google AI Overviews to read, understand, and cite. SEO gets you ranked as a link; AEO gets you named inside the answer. They overlap heavily, and Bot Appétit checks both.

Does it scan my whole site?

Yes, if you ask it to. The quick scan reads one page plus the site-wide signals around it: robots.txt, sitemap, llms.txt, security.txt, /.well-known/ agent files, an AI-discovery DNS lookup, and a missing-page probe. The whole-site scan works through a representative sample of your site and gives each page its own report. It is free like everything else; you sign in so the pages and their history tie together.

Can I share my results?

Yes. A site scan is private until you share it, and anyone with the link can open it without signing up.

Is the dead internet theory actually true?

The conspiracy version, no. But bots did pass humans as a share of web traffic in 2024, and most newly published pages now involve AI-generated content. The practical takeaway is real either way: the web's primary reader is increasingly a machine, and your site should be legible to it.

Sources are linked inline throughout. Stats reflect the research available as of July 2026 (the product sections were updated in October 2026), and given how fast this field moves, check the date before quoting us at a dinner party.

Until next time, keep building (for humans and robots).