Bot Appétit: The Internet Is (Mostly) Dead, AEO Is Eating What's Left, and We Built a Free Scanner for It

Let's be honest — the most important visitor to your website this week probably wasn't a person.

It was a bot. Possibly several. One from Google, sure, but also one from OpenAI, one from Anthropic, one from Perplexity, and a rotating cast of scrapers with names you've never heard of. They read your homepage, squinted at your heading structure, checked whether you'd locked the front door in robots.txt, and then quietly decided whether your business exists.

That decision — whether the machines can read you, understand you, and cite you — is quickly becoming the difference between being found and being invisible. So we built a tool that tells you, in about 20 seconds, how your site tastes to the bots.

It's called Bot Appétit. It's completely free, no signup, and it works on any site — not just Webflow. But before we get to the tool, let's talk about why we built it. Because the web has been changing under our feet, and the story starts with a conspiracy theory that turned out to be uncomfortably close to the truth.

The Dead Internet Theory: From Tinfoil Hat to Traffic Report

Around 2021, a theory started doing the rounds on fringe forums (and later The Atlantic): the internet "died" years ago, and most of what you see online is generated by bots talking to bots, with a shrinking number of actual humans wandering around the ruins.

As originally stated — a coordinated plot to replace people with machines — it was nonsense. Fun nonsense, but nonsense.

Then generative AI happened, and the numbers started catching up with the paranoia:

• Bots now generate more traffic than humans. Imperva's 2025 Bad Bot Report found automated traffic crossed 51% of all web traffic in 2024 — the first time in a decade that machines outnumbered us on our own network.
• A huge share of new content is machine-made. An Ahrefs study found 74% of newly published pages contain at least some AI-generated content, and a separate study found roughly 17.6% of new websites are completely AI-generated.
• Academics have stopped laughing. Computer scientists are now publishing stripped-down, conspiracy-free versions of the theory built on observable facts: algorithmic feeds, generative content, and humans who genuinely can't tell the difference anymore.

Nobody planned it. There's no lizard cabal. But functionally? A web where the majority of traffic is automated and a majority of new pages involve AI is a web where the primary reader is a machine.

Sit with that for a second, because it changes everything about how you should think about your website.

For twenty-odd years we've written web pages for humans, with a polite nod to Googlebot on the way out the door. That ratio has now flipped. The bots aren't the side audience anymore. They're the audience. The humans increasingly sit downstream of them, reading whatever summary the machines decided to serve.

The Part That Actually Hurts: Nobody's Clicking

"Fine," you say, "bots read my site. Humans still click the blue links."

Do they, though?

• Roughly a quarter of Google searches now trigger an AI Overview, according to Conductor's 21.9-million-search study.
• When an AI Overview shows up, the #1 organic result loses about 58% of its clicks, per Ahrefs' 300k-keyword study.
• Pew found users click a traditional result on just 8% of searches with an AI summary, versus 15% without. Nearly half your clicks, gone, before anyone even scrolls.
• Meanwhile, AI search itself is exploding: visits to AI search platforms grew 42.8% year-on-year to 27.4 billion while classic Google search grew 2.4%. Basically flat.

And here's the twist that stops this from being purely depressing: the traffic that does arrive from AI answers is good. Really good. Semrush found an AI search visitor is worth about 4.4× the average organic visitor. Which makes sense — by the time someone clicks through from ChatGPT or Perplexity, the machine has already done the comparison shopping and served you up as the answer. They arrive pre-convinced.

So the game isn't "rank #1 and harvest clicks" anymore. The game is: be the source the machine quotes. Fewer clicks, but the ones you get are warm. That game has a name.

AEO: The Two-Year-Old Discipline Eating a Twenty-Five-Year-Old Industry

AEO — answer engine optimisation — is the practice of making your content easy for AI answer engines (ChatGPT, Perplexity, Google AI Overviews, Claude, Gemini) to read, understand, trust, and cite. SEO gets you ranked in a list of links. AEO gets you named inside the answer.

They overlap heavily — nearly everything that's good AEO is also good SEO — but AEO shifts the unit of optimisation from the page to the individual, independently-citable fact. Answer engines don't read your page top to bottom like a person. They parse it in chunks, lift the sections that cleanly answer a question, and cite whoever made that easiest.

Here's the thing people miss: this discipline barely existed two years ago. Look at the timeline:

• May 2024 — Google rolls out AI Overviews in the US. The blue links get a roommate.
• September 2024 — llms.txt is proposed: a curated, markdown map of your site for AI. Brand new convention, still unconfirmed by the major engines, already showing up in audits everywhere.
• October 2024 — OpenAI launches ChatGPT search. An answer engine with a browser, pointed at your website.
• Late 2024 — Anthropic open-sources MCP, and suddenly "agents that act on your site" goes from sci-fi to spec.
• September 2025 — Cloudflare launches its Content Signals Policy, letting sites declare how AI may use content — rolled out to 3.8 million domains overnight. Google promptly shrugged and said it has no effect on their crawlers. The standards are being argued about in production.
• July 2026 — literally this month — Cloudflare gave AI companies a September deadline to separate their search crawlers from their training crawlers or get blocked by default across its network.

That's not a mature field. That's a field where the ground moves roughly every two weeks. Best practice from six months ago is already quaint. Some of what's in this post will be quaint by Christmas — and we say that as the people who now maintain a scanner that has to keep up with it.

Which brings us to why we built the thing.

Why We Built Bot Appétit

The honest origin story: we kept doing this by hand.

Every project that went out the door at Milk Moon Studio got the same ritual — check robots.txt isn't blocking GPTBot, check the schema validates, check there's exactly one H1, check the headings actually nest, check the FAQ markup matches the visible FAQ, check llms.txt, check the security headers, run PageSpeed, squint at fifteen browser tabs. Then a client would ask "but how do we look to AI?" and we'd do it all again on their site, manually, like it was 2009 and we were hand-checking meta keywords.

The tools that existed were either paywalled, demanded your email before showing you anything, checked classic SEO but ignored AI entirely, or checked "AI readiness" while ignoring the SEO fundamentals that still do most of the heavy lifting. Nothing looked at the whole plate.

So we built Bot Appétit. Here's what it is, answer-first, the way we'd want an answer engine to quote it:

Bot Appétit is a free website scanner by Milk Moon Studio that checks how ready a page is for both AI answer engines and classic search. Enter a URL and you get a 0–100 score with a category-by-category breakdown in about 20 seconds — no signup, no email, no paywall.

A few things we decided early and feel strongly about:

• Completely free, for everyone. Not free-trial free. Not "enter your email and a sales rep will circle you like a seagull" free. Free. You paste a URL, you get your score. We're a Webflow studio, but the scanner doesn't care what your site is built on — WordPress, Shopify, Squarespace, hand-rolled HTML from 2011, scan away.
• It checks the whole menu. 60+ checks across security, classic SEO, structured data, AEO/content structure, AI access, link health, and an Advanced course covering the emerging agent and MCP-readiness signals — plus your Core Web Vitals measured live through Google PageSpeed on every scan.
• It tells you why, not just what. Every check comes with why it matters and what good looks like. The full catalogue is public on the what we check page — it's the exact list the scanner runs, no mystery meat.
• We'll keep it current. This is the big one. AEO changes basically every two weeks — new crawlers, new directives, new well-known files, new arguments between Cloudflare and Google. Keeping up is our job now. When the ground moves, the scanner moves with it. That's the deal.

One honest limitation, stated plainly because we'd flag it in anyone else's tool: it scans one page at a time. It also grabs the site-wide stuff around that page — robots.txt, sitemap, llms.txt, security.txt, the /.well-known/ agent files, an AI-discovery DNS lookup, and a missing-page probe — but it's not crawling your whole site. Capped multi-page deep scans are on the roadmap.

What's Actually on the Menu

The full list lives on the checks page, but here's the quick tour of the courses:

AEO / content. Can a machine parse you in chunks? Semantic landmarks instead of div soup, self-contained sections with descriptive headings, visible authorship, dates, llms.txt. Answer engines lift passages, not pages — this course measures how liftable you are.

AI access. The one that produces the most facepalms. Plenty of sites block GPTBot, ClaudeBot, and PerplexityBot in robots.txt — sometimes deliberately, often because a plugin did it in 2023 and nobody noticed. Blocked crawlers can't cite you. You've unplugged yourself from AI answers and called it a privacy strategy.

Structured data. Your JSON-LD: present, valid, and the right type for the page. FAQ schema, Organization schema with sameAs — the machine-readable version of "this is who we are," and the foundation of being confidently named by an AI rather than vaguely gestured at.

SEO. The classics, because they still carry the meal: titles, metas, one H1, heading order, canonicals, sitemaps, alt text, Open Graph. Answer engines were trained on a web organised by these rules. They still matter — arguably more, because now two kinds of machine are grading you on them.

Links, security, performance. Healthy anchors, HTTPS done properly, secure headers, no leaked server internals — and your Core Web Vitals pulled live from Google PageSpeed, both field data and a lab run.

Advanced. The frontier course: agent and MCP-readiness signals, the /.well-known/ files that let AI agents discover and act on your site. Early days, low stakes today, moving fast. This is the section we expect to rewrite most often.

Scanned It. Now What?

You'll get a score, and unless you've been unusually diligent, some red.

Don't panic — fix things in order. High-impact first: unblock the AI crawlers you actually want, sort your titles and H1s, get valid schema on the page, add the FAQ. Then work down through the mediums (and if performance is where you're bleeding, our Cloudflare optimisation guide covers how we fix that on Webflow). If you want the full prioritised workflow, we (conveniently) maintain one: our free Webflow SEO & AEO Checklist — 52 steps across 7 phases, each rated by impact and effort, with the what, the how, and the why. It's Webflow-flavoured in its instructions, but the principles apply anywhere. Between the scanner telling you what's wrong and the checklist telling you how to fix it, you're most of the way there without paying anyone anything.

And if you'd rather have a human walk through your results with you — that we do charge for, sort of, in the sense that you have to endure a video call with Jakes. Book a call, it's a plain-English chat, no pitch.

The Honest Caveats

We ran our own checklist writing this post, so in the spirit of the "schema-content parity" check: some disclosure.

AEO is young, and the numbers move. Some of the stats above will drift; where the research is genuinely uncertain — llms.txt adoption, for instance, which no major engine has confirmed using — the scanner weights it accordingly and we say so rather than dressing a blog estimate up as gospel. And no scanner, ours included, can score the thing that still matters most: whether your content is actually worth citing. Structure gets you eligible. Authority, links, and genuinely useful content get you chosen. A perfectly-marked-up page of nothing is still nothing, just easier to parse.

The dead internet, it turns out, has a very discerning palate.

FAQ

What is Bot Appétit?

A free website scanner by Milk Moon Studio that checks how ready your site is for AI answer engines and classic search. Enter a URL and get a 0–100 score across 60+ checks — AEO, SEO, structured data, AI access, security, links, and live Core Web Vitals — in about 20 seconds.

Is Bot Appétit really free?

Yes — completely free, no signup, no email, no trial that ambushes you later. It works on any website on any platform, not just Webflow. We built it because we needed it for our own projects, and gating it felt silly.

What is AEO and how is it different from SEO?

AEO (answer engine optimisation) makes your content easy for AI engines like ChatGPT, Perplexity, and Google AI Overviews to read, understand, and cite. SEO gets you ranked as a link; AEO gets you named inside the answer. They overlap heavily, and Bot Appétit checks both.

Does it scan my whole site?

Not yet — it scans one page plus the site-wide signals around it: robots.txt, sitemap, llms.txt, security.txt, /.well-known/ agent files, an AI-discovery DNS lookup, and a missing-page probe. Capped multi-page deep scans are on the roadmap.

Is the dead internet theory actually true?

The conspiracy version, no. But bots did pass humans as a share of web traffic in 2024, and most newly published pages now involve AI-generated content. The practical takeaway is real either way: the web's primary reader is increasingly a machine, and your site should be legible to it.

Sources are linked inline throughout. Stats reflect the research available as of July 2026 — and given how fast this field moves, check the date before quoting us at a dinner party.

Until next time, keep building (for humans and robots). ✌️