Investigation
Audience Composition: Method, Row Tables and What the Beacon Cannot Support
How the site's view counter records a row, two windows of rows, spring and summer, regrouped by user agent family, a probe of every surface the site advertises to machines, the classifier's pattern list, and the five things this data cannot say.
How a row is recorded
Every tier carries the same beacon: a client script that fires POST /api/views/{page} on page
load and then GETs the same path to render the count. The API (api/src/routes/views.ts) stores
page_path, user_agent, ip_address, is_ai and created_at, deduplicating on
(page_path, ip_address, hour bucket) through a unique index (api/migrations/0017_views-dedup.sql).
The is_ai flag is set at write time when the user agent contains, case insensitively, any of
thirteen substrings: GPTBot, ClaudeBot, Claude-Web, anthropic-ai, ChatGPT-User, cohere-ai,
PerplexityBot, Meta-ExternalAgent, Googlebot, Bingbot, CCBot, Bytespider, Applebot
(views.ts:4-8). Nothing about the tier is stored, so a raw tier read and an editorial tier read of
the same slug are the same row. Because the beacon is JavaScript, a row is produced by a client that executes the page script, or
by anything that calls the endpoint directly under whatever user agent it presents; the table cannot
tell the two apart.
Two consumers read the table. GET /api/views/{page} returns total, ai, human for one page
and feeds the per post widget. GET /api/top-pages?days=N&limit=M (api/src/routes/metrics.ts)
returns per page views (rows with is_ai = 0) and ai_views over a trailing window, capped at
365 days and 50 pages, and feeds the tier 3 "most read" rail. The July review's 112 human / 110 AI figure is the sum over the top ten rows of that endpoint at 365 days. That review is an internal meta-review of the site dated 2026-07-25; its fourth recommendation asked for this post and the counter, and the sentence the post quotes reads "This is the founding thesis confirmed with original data."
The two windows
The analysis rests on two datasets, extracted by different methods, whose spans overlap for seventeen calendar days.
| Window A (export) | Window B (live) | |
|---|---|---|
| Source | api/backups/d1-pre-migration-2026-07-04.sql |
page_views, read 2026-08-20 |
| Method | rows loaded into scratch SQLite, grouped by family | read-only D1 SELECT over --remote |
| Span | 2026-02-11 15:57 to 2026-06-09 15:49 UTC | 2026-05-24 03:46 to 2026-08-19 23:51 UTC |
| Rows | 564 | 556 |
Stored flag is_ai = 1 |
327 (58.0%) | 379 (68.2%) |
| Machine share, regrouped by user agent | 79.1% | 85.6% |
| Rows from OpenAI, Anthropic, Perplexity, Common Crawl or Cohere crawlers | 0 | 0 |
| Rows from Meta's and ByteDance's training crawlers | 3 | 53 |
| Rows from a language model product's fetcher | 0 | 1 (Google-NotebookLM, 2026-07-29) |
The two spans overlap from 2026-05-24 to 2026-06-09. The export holds 28 rows in those seventeen days (Applebot 11, browser 9, GoogleOther 6, Googlebot 2); by its own monthly counts the live table held 14 rows dated in May and 10 in the whole of June on 2026-08-20, so at most 24 rows can appear in both windows. Whether they are the same rows was not checked by identifier, and the export holding more June rows than the live table did for the same days is unresolved, alongside the 25-day gap below. If rows sit in both windows, window B's machine share moves by at most about half a point down, if the shared rows are all machines, or eight tenths of a point up, if they are browser rows; at most five browser rows can be shared, because window A's nine overlap browser rows all fall in June and window B holds only five human-flagged rows dated June. The machine majority in both windows does not move. Both datasets carry raw addresses and full user agent strings in their source; only aggregates are reproduced here or in the post.
Window A's last row precedes its export date by 25 days. Whether the beacon or the API was silent between 2026-06-09 and 2026-07-04, or the export was taken from an older snapshot, is not resolved by the file.
Window A: rows by family and by stored flag
| Family (from user agent) | Stored flag | Rows |
|---|---|---|
| Googlebot, smartphone string | AI | 192 |
| Applebot | AI | 120 |
| GoogleOther, smartphone string | human | 116 |
| GoogleOther, desktop string | human | 2 |
| Ordinary browser strings | human | 118 |
| Googlebot, desktop string | AI | 7 |
| Bingbot | AI | 5 |
| Meta-ExternalAgent | AI | 3 |
| curl | human | 1 |
The GoogleOther rows carry Google's documented smartphone string, the Nexus 5X Chrome user agent
ending in (compatible; GoogleOther); 107 of the 118 fell in March. Google documents a desktop
variant as well, which continues with Chrome and Safari identifiers after compatible; GoogleOther; two window A rows, both in March, carried it, and none in window B. GoogleOther is not in the pattern list,
so every one of those rows was stored as human. Regrouped, the corpus is 118 browser rows against
445 crawler rows and one command-line tool, a machine share of 79.1% against the 58.0% the flag reports.
The browser rows in window A
The 118 rows with ordinary browser strings come from 61 distinct addresses over 32 distinct pages. The single most frequent address accounts for 17 rows and the top five for 49 (42%). Platform mix: Windows 53, Mac 26, Linux 22, iOS 11, Android 6. By month: February 71, March 38, April 0, May 0, June (to the 9th) 9. The site launched in February, so the concentration in the first two months and in a few addresses is consistent with the author and early readers checking the new site; the two empty months are consistent either with no human visitors or with a beacon fault on the human tiers that did not affect the crawlers, and the data cannot separate those. No headless browser or automation user agent (HeadlessChrome, Playwright, Puppeteer, python-requests) appears in window A.
Window A crawler rows by month
| Month | Googlebot | Applebot | GoogleOther | Bingbot | Meta | Browser |
|---|---|---|---|---|---|---|
| 2026-02 | 85 | 15 | 2 | 0 | 0 | 71 |
| 2026-03 | 104 | 49 | 107 | 0 | 3 | 38 |
| 2026-04 | 4 | 29 | 2 | 0 | 0 | 0 |
| 2026-05 | 6 | 22 | 7 | 5 | 0 | 0 |
| 2026-06 (to 9th) | 0 | 5 | 0 | 0 | 0 | 9 |
The July figure, approached from the export
/api/top-pages ranks pages by rows with is_ai = 0 (api/src/routes/metrics.ts, ORDER BY views DESC, no secondary key). Ranked that way, the export's top ten pages sum to 112 human-flag rows, the July review's figure exactly, but four pages tie for tenth place with six human-flag rows each, so the tenth page, and with it the AI sum and the GoogleOther count, depends on the tie-break:
| Tenth page | Human-flag rows | AI-flag rows | GoogleOther among the human rows |
|---|---|---|---|
| 013-persona-testing | 112 | 102 | 32 |
| 014-behaviorisms-hidden-legacy-in-reinforcement-learning | 112 | 102 | 33 |
| 025-the-logistics-gap | 112 | 94 | 32 |
| 034-benchmarking-bullshit-detection | 112 | 92 | 35 |
The review saw 112 / 110 on 2026-07-25, 46 days after the export's last row and three weeks after the export itself. The eight to eighteen AI-flag rows the export cannot supply cannot be placed either: they may be later rows, a different tenth page, or a difference the two readings do not share. The matching human sum is a coincidence the export can neither confirm nor rule out, not an identification of the review's rows, which no longer exist. Ranked by total rows instead, the top ten sum to 96 / 145. Recount in export-recount-2026-09-01.json.
Verifying the spring window's crawler identities
A user agent string is a claim, and Google warns that its crawler strings are spoofed. Every crawler-labelled row in window A was checked on 2026-09-01 against the publisher's own verification method: Googlebot and GoogleOther addresses against the ranges Google publishes for its common crawlers (googlebot.json, generated 2026-09-01; Google files GoogleOther among them, and none of the addresses fell in the separate special-case list), Bingbot addresses against Microsoft's bingbot.json and by reverse DNS to search.msn.com with a forward lookup, Applebot addresses by reverse DNS to Apple's crawler hostnames with a forward lookup confirming the address, and Meta-ExternalAgent addresses against the routes announced by Meta's autonomous system; the first three are the checks Google, Microsoft and Apple each document, and Meta's pages document only its user agent strings. Aggregates only; no address was retained.
| Family | Rows | Distinct addresses | Addresses verified | Rows from verified addresses |
|---|---|---|---|---|
| Googlebot | 199 | 22 | 22 | 199 |
| GoogleOther | 118 | 23 | 23 | 118 |
| Applebot | 120 | 120 | 120 | 120 |
| Bingbot | 5 | 5 | 5 | 5 |
| Meta-ExternalAgent | 3 | 3 | 3 | 3 |
None of the 118 browser rows came from a Google range. The three Meta-ExternalAgent addresses fall inside the routes Meta's network announces (its autonomous system 32934, whose route objects were read from the RADB registry). Every crawler-labelled row in window A is therefore verified at the address level. Window B's addresses were not checked: the check needs a read of the live table that this investigation did not make, and the purge deletes each row ninety days after it was written, so the oldest summer rows are already gone and the newest can be checked until mid-November 2026. Until such a read is made, every identity in the summer window is self-declared.
Window B: the live table, reclassified
The live table was read on 2026-08-20 by read-only SELECT statements against the production database (every statement returned rows_written 0, changed_db false). All 556 retained rows, reclassified from user_agent rather
than from the stored flag:
| Class | Rows | Share |
|---|---|---|
| Conventional search indexers (Google, Apple, Bing) | 326 | 58.6% |
| Plausible human browsers | 80 | 14.4% |
| GoogleOther (Google non-search crawler) | 76 | 13.7% |
| Meta's and ByteDance's training crawlers (Meta-ExternalAgent, Bytespider) | 53 | 9.5% |
| HeadlessChrome (attributed to own automation, not matched to a job) | 18 | 3.2% |
| SEO and utility bots (AhrefsBot, quillbot) | 2 | 0.4% |
| OpenAI, Anthropic, Perplexity, Common Crawl and Cohere crawlers | 0 | 0.0% |
| Language model product fetcher (Google-NotebookLM) | 1 | 0.2% |
The 80 plausible-browser views come from some seventy distinct addresses across about thirty paths (the 2026-08-20 grouping recorded 81 rows from 72 addresses over 30 paths before the notebook row was moved out, and whether that row's address and path were unique among them was not recorded), which over the
window is a little under one plausible-browser row a day. The eighteen HeadlessChrome rows begin on
2026-08-08 and appear in no window A row; they are almost certainly the author's own browser sessions against the live site, though they were not matched to a specific job. The site's own
scheduled checks are not the source: scripts/check-invariants.sh and the sixteen checkers under
scripts/invariants/ are plain HTTP clients, none of which drives a browser, and the crontab holds
no other browser job against the production site.
Both windows agree on the point the post rests on. Every OpenAI, Anthropic, Perplexity, Common Crawl and Cohere agent the pattern list knows, GPTBot, ClaudeBot, Claude-Web, anthropic-ai, ChatGPT-User, PerplexityBot, CCBot and cohere-ai, returns exactly zero rows in both, by two extraction methods. The two training crawlers the list also knows, Meta-ExternalAgent and Bytespider, do appear: 3 rows in window A and 53 in window B. The one row in window B from a model product acting for a person is a different kind of visitor. Google documents Google-NotebookLM as
the former user agent token (supported until August 2026) of the Gemini Notebook fetcher, which
"requests individual URLs that Gemini Notebook users have provided as sources for their projects";
the current token is Google-GeminiNotebook. A read of the live table on 2026-08-30 (714 rows,
2026-06-01 to 2026-08-30) found three such rows in all, the first on 2026-07-29 and two after the
window B snapshot, on 2026-08-21 and 2026-08-23, against two posts. Google states that the general
technical properties of its crawlers apply to its user-triggered fetchers, and says nothing about
whether this one executes JavaScript. The three rows were written by POST /api/views, which
nothing the site ships calls except the page script; they are consistent with the fetcher rendering the page and equally with a direct call to the endpoint under that token, and the table cannot distinguish the two. No other user-triggered model fetcher appears in the 2026-08-30 read either: the replacement classifier, run over every distinct user agent in the table, yields 403 search crawler rows, 86
training crawler rows (Meta-ExternalAgent, Bytespider), 3 Gemini Notebook rows, 96 other machine
rows and 126 plausible browser rows, a machine share of 82.4% over a window that now overlaps window
B rather than replacing it.
The machine surfaces, probed
The absence above is a property of the beacon rather than of the agents, and the same property governs every surface the site offers machines. Each was fetched over public HTTPS on 2026-08-20 and the returned bytes searched for the view beacon. This is an inspection of returned bytes, not an exercised MCP session, a tool call or a before-and-after test of the counter; scripts those bytes link were not followed.
| Surface | Status | Bytes | Beacon present |
|---|---|---|---|
MCP server (mcp.ashitaorbis.com/mcp) |
406 to a bare GET | 116 | no |
| OpenAPI description | 200 | 7,368 | no |
| llms.txt at the raw tier | 200 | 1,784 | no |
| llms.txt at the apex domain | 404 | 13,627 | no |
| JSON Feed | 200 | 44,116 | no |
| RSS | 200 | 41,460 | no |
| API markdown for one post | 200 | 13,337 | no |
| Developer page (WebMCP surface) | 200 | 28,009 | no |
| Hosted GPT (chatgpt.com) | 200 | 1,044,320 | no |
| Agent write endpoint, GET side | 200 | 25,844 | no |
| Raw tier post, HTML | 200 | 25,274 | yes |
| Editorial tier post, HTML | 200 | 67,067 | yes |
Ten machine surfaces whose returned bytes carry no reference to the beacon, and two HTML tiers whose bytes do; the seven channels the developer page advertises are all among the ten. The write endpoint was fetched on its GET side only, since a POST would have written an activity row. The apex 404 is its own small finding, and it is
narrower than it first looks. The llms.txt proposal, at v2, permits the file "at the root path
/llms.txt of a website or at any subpath", so serving it from raw.ashitaorbis.com is in spec and
tier-2-astro/src/pages/index.astro:123, developers.astro:40 and
tier-3-nextjs/src/lib/surfaces.ts:60 all link that address. What is missing is a file at the
conventional path on the domain the site publishes as its own: ashitaorbis.com/llms.txt returns an
HTML error page, so a model that guesses the path rather than reading the front page finds nothing.
What the beacon does not collect, and what a zero cannot settle
Google documents that Googlebot renders pages in a headless Chromium and executes JavaScript, and
also warns that its renderer may skip requests that do not contribute to essential page content and
that client-side analytics may not fully represent its activity, so the beacon may have missed some of what
Googlebot rendered; the rows that exist record requests that reached the endpoint, not renderings; Apple documents Applebot as a crawler that may render content within a browser, under Safari-shaped
desktop and mobile user agents. Apple does not say every page is rendered, and the site's own data cannot say it either: the 120
Applebot rows in window A and 83 in window B were written by an endpoint only the page script calls, so each client either ran the beacon or called the endpoint directly, and the rows do not say which. Vercel's December 2024
study of crawler traffic on its network reported that none of the major AI crawlers rendered
JavaScript, naming OpenAI's GPTBot, ChatGPT-User and OAI-SearchBot, Anthropic's ClaudeBot,
Meta-ExternalAgent, Bytespider and PerplexityBot, while Googlebot and Applebot did. A crawler that
never runs the page script leaves no row unless something in its pipeline calls POST /api/views directly, so if Vercel's observation held for this site's traffic, the pattern list's GPTBot and ClaudeBot entries could have matched only a request that arrived at the API some other way. Two of the names the study lists wrote rows here anyway: Meta-ExternalAgent wrote 3 rows in window A and 49 in window B, Bytespider wrote 4 in window B, and every row in this table was written by POST /api/views, which only the page script calls. Either those two crawlers execute the script now, at least sometimes, or something in their pipelines calls the endpoint directly. The rows cannot separate those two readings, so they are not evidence against the study's rendering observation, which a direct call to the endpoint leaves untouched. Meta's documentation of its crawlers, read 2026-09-01, describes five agents and says nothing about JavaScript for any of them. The study is twenty months old, and the absence of the remaining families cannot be attributed to non-rendering on its authority alone, because the beacon cannot distinguish a crawler that never came from one that came and did not render. The pattern list is also incomplete on its own terms: it lacks GoogleOther,
OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, Amazonbot and DuckAssistBot (Google-Extended is a robots token with no user agent string of its own, so no list could match it), and it files Googlebot, Bingbot and Applebot, all
search index crawlers, under AI. It also lacks every one of Google's documented user-triggered
fetchers, the Gemini Notebook fetcher among them, and Google-Agent, which Google describes as
"used by agents hosted on Google infrastructure to navigate the web and perform actions upon user
request". Google does not document whether either executes JavaScript; the three Gemini Notebook
rows show only that requests under that token reached the beacon's endpoint, by rendering or by a
direct call, which is more than any GPTBot or ClaudeBot request has done in either window. The replacement classifier described in the companion post names both.
The purge, and why both windows are now files
api/src/routes/psyche.ts:1343-1345 deletes page_views rows older than 90 days as part of the
psyche retention sweep, on a daily 04:17 UTC trigger (api/wrangler.toml). The DELETE was written deliberately, as a retention trim for tables with no natural lifecycle (the code comment beside it reads "keep 90 days of analytics"); the schedule that runs it arrived five weeks later with the psyche retention work, and the change that added it does not mention page views. The dated chain, with commit dates in the author's local zone (UTC minus seven) and the trigger in UTC: the DELETE was written 2026-07-04, the cron trigger was added 2026-08-09, and an orphaned statement that aborted the sweep's first run before the pruning block was removed on 2026-08-10, so the first effective purge falls on or after 2026-08-11; the first successful run is recorded at 04:52:59 UTC on 2026-08-11 (api/wrangler.toml). The July review's 365 day figures therefore cannot be
recomputed, and every window longer than 90 days now returns the 90 day rows. Probed on
2026-08-20, top-pages at 90, 120, 180 and 365 days returns an identical 172 human / 295 AI.
The consequence for this post is that neither window survives in production. Window A survives because a database dump taken on 2026-07-04, ahead of that day's migration, still exists in api/backups/, a directory the repository's .gitignore excludes, so the file has never been committed and exists on one machine; window B survives because it was aggregated to family counts before the purge reached it, and the oldest of its rows left the table two days after that read. That dump is the only copy of the site's February to May audience history this investigation could find, it is one untracked file, and its name does not say what it is.
What the data cannot support
Five claims the post deliberately does not make. First, that no OpenAI, Anthropic or Perplexity agent has ever read the site: the beacon records nothing on its own from a fetch that does not run the page script, this investigation had no per-request record at the edge to consult, and the three Gemini Notebook rows are fetches a person initiated, not a model choosing what to read.
Second, that the browser rows are humans: an automated browser presenting a stock user agent is
indistinguishable in this table, and window B's eighteen HeadlessChrome rows show that such traffic
now reaches production. Third, that the two windows are strictly comparable: the flag's meaning did
not change, but the classifier's blind spots interact with whichever crawlers happened to be active,
and window B carries 49 Meta-ExternalAgent rows against window A's 3. Fourth, anything about how
often the markdown negotiation path, llms.txt, the feeds or the MCP server are read: the probe
above establishes that the bytes those surfaces return carry no reference to the beacon; the scriptless formats among them (markdown, plain text, JSON) cannot run a script and so leave no row, while every response that does link scripts was left unexercised: the two advertised interfaces (the developer page, 6 script tags, and the hosted GPT, 14) and the HTML 404 the apex serves for llms.txt (5). A script tag count says nothing about whether those tags are external, executable or inert, so their behaviour beyond the returned bytes is not known. That is a statement about the instrument and not a measurement of that traffic. A raw tier page does carry the beacon, but the table does not record the tier, so its reads cannot be separated from the editorial tier's. Fifth, that the summer rows' crawlers are who they say: every identity in both windows is a user agent string, the spring rows were checked against the publishers' address ranges and all of them hold, but the summer rows were not checked, and the purge removes each of them ninety days after it was written, so window B's identities are self-declared for as long as no check is made and unverifiable once the rows are gone.
Reproduction
Seven aggregate files are published beside this investigation under /investigations/data/audience-composition/: window A as page-views-export-2026-07-04-aggregates.json; window B as page-views-live-2026-08-20-aggregates.json, with the notebook fetcher correction and the 2026-08-30 census in page-views-live-2026-08-30-aggregates.json; the surface probe as machine-surface-probe-2026-08-20.json; the recount behind the overlap and top-ten figures as export-recount-2026-09-01.json; the spring window's address verification as window-a-address-verification-2026-09-01.json; and the counter's day-one payload as day-one-widget-payload-2026-08-30.json. None of them carries an address or a full user agent string. Each file keeps its measured counts as taken; where a file's own commentary was later found wrong (the 2026-08-20 file's non-overlapping-windows verdict, its dependent machine total and its capped-query gloss, the 2026-08-30 file's no-training-crawler line and its rendering remark, the address-verification file's purged-addresses note, the probe's cannot-ever-produce-a-row note), the file carries a dated correction field at the top that supersedes the commentary without altering a number.
Two family rules produced the tables, and both are published here in full. The spring rule, applied to the export in order (first match wins; case-insensitive substring of the user agent): contains googleother, GoogleOther (desktop if the string continues with Chrome and Safari identifiers after the token, smartphone otherwise); contains googlebot, Googlebot (smartphone if it contains Mobile); contains applebot, Applebot; contains bingbot, Bingbot; contains meta-externalagent, Meta-ExternalAgent; starts with curl, curl; otherwise browser. The 2026-08-30 census and the corrected window B split use the classifier the site now ships in its API (api/src/lib/audience.ts, answering at GET /api/audience): an ordered list of lower-case substrings in which the first family whose substring appears wins. Model agents: chatgpt-user, claude-user, perplexity-user, claude-web, meta-externalfetcher, google-gemininotebook or google-notebooklm, google-agent;. Model crawlers: gptbot, oai-searchbot, claudebot or anthropic-ai, claude-searchbot, perplexitybot, meta-externalagent, ccbot, bytespider, cohere-ai or cohere-training-data-crawler, amazonbot, duckassistbot, diffbot, youbot. Other machines: googleother, then Google's other documented fetchers (google-read-aloud, google-speakr, google-pinpoint, google-cws, feedfetcher-google, googlemessages, googleproducer, google-site-verification). Search crawlers: googlebot, bingbot, applebot, duckduckbot, yandexbot, baiduspider. Other machines again: headlesschrome, playwright, puppeteer, phantomjs; curl/, wget/, python-requests, python-urllib, go-http-client, axios/, node-fetch, undici, okhttp, java/, libwww; ahrefsbot; quillbot; and the generic tokens bot/, bot;, bot), crawler, spider, scraper, facebookexternalhit, twitterbot, linkedinbot, slackbot, discordbot, telegrambot, whatsapp. An empty user agent is an other machine; anything unmatched is a browser. The window B class table groups the families as: conventional search indexers, Googlebot, Applebot and Bingbot; training crawlers, Meta-ExternalAgent and Bytespider; HeadlessChrome; SEO and utility bots, AhrefsBot and QuillBot; the model product fetcher, Google-NotebookLM. The window B file records family counts rather than user agent strings, so the rule cannot be re-run on it; its 81 plausible browsers were grouped by hand before the shipped rule existed, and the one-row difference from the corrected 80 is the notebook token. The 2026-08-30 file records the shipped rule's output over every distinct user agent then in the live table, and GET /api/audience answers with the same rule over the live table today.
curl -s "https://api.ashitaorbis.com/api/top-pages?days=365&limit=50"
npx wrangler d1 execute ashitaorbis-db --remote --json \
--command "SELECT user_agent, is_ai, COUNT(*) FROM page_views GROUP BY 1,2 ORDER BY 3 DESC"
Run from applications/ashitaorbis/. The second command needs the site's own Cloudflare credentials and is recorded so that the query itself is public. Any live figure is valid only for a 90 day trailing window
and will drift, so the post binds its arithmetic to the aggregate files instead.