Tooru
Scoring Methodology
Everything Tooru measures, how it measures it, and how many points each check is worth. Scores come from mechanically analysing the HTML, robots.txt, structured data and agent-facing files of each page. No human judgement is involved.
Principles
The diagnostic is designed around three principles.
- Objectivity
- Every check is derived from a published technical standard or an official vendor document. The result is a mechanical reading of the page as served; it does not include subjective judgement.
- Reproducibility
- The same URL measured under the same conditions yields the same score. Tooru does not cache results: every run fetches the page again, so a change on your site shows up in the next run and is kept as history for Before/After comparison.
- Transparency
- All 40 scored checks, their points, judgement rules and referenced standards are published on this page, together with the 14 reference-only items that are recorded without points.
Scoring system
The total score is 5 categories × 20 points = 100 points. Each page (URL) is scored independently; domain-level resources such as robots.txt, sitemap.xml, llms.txt and /.well-known/ files are fetched once per origin.
| Question | Points | Checks | Standards |
|---|---|---|---|
| 1. Can AI find you? | 19 | 11 | RFC 9309: Robots Exclusion Protocol |
| 2. Can AI read your pages? | 29 | 11 | HTML Living Standard (WHATWG) |
| 3. Does AI understand your business? | 38 | 12 | Schema.org |
| 4. Will AI use you in its answers? | 14 | 6 | GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024) |
| Total | 100 | 40 |
In addition to the 40 scored checks, 14 reference-only items (6 bots and 3 agent-layer declarations) are recorded with zero points. They never change a score.
Ranks
Each page receives one of five ranks from its total score.
| Rank | Score | Meaning |
|---|---|---|
| S | 90–100 | Very high conformance to the standards; the major AI agents can read and reach the page without friction. |
| A | 70–89 | The core technical foundations are in place; strengthening one or two categories will lift the score further. |
| B | 50–69 | Basic support exists, but some checks still stop AI agents from retrieving information accurately. |
| C | 30–49 | Conformance is low in several categories; AI agents have difficulty retrieving information. |
| D | 0–29 | Conformance is insufficient; the major AI agents cannot retrieve accurate information from the page. |
A site’s overall rank is derived from the average of the latest score of every URL diagnosed under that site (rounded to one decimal place), applied to the same thresholds.
Checks by question
1. Can AI find you? (19 Points)
Whether search-type AI crawlers can reach the site and find their way around it: the sitemap, the canonical URL, machine-readable declarations under /.well-known/, feeds, llms.txt and the Content-Signal policy. If a page cannot be found, nothing else on it matters.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 1-1 | /.well-known/ protocol discovery (ui.methodology.botTag.llm) | 3 | Fetch and content validation | Three declarations are fetched and validated, not merely detected: MCP Server Card (/.well-known/mcp/server-card.json, /.well-known/mcp.json or /.well-known/mcp; a JSON object is valid), OAuth Protected Resource (/.well-known/oauth-protected-resource, RFC 9728; resource required) and API Catalog (/.well-known/api-catalog, RFC 9727; a non-empty linkset required). Two or more valid = 3; one valid = 2; only a link to an MCP configuration in the HTML = 1; files that exist but are empty, invalid JSON or missing required fields score 0 with a warning; nothing published = 0 without penalty. |
| 1-2 | sitemap.xml (ui.methodology.botTag.crawler) | 2 | Fetch and validate | Fetched and contains a urlset or sitemapindex element with loc entries = 2; one of the two present, or the XML is incomplete = 1; cannot be fetched (an HTML response counts as missing) = 0. |
| 1-3 | Canonical URL (ui.methodology.botTag.content) | 2 | link[rel=canonical] | A link[rel="canonical"] with an href = 2; missing = 0. |
| 1-4 | OAI-SearchBot allowed (ui.methodology.botTag.crawler) | 2 | robots.txt parsing | OpenAI’s search crawler (ChatGPT search) is not disallowed. Blocking it removes the site from ChatGPT search results. Allowed = 2; disallowed = 0. |
| 1-5 | PerplexityBot allowed (ui.methodology.botTag.crawler) | 2 | robots.txt parsing | Perplexity’s search crawler is not disallowed. Allowed = 2; disallowed = 0. |
| 1-6 | Claude-SearchBot allowed (ui.methodology.botTag.crawler) | 1 | robots.txt parsing | Anthropic’s search crawler is not disallowed. Blocking it removes the site from Claude’s search citations. Allowed = 1; disallowed = 0. |
| 1-7 | RSS / Atom feed (ui.methodology.botTag.llm) | 2 | Feed link detection | A link element of type application/rss+xml or application/atom+xml, or a rel="alternate" link whose type mentions rss, atom or xml = 2; none = 0. |
| 1-8 | /llms.txt present (ui.methodology.botTag.llm) | 2 | File fetch | /llms.txt is fetched as text (an HTML response counts as missing). Present = 4; present but under 10 characters = 2; missing = 0. |
| 1-9 | llms.txt spec compliance (ui.methodology.botTag.llm) | 1 | Format validation | Five llmstxt.org conventions are counted: a “# ” title line, a “> ” summary, a “## ” section, a Markdown link or URL, and more than 200 characters of content. Points = number met, capped at 4. Not applicable when llms.txt is missing. |
| 1-10 | /llms-full.txt present (ui.methodology.botTag.llm) | 1 | File fetch | Present = 3; present but under 100 characters = 1; missing = 0. |
| 1-11 | Content-Signal (ui.methodology.botTag.crawler) | 1 | robots.txt parsing | robots.txt contains a Content-Signal line that names search, ai-input or ai-train = 1. No line = 0; a line with unrecognised values only = 0 (warning). |
Reference-only items (not scored)
Recorded so the report can show them, but worth zero points. Reference bots are judged like scored bots (allowed / blocked). Agent-layer declarations are judged in three states: valid, malformed (present but not usable) and absent. Reference bots recorded without points: ChatGPT-User, Googlebot, bingbot, Perplexity-User, Google-Agent, Applebot. User-initiated fetchers (ChatGPT-User, Perplexity-User, Google-Agent) act only on a person’s request and are outside robots.txt control. Googlebot, bingbot and Applebot are the search engines themselves; blocking them removes the site from those search results, so the state is recorded rather than scored.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| — | GPTBot allowed (ui.methodology.botTag.crawler) | Info | robots.txt parsing | OpenAI’s training crawler is not disallowed for the page. Allowed or no rule = 2; disallowed = 0. |
| — | ClaudeBot allowed (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Anthropic’s training crawler is not disallowed. Allowed = 1; disallowed = 0. |
| — | Google-Extended allowed (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Google’s token for Gemini training is not disallowed. Blocking it does not affect Google Search. Allowed = 2; disallowed = 0. |
| — | Applebot-Extended allowed (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Apple’s token for Apple Intelligence training is not disallowed. Allowed = 1; disallowed = 0. |
| — | Other AI crawlers (ui.methodology.botTag.crawler) | Info | robots.txt parsing | None of Meta-ExternalAgent, Amazonbot, Bytespider, CCBot, cohere-ai, Google-CloudVertexBot, DeepSeekBot is disallowed = 1; one or more disallowed = 0. Judged as a group because blocking any of them is a policy choice rather than a defect. |
| — | Googlebot (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Google Search itself. Blocking it removes the site from Google Search entirely, so the state is recorded for reference rather than scored. |
| — | bingbot (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Bing Search and Microsoft Copilot share this crawler and cannot be separated. Recorded for reference. |
| — | Applebot (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Apple’s search crawler (Siri, Spotlight, Safari). Recorded for reference; training use is controlled separately through Applebot-Extended. |
2. Can AI read your pages? (29 Points)
Whether an AI that arrives on a person’s behalf can get in and read the page: robots.txt, the user-initiated fetcher, and whether the HTML the server returns carries the content at all (agents do not run JavaScript). Also the heading structure, landmarks, image alternatives, text-to-markup ratio, document language, Markdown negotiation and response time.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 2-1 | JavaScript dependence (ui.methodology.botTag.readability) | 4 | Initial HTML analysis | Whether the main content is present in the HTML the server returns, before any script runs. Body text over 100 characters with fewer than 10 external scripts = 4, fewer than 20 = 3; content present (or a non-empty noscript fallback) = 2; body text over 20 characters = 1; otherwise 0. |
| 2-2 | Heading hierarchy (ui.methodology.botTag.readability) | 4 | HTML parsing | No h1 = 0. Exactly one h1 = 2 (more than one = 1); an h2 present = +1; h2 or h3 present = +1. If any heading skips a level (e.g. h2 → h4) the score is capped at 2. |
| 2-3 | Landmark elements (ui.methodology.botTag.readability) | 4 | HTML parsing | Each of main, nav, header and footer present = 0.8 (max 3.2); an article or section = +0.8. Rounded to the nearest whole point. |
| 2-4 | Image alt coverage (ui.methodology.botTag.readability) | 3 | HTML parsing | Share of img elements with a non-empty alt attribute. 90% or more = 3, 70% = 2, 50% = 1, below = 0. A page with no images scores 3. |
| 2-5 | Text-to-HTML ratio (ui.methodology.botTag.readability) | 3 | HTML parsing | Length of the visible text (script, style and noscript removed) divided by the length of the whole HTML. 25% or more = 3, 15% = 2, 10% = 1, below = 0. |
| 2-6 | robots.txt (ui.methodology.botTag.crawler) | 2 | Fetch and parse (RFC 9309) | Fetched and at least 5 characters long = 2; present but shorter = 1; cannot be fetched = 0. |
| 2-7 | Language attribute (ui.methodology.botTag.readability) | 2 | HTML parsing | The html element has a non-empty lang attribute = 2; missing = 0. |
| 2-8 | Markdown content negotiation (ui.methodology.botTag.llm) | 2 | Accept header probe | The page is requested with Accept: text/markdown. A text/markdown response = 2; HTML = 0. Not applicable when the probe cannot be judged. |
| 2-9 | Public API reference (ui.methodology.botTag.llm) | 2 | Link and text analysis | A link to swagger, openapi or api-doc, or visible text mentioning Swagger UI, OpenAPI spec or API document = +1. A link to an /api or /graphql URL = +1 (script contents are ignored to avoid false positives). Not applicable (no penalty) when neither is found. |
| 2-10 | Server response time (ui.methodology.botTag.crawler) | 2 | HTTP timing | Time to fetch the page HTML, measured from Tooru’s crawler. Under 1 second = 2; up to 3 seconds = 1; longer = 0. |
| 2-11 | Claude-User allowed (ui.methodology.botTag.crawler) | 1 | robots.txt parsing | Anthropic’s user-initiated fetcher (Claude visiting a page on a user’s request) is not disallowed. Allowed = 1; disallowed = 0. |
Reference-only items (not scored)
Recorded so the report can show them, but worth zero points. Reference bots are judged like scored bots (allowed / blocked). Agent-layer declarations are judged in three states: valid, malformed (present but not usable) and absent.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| — | ChatGPT-User (ui.methodology.botTag.crawler) | Info | robots.txt parsing | OpenAI’s user-initiated fetcher. It acts only on an explicit user request and does not honour robots.txt, so the allow/disallow state is recorded but cannot be acted on. |
| — | Perplexity-User (ui.methodology.botTag.crawler) | Info | robots.txt parsing | Perplexity’s user-initiated fetcher. Acts only on an explicit user request; robots.txt does not control it. Recorded for reference. |
| — | Google-Agent (ui.methodology.botTag.crawler) | Info | robots.txt parsing | The user agent Gemini Agent and similar tools use when visiting a site on a user’s instruction (added to Google’s crawler list in March 2026). robots.txt does not control it. Recorded for reference. |
| — | WebMCP Declarative API (ui.methodology.botTag.llm) | Info | HTML parsing | form elements carrying both toolname and tooldescription (W3C WebML Community Group draft; Chrome 149+ origin trial). Both attributes non-empty on at least one form = valid; only one of the two = malformed (the tool is not registered); none = absent. The imperative JavaScript API is not judged because the crawler does not execute scripts. |
| — | A2A Agent Card (ui.methodology.botTag.llm) | Info | Fetch and content validation | /.well-known/agent-card.json (A2A Protocol 1.0). A JSON object with a non-empty name and either url or a non-empty interfaces array = valid; fetched but not meeting that = malformed; not published = absent. |
| — | UCP profile (ui.methodology.botTag.llm) | Info | Fetch and content validation | /.well-known/ucp (Universal Commerce Protocol, 2026-08-25 release). A ucp object with version in YYYY-MM-DD form, services and payment_handlers = valid; fetched but not meeting that = malformed; not published = absent. The keys array is not required at this stage. |
3. Does AI understand your business? (38 Points)
Whether an AI can work out what the business is: what you offer, who it is for and where you operate, stated in the opening of the page, plus the structured data that identifies the organisation — address, contact, an external reference — and the Schema.org types, property values and Open Graph metadata.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 3-1 | JSON-LD present (ui.methodology.botTag.structuredData) | 5 | HTML parsing | Parseable script[type="application/ld+json"] blocks: two or more = 5, one = 3, none = 0. |
| 3-2 | Schema.org types (ui.methodology.botTag.structuredData) | 5 | JSON-LD parsing | Number of distinct recognised types (Organization, WebSite, WebPage, LocalBusiness, Person, Article, BlogPosting, Product, FAQPage, BreadcrumbList, Event, Service, HowTo, Recipe, Course, ItemList, CollectionPage, SearchAction; entities inside @graph are expanded). Four or more = 5, three = 4, two = 3, one = 2, none = 0. |
| 3-3 | Property value format (ui.methodology.botTag.structuredData) | 3 | JSON-LD value validation | Format of the values present: datePublished, dateModified, startDate, endDate and uploadDate must be ISO 8601; url, image and logo must be absolute URLs (image and logo may be ImageObject); inLanguage must be a BCP 47 tag; headline has a length limit; author must be structured; publisher.logo is checked too. Valid share 95% or more = 3, 80% = 2, 50% = 1, below = 0. Not applicable when no checked value is present. |
| 3-4 | Entity connections (ui.methodology.botTag.structuredData) | 3 | JSON-LD parsing | @graph used = +1. An @id reference that resolves to another entity’s @id = +2; @id present without cross-references = +1. |
| 3-5 | Organization identifiable (ui.methodology.botTag.structuredData) | 3 | JSON-LD validation | One point each, on the Organization or LocalBusiness entity, for a postal address (addressLocality or addressRegion), a telephone or contactPoint, and at least one sameAs reference to an external profile. Not applicable (no penalty) when neither type is present. This differs from required properties: that check asks whether a type’s required fields are filled, this one asks whether the organisation can be told apart from a company with a similar name. |
| 3-6 | Required properties (ui.methodology.botTag.structuredData) | 2 | JSON-LD parsing | Fill rate of the required properties per type: Organization and WebSite need name and url; WebPage, Product and Person need name; Article and BlogPosting need headline, image, datePublished and author; FAQPage needs mainEntity; BreadcrumbList needs itemListElement; LocalBusiness needs name and address. 90% or more = 2, 50% = 1, below = 0. Not applicable (no penalty) when none of these types is present. |
| 3-7 | Open Graph metadata (ui.methodology.botTag.structuredData) | 2 | Meta tag analysis | og:title, og:description, og:image and og:url all present = 2; two or more (or one plus twitter:card) = 1; otherwise 0. |
| 3-8 | Business stated up front (ui.methodology.botTag.content) | 3 | Title, description, h1 and first 300 characters | One point each for what you offer, who it is for, and where you operate, found in the title, meta description, h1 or the opening of the body text. An AI reads the opening first; if the business is not clear there, structured data cannot make up for it. |
| 3-9 | Title (ui.methodology.botTag.content) | 3 | Meta tag analysis | A title element is present = 1; 10–60 characters = +1; not a generic word (home, index, untitled, welcome, ホーム, トップ) and longer than 5 characters = +1. Missing = 0. |
| 3-10 | Meta description (ui.methodology.botTag.content) | 3 | Meta tag analysis | meta[name="description"] present = 1; 20 characters or more = +1; 50–160 characters = +1. Missing = 0. |
| 3-11 | Information structure (ui.methodology.botTag.content) | 3 | Content structure analysis | Share of h2/h3 headings directly followed by a body block (p, ul, ol, div, table or dl) with more than 20 characters. 70% or more = 3; 40% = 2; two or more headings otherwise = 1; no h2/h3 = 0. |
| 3-12 | Internal links (ui.methodology.botTag.content) | 3 | Link analysis | Anchors pointing to the same host, a relative path or a fragment. 10 or more = 3; 5 or more = 2; 1 or more = 1; none = 0. |
4. Will AI use you in its answers? (14 Points)
Whether the page gives an AI grounds to use it in an answer: concrete figures with units, cited sources and quotations, content shaped like answers (questions, definitions, steps, comparisons), author and dates, and how recently it was updated. Research on generative engines (Aggarwal et al., KDD 2024) found these are what raise how often a source is cited.
| # | Check | Points | Method | Judgement |
|---|---|---|---|---|
| 4-1 | Concrete figures (ui.methodology.botTag.content) | 3 | Body text analysis | Count of figures with a unit in the visible body text (currency, years, months, days, items, companies, people, %, kg, mm, times, "or more"). 10 or more = 3, 5 = 2, 1 = 1, none = 0. Proper nouns are not counted. Research on generative engines found that adding verifiable statistics raised citation rates by 30–40%. |
| 4-2 | FAQ / Q&A content (ui.methodology.botTag.content) | 3 | Pattern matching | FAQPage structured data = +2; details/summary elements = +1; a heading containing FAQ, よくある質問, Q&A or 質問と回答 = +1 when no FAQPage schema is present. Capped at 3. |
| 4-3 | Author and dates (ui.methodology.botTag.content) | 3 | Metadata and body analysis | Author (meta author, JSON-LD author or rel="author") = +1; publication date (datePublished, a time element with datetime, or article:published_time) = +1; modification date (dateModified or article:modified_time) = +1. |
| 4-4 | Shaped like answers (ui.methodology.botTag.content) | 2 | HTML structure analysis | One point for two or more question-style h2–h4 headings (a question mark, or words like how, difference, comparison, steps), and one point for a definition list, an ordered list of three or more steps, or a table with two or more columns. These blocks can be lifted straight into an AI answer. |
| 4-5 | Sources and quotations (ui.methodology.botTag.content) | 2 | Link and markup analysis | One point for an external link whose text or surrounding sentence attributes a source (source, reference, according to), and one point for a blockquote or cite element. Pages that show their sources are treated as more reliable. |
| 4-6 | Recently updated (ui.methodology.botTag.content) | 1 | Date extraction | The most recent of dateModified in the structured data, a time element with a datetime attribute, or article:modified_time. Within 12 months of the diagnosis = 1, older = 0. Not applicable when no date can be read — the author and dates check already covers a missing date, so it is not counted twice. |
AI agent readiness summary
Every report opens with a summary that answers “can AI agents reach this page, and can they find their way around?”. It is derived from the checks above and does not add or change points.
- Verdict
- Blocked — an Agent-type or Search-type bot is disallowed in robots.txt, or robots.txt cannot be read. Nothing else matters until agents can get in.
- Partial — agents can reach the page, but the discovery layer has gaps: llms.txt missing, no /.well-known/ declaration, or no Markdown content negotiation.
- Ready — agents can reach the page and find guidance.
Blocking Training-type bots does not lower the verdict. Declining AI training is a legitimate choice and is shown as such.
- First thing to fix
- The summary names exactly one blocker, in the order an agent meets them: an Agent-type bot blocked → robots.txt broken → a Search-type bot blocked → llms.txt missing → no /.well-known/ declaration → no Markdown negotiation.
- Reachability by bot type
- Reach is shown per bot group (Search, Training, Agent) from the individual bot checks. Bots that ignore robots.txt are marked so the reader knows the setting cannot control them.
Standards referenced
Standards and official documents the checks are derived from.
- 1. Can AI find you?
- 2. Can AI read your pages?
- 3. Does AI understand your business?
- 4. Will AI use you in its answers?
Measurement constraints
What the diagnostic can and cannot see.
- What is measured
- Only pages reachable over public HTTPS are measured. Pages behind Basic authentication, IP restrictions or a login cannot be diagnosed.
- The crawler reads the HTML the server returns and does not execute JavaScript. Content rendered only by scripts is invisible to it, which is what most AI crawlers also see.
- Each check is judged on the page as served to Tooru’s crawler (User-Agent “TooruBot”). Pages that serve different content to different clients may score differently from what a person sees.
- Accuracy
- robots.txt is interpreted with an RFC 9309-compliant parser. It shows what the file declares, not what the server or a CDN actually enforces.
- Edge-level bot management (for example CDN rules that challenge or block AI agents) is not visible in robots.txt. A page can allow a bot in robots.txt and still block it at the edge.
- Structured-data “appropriateness” is a mechanical reading against Schema.org type definitions, not a judgement of business fit.
- Some content-quality checks (information structure, FAQ detection, author and date) use pattern matching and can miss unconventional markup.
- Timing
- Results are not cached. Every run fetches the page again at the time of the job and consumes credits per page.
- Server response time depends on network conditions at the moment of measurement and can vary between runs.
- Large jobs are processed in chunks; pages within one job may be measured minutes apart.
- What the score means
- The score measures conformance to published technical standards. It is an indicator, not a guarantee of business outcomes.
- How each AI platform retrieves, ranks and cites content is decided by that platform and is independent of this methodology.
- Reference-only items never affect the score. They are recorded so the report can show emerging standards before they are widespread enough to score.
Version history
| Version | Date | Changes |
|---|---|---|
| v2.0 | 2026-09-08 | First Tooru release. Inherits the 39 checks and points of the untype.jp AI diagnostic (V1) methodology v2.3 (2026-06-05) unchanged, adds three reference-only agent-layer items (WebMCP Declarative API, A2A Agent Card, UCP profile) and publishes the rules of the AI agent readiness summary. The V1 methodology is frozen; later changes are made here. |
The methodology is revised as AI agent standards evolve. Changes are announced on this page with a new version number. Scored items and points change only in a versioned release so that historical scores stay comparable.
Feedback
Corrections, requests to support a new standard and opinions on point allocation are welcome at contact@untype.jp. Please mention “scoring methodology” in the subject.