# LLCrawler — LLM Visibility Analyzer (Full Documentation)

> LLCrawler (llcrawler.com) is a free tool that analyzes any website and scores how visible it is to LLM-powered search engines like ChatGPT, Claude, Perplexity, and Gemini. It produces an actionable report in 30 seconds with a score out of 100 and prioritized fixes.

## Overview

AI answer engines are replacing traditional search results. When a user asks ChatGPT "what is the best project management tool?" or Claude "recommend a local restaurant in Austin," those models draw answers from the websites they have crawled and understood. If your site is not prepared for LLM crawlers, you are invisible to this new wave of search.

LLCrawler measures exactly how visible your site is across six categories and tells you what to fix, in priority order.

## Scoring methodology

The total score is 100 points distributed across six categories:

### Structured data (15 points)
- JSON-LD detected on homepage: 6 points
- Relevant schema type present (Organization, Product, Article, FAQPage, LocalBusiness, WebSite): 6 points
- BreadcrumbList or SiteLinksSearchBox detected: 3 points

LLMs rely on schema.org to understand what your site is and who stands behind it. Without an Organization schema, ChatGPT has no structured way to describe your company.

### Content structure (15 points)
- Exactly one H1 on homepage: 3 points
- FAQ section detected: 6 points
- Lists or tables with data present: 3 points
- Extensive content (300+ words): 3 points

LLMs pull citable answers from clear hierarchy. A well-marked FAQ becomes a direct citation. A wall of undifferentiated text does not.

### Meta and social (10 points)
- Meta description present, 50-160 characters: 4 points
- Open Graph complete (og:title, og:description, og:image): 3 points
- Twitter card present: 3 points

These are the first signals crawlers read before processing the full HTML.

### Performance (15 points)
- Viewport meta tag present: 5 points
- Mobile Lighthouse score 70 or above: 10 points

AI crawlers have timeouts. A slow page means bots bail before they finish parsing.

### AI readiness (35 points)
- Valid /llms.txt file (H1 + sections): 5 points
- /llms.txt depth (3+ sections, 200+ words, links, blockquote description): 10 points
- /llms-full.txt present with 500+ words of comprehensive documentation: 5 points
- GPTBot and ClaudeBot allowed in robots.txt: 5 points
- Content visible without JavaScript (200+ words): 5 points
- 80% or more of images have alt text: 5 points

LLMs do not run JavaScript and they obey robots.txt. Block their bots or render only with JS, and you are invisible. A rich /llms.txt and /llms-full.txt tell AI models exactly what your site does and why it matters.

### Sitemap (10 points)
- Sitemap found: 4 points
- Declared in robots.txt: 2 points
- Contains URLs: 2 points
- All URLs on same host: 1 point
- 50%+ URLs have lastmod dates: 1 point

A clean sitemap is the fastest way to tell crawlers what to index and when it changed.

### Bonus section (not scored)

In addition to the 100-point score, LLCrawler evaluates aspirational signals that can further boost LLM visibility:

- **Educational content** — Does the site have blog posts, guides, or how-to articles? LLMs recommend sites that are the authoritative reference on a topic.
- **External authority** — Backlinks and citations from other sites increase LLM trust. This cannot be auto-detected but is always recommended.
- **FAQ depth** — Having 5 or more detailed FAQ items in structured data provides rich content for LLMs to cite directly.

## How it works

1. User pastes a URL at [llcrawler.com](https://llcrawler.com)
2. LLCrawler crawls the homepage and extracts HTML signals (meta tags, structured data, headings, content, robots.txt, llms.txt, sitemap.xml)
3. Five analyzers run in parallel on the crawled data
4. A PageSpeed Insights audit runs asynchronously for performance scoring
5. The report is generated with a score out of 100 and issues grouped by severity (critical, warning, ok)
6. The report URL is public and shareable

## Who should use LLCrawler

**Founders and indie hackers** launching SaaS products who need ChatGPT and Claude to recommend them when users ask for solutions in their category.

**SEO specialists** adapting their work to the new AI search paradigm where traditional ranking factors alone are not sufficient.

**Content and marketing teams** who want their articles, guides, and documentation cited by LLMs when users ask related questions.

**Local businesses and e-commerce shops** that depend on appearing in AI assistant recommendations when users ask "where can I buy X" or "best Y near me."

## Key features

- **Free analysis** — No credit card, no signup required for anonymous analyses
- **Bilingual** — Full English and Spanish interface and reports
- **Shareable reports** — Every completed analysis has a permanent public URL
- **Embeddable badge** — shields.io-style SVG badge with Markdown and HTML snippets for GitHub READMEs
- **Auto-generated llms.txt** — If your site is missing /llms.txt, LLCrawler generates one from your crawl data, ready to copy
- **Contextual recommendations** — Tool suggestions tailored to the specific issues detected in each analysis
- **User accounts** — Free accounts save analysis history and enable weekly monitoring per domain

## Frequently asked questions

**What is LLM visibility?**
It is your site's ability to be found, understood, and cited by models like ChatGPT, Claude, Perplexity, and Gemini when they answer user questions.

**Is it really free?**
Yes. You can analyze sites without creating an account, with a daily limit. A free account gives you more analyses and saves your weekly history.

**What is the llms.txt file and why does it matter?**
It is a Markdown file at the root of your site that tells LLMs which content is most important and how to understand your product. It is to AI models what robots.txt was to traditional SEO crawlers.

**How long does the analysis take?**
About 30 seconds. The tool crawls the homepage, runs five analyzers in parallel, and generates the report.

**Does it analyze the whole site or only the homepage?**
Currently the homepage only, which is where most critical signals live (schemas, meta tags, llms.txt, robots.txt). Multi-page crawling is on the roadmap.

**What do I do with the report?**
Each issue includes an explanation and a concrete fix. Start with the critical issues, then improvements. Share the report link with your team.

**Can I embed the score in my GitHub README?**
Yes. Every completed report includes a shields.io-style badge with Markdown and HTML snippets ready to copy.

## Technical details

- Built with Ruby on Rails 8.1
- Analysis powered by Nokogiri for HTML parsing, HTTParty for HTTP requests
- Performance scoring via Google PageSpeed Insights API
- Background job processing via Sidekiq
- Deployed on Docker with PostgreSQL and Redis

## Links

- [Homepage](https://llcrawler.com)
- [Blog](https://llcrawler.com/blog)
- [Privacy policy](https://llcrawler.com/privacy)
- [Terms of service](https://llcrawler.com/terms)
- [Summary](https://llcrawler.com/llms.txt)

## Contact

For issues, feedback, or partnerships: support@llcrawler.com