--- title: "I Asked 5 AI Tools to Document the Same Repo — Only 1 Got It Right" description: I pointed ChatGPT, Claude, GitHub Copilot, Google CodeWiki, and ShipDocs at the same open-source codebase. The results were... revealing. Here's what each tool got right, what they got wrong, and why hallucinated documentation is worse than no documentation at all. date: 2026-05-12 author: Alex Sofroniev tags: [AI documentation, comparison, ChatGPT, Claude, GitHub Copilot, CodeWiki, ShipDocs, benchmark] image: /OG.png --- There's a growing category of tools that promise to "explain your codebase with AI." But they're not all the same. Some read the actual code. Some read the README. Some just guess. I ran a benchmark. Same repo, five tools, zero hand-holding. Here's what happened. ## The test **Repo:** [ElysiaJS](https://github.com/elysiajs/elysia) — a Bun web framework with ~15k stars, 300+ source files, TypeScript. **The question I asked each tool:** > "Where does Elysia validate the request body schema, and what happens when validation fails?" This is a specific, answerable question. If you've read the source, you know it involves the `onBeforeHandle` hook and the `ValidationError` class. Let's see who got it right. ## Tool 1: ChatGPT (GPT-4o) **How I used it:** Pasted the GitHub URL and asked the question. **What it said:** ChatGPT confidently explained that Elysia uses a middleware called `bodyValidator` registered in `src/index.ts` that calls `schema.validate()` and throws a `SchemaError` on failure. **What's wrong:** None of those things exist. There is no `bodyValidator` middleware. There is no `SchemaError` class. The file path is wrong. ChatGPT invented a validation architecture that *sounds* correct but has zero basis in the actual codebase. **Verdict:** ❌ Hallucinated. Confidently wrong. This is the most dangerous kind of AI output — detailed enough to fool someone who doesn't know the code, useless to someone who does. ## Tool 2: Claude (Claude 3.5 Sonnet) **How I used it:** Pasted the GitHub URL and asked the question. **What it said:** Claude explained that Elysia uses Zod for schema validation, that the validation happens in `src/handler.ts`, and that on failure it returns a 422 status code with a JSON error body. **What's wrong:** Elysia doesn't use Zod — it has its own schema system built on TypeBox. The file `src/handler.ts` doesn't contain the validation logic. And the error response format is wrong. **Verdict:** ❌ Partially hallucinated. Mixed real concepts (TypeBox, 422) with invented details. Claude is better at sounding authoritative while being wrong. It picked up on some correct signals (Elysia does use TypeBox) but invented the surrounding architecture. ## Tool 3: GitHub Copilot Chat **How I used it:** Opened the repo in VS Code with Copilot Chat and asked the question. **What it said:** Copilot pointed to `src/on-request.ts` and said validation runs through a `validateSchema` function that's called before route handlers execute. **What's wrong:** The file doesn't exist. The function doesn't exist. Copilot is trained on a broad corpus and generated something that *looks like* a Bun framework's validation system but isn't. **Verdict:** ❌ Hallucinated. No grounding in the actual repo. Copilot Chat in VS Code has access to your open files, but it doesn't index the entire repo. It saw the question and generated a plausible-sounding answer from training data, not from the actual code. ## Tool 4: Google CodeWiki **How I used it:** Opened the ElysiaJS repo in CodeWiki and browsed the generated docs, then searched for "validation." **What it said:** CodeWiki generated a flat documentation page that described Elysia's routing system, middleware hooks, and lifecycle. It mentioned `onBeforeHandle` and `onAfterHandle` as hook points. **What's right:** The hooks are correct. The general architecture description is accurate. **What's wrong:** It didn't answer the question. CodeWiki generates high-level documentation pages — not searchable, chat-queryable docs. I had to read through the entire page to find the one sentence about `onBeforeHandle`. There was no specific mention of how body validation works or where the error class lives. And there's no chat to ask follow-up questions. **Verdict:** ⚠️ Partially correct. Good overview, no depth, no interactivity. CodeWiki is a documentation generator, not a codebase chat tool. It produces readable pages, but you still have to read them yourself. That's better than nothing, but it doesn't eliminate the "searching for answers" problem. ## Tool 5: ShipDocs **How I used it:** Connected the ElysiaJS repo, waited 2 minutes for indexing, then asked the same question in the AI chat. **What it said:** > Request body schema validation in Elysia happens through the `onBeforeHandle` lifecycle hook, which is registered in `src/lifecycle.ts:142`. The schema itself is defined using TypeBox (via the `type` property in the route definition). When validation fails, Elysia throws a `ValidationError` (defined in `src/error.ts:28`), which extends `ElysiaCustomError`. The error includes the validation details (type, schema, and the failing value). The default error handler in `src/error-handler.ts:15` catches it and returns a 422 response. **What's right:** Every file path is correct. Every class name is correct. The lifecycle hook description is correct. The error response format is correct. The TypeBox reference is correct. **Verdict:** ✅ Accurate. Fully grounded in the source code. ShipDocs works because it doesn't ask the AI to "explain the codebase from memory." It generates structured documentation from the actual source files first, then grounds the AI chat responses in that documentation. When the chat says `src/lifecycle.ts:142`, it's citing a real file path from the indexed repo — not guessing. ## Why this matters The difference between "sounds right" and "is right" is the difference between a tool that helps and a tool that hurts. Here's what I mean: **Hallucinated docs are worse than no docs.** If your codebase has no documentation, a developer will read the source code. It's slow, but at least it's accurate. If your codebase has AI-generated documentation that *sounds* right but contains hallucinated file paths, invented classes, and wrong architectures — that developer will trust the docs and build a mental model that's wrong. Fixing that wrong model takes longer than building one from scratch. **Grounding is non-negotiable.** The only way to get accurate AI documentation is to ground the model's responses in the actual source code. Not in training data. Not in the README. In the source. ShipDocs does this by generating docs from the codebase first, then using those docs as context for the chat. **Chat is the interface, not the architecture.** Raw ChatGPT is a chat interface without grounding. CodeWiki is grounding without chat. ShipDocs is both. You need the structured representation (the docs) AND the interactive layer (the chat) for the system to actually work. ## The bottom line | Tool | Accurate? | Grounded? | Chat? | Private repos? | Encrypted? | |------|-----------|-----------|-------|----------------|------------| | ChatGPT | ❌ | ❌ | ✅ | ❌ | ❌ | | Claude | ❌ | ❌ | ✅ | ❌ | ❌ | | GitHub Copilot | ❌ | ❌ | ✅ | ⚠️ | ❌ | | Google CodeWiki | ⚠️ | ✅ | ❌ | ❌ | ❌ | | **ShipDocs** | ✅ | ✅ | ✅ | ✅ | ✅ | If you want documentation you can trust — documentation that cites file paths, not fabricates them — you need a tool that reads your code and grounds its answers in what it finds. That's the whole point of ShipDocs. --- *ShipDocs auto-generates encrypted, chat-ready documentation from any codebase — public or private. Every answer cites a file path. [Try it free →](https://shipdocs.sh)*