When you submit a prompt to an LLM, the inference engine executes 2 distinct phases: the Prefill Phase and the Decoding Phase.

Phase 1: The Prefill Phase (Processing the Context)

The prefill phase occurs the instant you hit “Send”. Its objective is to ingest your entire prompt, calculate the semantic relationships between all your input words, and prepare the model to write the first token of its response.

Step 1: Tokenization and Embedding Tensor Construction

Your raw text prompt is split into standard units called tokens. Each token corresponds to a specific ID inside the model’s vocabulary matrix.

• These token IDs are looked up in a massive weight matrix known as the Embedding Layer.

• The embedding layer converts each token ID into a dense mathematical vector of continuous numbers (often 4096 or 8192 dimensions deep).

• Positional Encodings are mathematically added directly using these vectors. This ensures the model retains structural context, understanding the exact difference between “The dog bit the cat” and the “The cat bit the dog”.

Step 2: Parallel Matrix Computation

Unlike the output phase, the prefill phase is highly parallelized. Because the entire prompt is available upfront, the graphics processing unit (GPU) processes all input tokens simultaneously. The sequence of embedding vectors is passed into the first Transformer layer as a single massive matrix

Step 3: Calculating Queries, Keys, and Values (Q,K,V)

Inside every attention head of Transformer layer, the token matrix is multiplied by 3 distinct learned weight matrices to produce 3 new matrices:

• Queries (Q): What a token is actively searching for in the rest of the sentence

• Keys (K): What information a token contains that might be relevant to other tokens.

• Values (V): The actual semantic content of the token.

Step 4: The KV Cache Initialization (The Memory Anchor)

This is the single most critical performance optimization in modern LLM inferencing. During the prefill phase, the calculated Key (K) and Value (V) matrices for all your prompt tokens are computed once and stored directly inside the GPU’s high-bandwidth memory. This storage space is called the KV Cache. By saving these values now, the model avoids recalculating the historical context for every single word it generates later.

Phase 2: The Decoding Phase (The Generation Loop)

Step 5: Masked Multi-Head Attention

To generate the very first new token, the model executes the standard self-attention equation across the Q,K and V matrices:

However, during this phase, the attention mechanism is masked. Because it is generating text chronologically, a token is mathematically forbidden from looking at any information that supposedly comes after it. It can only compute scores against the historical tokens stored inside the KV Cache.

Step 6: The Feed-Forward Network Processing

The output matrix from attention mechanism is passed through a sequence of Feed-Forward Network layers. While the attention layers are designed to map the spatial relationships between tokens, the FFN layer acts as the model’s static factual database, extracting semantic facts based on the mapped context.

Step 7: The Un-embedding Layer

The final hidden vector emerging from the last Transformer layer represents the mathematical essence of the next token. To translate this back into vocabulary words, it is multiplied by the inverse of the embedding matrix (the un-embedding layer).

• This operation outputs a sequence of raw numerical values called logits.

• There is exactly one logit score for every single word/token in the model’s dictionary.

Step 8: Softmax Transformation and Sampling Filters

The raw logits are passed through a Softmax function to turn them into clean percentages. If your model vocabulary is 32,000 words, you now have 32,000 distinct probability scores that altogether sum up to 1.0. The inference engine then applies your generation parameters:

• Temperature: Divides the logits before Softmax. A lower temperature crushes lower-probability options, rendering the model predictable. A higher temperature flattens the distribution, raising the probability of unusual word choices.

• Top-P (Nucleus Sampling): Sorts the tokens by probability and discards everything outside the top cumulative percentage pool (e.g., the top 90% of probability weight), eliminating chaotic outliers entirely.

Step 9: Autoregressive Loop Update

The final winning token is selected, converted back into text via the de-tokenizer, and streamed to your user interface. Crucially, the Key and Value matrices for this brand-new token are calculated and immediately appended to the existing KV Cache. The model then restarts Phase 2, feeding only this single new token back into the input layer while referencing the expanded KV cache for historical memory.

The complete diagram is shown below:

It always starts the same way. You open a fresh chat, explain your project, and the AI splits out flawless code. Encouraged, you keep chatting. You add features, fix bugs, and alter requirements. But by Prompt 30, something breaks. The AI begins forgetting core constraints, hallucinating functions and generating messy, cyclical code.

This isn’t an execution problem; it’s a context fatigue problem. When an LLM’s context window is crammed full of conversational chatter, tangential debugging loops, and shifting requirements, its cognitive capability degrades.

To solve this, researchers at Google introduced a new framework: The Elephant-Goldfish Model.

What is the Elephant-Goldfish Model ?

Introduced in the foundational Google Research paper, Elephants, Goldfish and New Golden Age of Engineering, this framework completely flips the traditional developer-AI relationship.

Instead of treating the AI as an on-demand code monkey, the Elephant-Goldfish model forces the developer into the role of an Architect. It splits your interactions into 2 distinct phases, handled by 2 completely different mental modes.

1. The Elephant:

Like the proverb says, an elephant never forgets. In this phase, you use a long-lived continuous AI session. Together, you and the Elephant brainstorm features, challenge architectural assumptions, and map out edge cases. The ultimate goal of Elephant is to not to write code, but to draft a flawless, highly detailed Design Document (PRD or functional specification).

2. The Goldfish:

A goldfish has no long-term memory. In this phase, you spin up brand-new, completely stateless AI sessions. These “Goldfish” agents have 0 historical knowledge of your previous conversations. Their sole job is to act a cold-reader to stress-test the Elephant’s design document.

The 3 Gate Protocol: How it Works

The core rule of Elephant-Goldfish model is simple: No production code is written until the design document passes the Goldfish Gates.

Before execution, the design document is fed to isolated Goldfish agents through a three-stage lifecycle:

Gate 1: The Comprehension Gate

A stateless Goldfish reads the document. Can an outside observer with zero historical context understand exactly what this software is supposed to do ? If the Goldfish asks a clarifying question, the document fails.

Gate 2: The Critic Gate

Specialized Goldfish agents (acting as Security, UX or Performance specialists) aggressively audit the design. They hunt for hidden logic flaws, missing edge cases, and architectural bottlenecks.

Gate 3: The Readiness Gate

An implementation Goldfish looks at the blueprint and answers one question: Could a junior developer build this entire system right now without asking a follow-up question ?

If the document fails any of these gates, you take the Goldfish’s feedback back to the Elephant session, refine the blueprint, and run it through the gates again. Once it passes, the Elephant is finally allowed to generate the code. Because the blueprint is airtight, the resulting code is modular, clean, and free of technical debt.

Scaling to Massive Applications: “Peanuts and Hay”

A common objection to this model is: “This sounds great for greenfield projects, but how do I apply this to an enterprise codebase with millions of lines of legacy code?”

The framework handles this through a context-compression technique metaphorically called Peanuts and Hay:

1. Micro-Context Chunking: Large codebases are broken down into microscopic directories (the “peanuts”).

2. Bottom-Up Summarization: Isolated Goldfish agents scan these tiny directories and write hyper-dense, highly structured local readme.md files detailing local logic dependencies.

3. Hierarchical Indexing: A higher-level agent summarizes those readmes into a parent structure (the “hay”).

This creates a highly compressed, completely accurate map of institutional memory that the Elephant can consult instantly, without ever overloading its context window.

The old way of using AI—writing code line-by-line in a messy, endless chat room—is dead. It creates fragile codebases and exhausting debugging loops.

The Elephant-Goldfish model proves that the design document is the code. By rigorously separating long-term strategic context (The Elephant) from isolated, stateless execution validation (The Goldfish), you can harness the maximum cognitive power of LLMs.

Next time you start a feature, resist the urge to ask your AI for code. Ask it for a design doc, bring in the goldfish to tear it apart, and watch your development velocity skyrocket.

The problem is not that Ai is incapable. The problem is our interface. We are stuck in a manual, single-turn mindset.

Loop Engineering:

Welcome to Loop Engineering. This is the paradigm shift from writing static prompts to designing systems that execute continuous, autonomous loops.

This visual guide breaks down the core architecture of loops across 10 logical sections. Let’s trace this journey from fundamental problem to the future of AI development.

Step 1 & 2: The Bottleneck of the “One-Turn World”

The current state of interacting with AI relies entirely on a human supervisor manually micro-managing every transaction.

When every output requires a new human decision, building with AI becomes incredibly repetitive, slow, and fragile. Because humans must decide what happens next every single time the AI finishes a task, the human becomes the ultimate bottleneck.

Step 3: The Big Idea (Letting AI Decide the Next Step)

What if we removed the human from the micro-management layer entirely?

The core premise of Loop Engineering is shifting from a linear sequence to a continuous cycle. Instead of a human executing every turn, a high-level goal is fed into an autonomous loop: Plan —> Build —> Test —> Review —> Improve.

The mindset shift is clear: Stop writing prompts. Start designing systems that write prompts.

Step 4 & 5: The Loop Engine and the Core Cycle

To bring this big idea to life, we need a standard operational architecture. This is managed by the Loop Engine.

The core premise of Loop Engineering is shifting from a linear sequence to a continuous cycle. Instead of a human executing every turn, a high-level goal is fed into an autonomous loop: Plan —> Build —> Test —> Review —> Improve.

The mindset shift is clear: Stop writing prompts. Start designing systems that write prompts.

Step 6: A Real-World Example (Building a Login Page)

How does this look in practice? Let's take a common development task: creating a functional application feature.

Instead of a human developer reviewing code, spotting a lack of input validation, writing a new prompt, running tests, seeing an error, and writing another prompt—the Loop Engine manages it natively. The system generates code, tests it, catches its own omissions, fixes its own bugs, and delivers a finished, verified asset in a fraction of the time.

Step 7 & 8: Why Loops Beat Prompts in Dynamic Workflows

When we compare traditional prompting side-by-side with Loop Engineering, the structural advantages become undeniable.

Traditional prompting is fundamentally linear, fragile, and manual. Loop Engineering is cyclical, scalable, and resilient.

Furthermore, loops break down the limitation of a single LLM instance. A single Loop Engine can orchestrate a Dynamic Workflow composed of multiple specialized sub-agents—such as Research, Coding, Testing, Reviewer, and Documentation agents—all collaborating within the same loop to deliver a unified result.

Step 9 & 10: The Five Building Blocks and the Future of AI Development

To construct a reliable Loop Engine, you need to bring together five core architectural building blocks, shifting the entire developer paradigm.

The robust loop environment relies on a shared Memory (Context) foundation supporting:

1. Automations: Background triggers and handlers.

2. Skills: Specific capabilities or tool executions.

3. Connectors: Integrations to external systems and data.

4. Sub-Agents: Specialized LLM personas.

5. Workspaces: Isolated sandboxes where code can be executed and tested.

This entirely reimagines the role of the software developer. In the old world, developers spent their time writing individual prompts to get individual outputs. In the new world, the developer designs the loop, and the loop builds the software.

The era of manual, chat-based micro-management is coming to a close. To build truly powerful, autonomous software systems, we must look past the individual prompt window.

Design the engine, orchestrate the sub-agents, and establish robust verification layers.

The concept is straightforward. Instead of scattering knowledge across Notion, Google Docs, browser bookmarks, and sticky notes, you keep everything as structured markdown files. Then you point Claude Code (or any other agent like Gemini CLI) at the folder and ask it questions. The LLM reads your files, finds what’s relevant, and gives you grounded answers drawn from your own knowledge - not the general internet.

What Makes This Different ?

Most note-taking apps are build for human reading. You browse, search, click. They’re optimized for you to find things manually.
An LLM wiki is optimized for the model to read on your behalf. That shift changes everything about how you structure information. Here’s the core difference:

• Traditional App: You remember where something is navigate to it

• LLM Wiki: You describe what you need in plain language, and LLM finds and synthesizes it across your entire knowledge base.

Why does this idea appear now ?

A lot of the current AI document workflows look like RAG. You upload documents, the model retrieves relevant pieces at query time, and then generates an answer. But here there are some limitations:

• Questions that requires synthesis across many documents often force repeated rediscovery

• Good answers disappear into chat logs

• Interpretations do not accumulate very well across sessions

The LLM Wiki reframes it like : ‘Instead of reassembling knowledge every time, compile it once and keep it updated‘

How LLM Wiki Actually Works ?

The architecture is the most minimal. There are 3 components:

1. A folder of markdown files: This is your knowledge base. It can contain anything: research notes, meeting summaries, project documentation, book notes, personal reference material, code snippets with explanations.

2. A consistent structure within each file: Good LLM wikis use a consistent internal format - a title, a brief summary, tagged topics, and then the content. The model uses this structure to locate relevant information faster.

3. Claude Code as the query interface: You open a terminal, navigate to your wiki folder, launch Claude Code, and ask it a question. It reads the files it needs, synthesizes an answer, and can even update or add notes when you ask it to .

1. Data Ingest: Compile Raw Materials into a Wiki

Karpathy describes dropping source materials such as articles, papers, repositories, datasets, and images into a raw directory and then using LLM to incrementally compile a wiki from them. This includes:

• Model write summaries

• Creating Backlinks

• Group information into concepts

• Write page for those concepts

• Links the page together

2. IDE: Obsidian acts as Frontend

Obsidian is the best thing we can use as IDE Frontend. Here we can view raw data, compiled wiki and derived visual outputs. The LLM writes and maintains the wiki, while the human mostly reads, explores and guides.

3. Q&A

He writes that one of his recent research wikis is roughly 100 articles and 400K words, and that at this kind of small scale, more elaborate RAG infrastructure was not strictly necessary.

The reasons he gives are practical:

• The LLM keeps index files updated

• It maintains brief summaries of the documents

• It can read and connect the important related material fairly well

So the claim is not “RAG is unnecessary.”
It is closer to: “at smaller scales, a structured wiki plus index files can go surprisingly far.”

4. Output: Answers filed back to the wiki

Karpathy says he often prefers outputs not as plain terminal text but as markdown files, Marp slides, or matplotlib figures that he can view again in Obsidian.

And one of the most important ideas is this:

Good outputs can be filed back into the wiki.

That means the results of exploration become part of the knowledge base.

• An analysis can become a new page

• A slide deck can become both a deliverable and a knowledge artifact

• A chart can become something queryable later

This is where the gist and the follow-up posts point in the same direction:
queries compound instead of disappearing.

5. Linting: the wiki gets ongoing health checks

He also describes running LLM-based health checks over the wiki: finding inconsistent data, filling gaps via web search, and identifying interesting connections that might deserve new pages.

This matters because it shows the wiki is not just a static archive.
It is something that can be continuously cleaned up and strengthened.

6. Extra tools: search and tooling naturally get added

In a follow-up post he says he vibe-coded a small search engine over the wiki.
He uses it directly through a web UI, but more often wants to pass it to an LLM through the CLI for larger queries.

That suggests a natural progression:

• Start with markdown files and index pages

• Add search tools as the repository grows

• Eventually make those tools available to the LLM as part of its working toolset

In that sense, LLM Wiki is both a documentation pattern and a starting point for a tool ecosystem.

Set Up Your LLM Wiki in Obsidian

Step 1: Install Obsidian and Create a Vault:

Download Obsidian from the official Obsidian site. Create a new vault in a folder you’ll remember — something like ~/wiki or ~/Documents/llm-wiki.

A “vault” in Obsidian is just a folder. Everything in it is plain markdown.

Step 2: Create the folder structure

Open your Coding Agent (Claude Code, Gemini CLI) and paste the following prompt:

You are now my LLM Wiki agent. Implement this exact idea file as my complete second brain. Guide me step-by-step: create the SKILL.md schema with full rules, setup index.md and log.md, define folder conventions, and show me the first ingest example. From now on, every interaction follows the schema.

# LLM Wiki

A pattern for building personal knowledge bases using LLMs.

This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.

## The core idea

Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.

The idea here is different. Instead of just retrieving from raw documents at query time, the LLM **incrementally builds and maintains a persistent wiki** — a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM doesn't just index it for later retrieval. It reads it, extracts the key information, and integrates it into the existing wiki — updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then *kept current*, not re-derived on every query.

This is the key difference: **the wiki is a persistent, compounding artifact.** The cross-references are already there. The contradictions have already been flagged. The synthesis already reflects everything you've read. The wiki keeps getting richer with every source you add and every question you ask.

You never (or rarely) write the wiki yourself — the LLM writes and maintains all of it. You're in charge of sourcing, exploration, and asking the right questions. The LLM does all the grunt work — the summarizing, cross-referencing, filing, and bookkeeping that makes a knowledge base actually useful over time. In practice, I have the LLM agent open on one side and Obsidian open on the other. The LLM makes edits based on our conversation, and I browse the results in real time — following links, checking the graph view, reading the updated pages. Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.

This can apply to a lot of different contexts. A few examples:

- **Personal**: tracking your own goals, health, psychology, self-improvement — filing journal entries, articles, podcast notes, and building up a structured picture of yourself over time.
- **Research**: going deep on a topic over weeks or months — reading papers, articles, reports, and incrementally building a comprehensive wiki with an evolving thesis.
- **Reading a book**: filing each chapter as you go, building out pages for characters, themes, plot threads, and how they connect. By the end you have a rich companion wiki. Think of fan wikis like [Tolkien Gateway](https://tolkiengateway.net/wiki/Main_Page) — thousands of interlinked pages covering characters, places, events, languages, built by a community of volunteers over years. You could build something like that personally as you read, with the LLM doing all the cross-referencing and maintenance.
- **Business/team**: an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents, customer calls. Possibly with humans in the loop reviewing updates. The wiki stays current because the LLM does the maintenance that no one on the team wants to do.
- **Competitive analysis, due diligence, trip planning, course notes, hobby deep-dives** — anything where you're accumulating knowledge over time and want it organized rather than scattered.

## Architecture

There are three layers:

**Raw sources** — your curated collection of source documents. Articles, papers, images, data files. These are immutable — the LLM reads from them but never modifies them. This is your source of truth.

**The wiki** — a directory of LLM-generated markdown files. Summaries, entity pages, concept pages, comparisons, an overview, a synthesis. The LLM owns this layer entirely. It creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. You read it; the LLM writes it.

**The schema** — a document (e.g. CLAUDE.md for Claude Code or AGENTS.md for Codex) that tells the LLM how the wiki is structured, what the conventions are, and what workflows to follow when ingesting sources, answering questions, or maintaining the wiki. This is the key configuration file — it's what makes the LLM a disciplined wiki maintainer rather than a generic chatbot. You and the LLM co-evolve this over time as you figure out what works for your domain.

## Operations

**Ingest.** You drop a new source into the raw collection and tell the LLM to process it. An example flow: the LLM reads the source, discusses key takeaways with you, writes a summary page in the wiki, updates the index, updates relevant entity and concept pages across the wiki, and appends an entry to the log. A single source might touch 10-15 wiki pages. Personally I prefer to ingest sources one at a time and stay involved — I read the summaries, check the updates, and guide the LLM on what to emphasize. But you could also batch-ingest many sources at once with less supervision. It's up to you to develop the workflow that fits your style and document it in the schema for future sessions.

**Query.** You ask questions against the wiki. The LLM searches for relevant pages, reads them, and synthesizes an answer with citations. Answers can take different forms depending on the question — a markdown page, a comparison table, a slide deck (Marp), a chart (matplotlib), a canvas. The important insight: **good answers can be filed back into the wiki as new pages.** A comparison you asked for, an analysis, a connection you discovered — these are valuable and shouldn't disappear into chat history. This way your explorations compound in the knowledge base just like ingested sources do.

**Lint.** Periodically, ask the LLM to health-check the wiki. Look for: contradictions between pages, stale claims that newer sources have superseded, orphan pages with no inbound links, important concepts mentioned but lacking their own page, missing cross-references, data gaps that could be filled with a web search. The LLM is good at suggesting new questions to investigate and new sources to look for. This keeps the wiki healthy as it grows.

## Indexing and logging

Two special files help the LLM (and you) navigate the wiki as it grows. They serve different purposes:

**index.md** is content-oriented. It's a catalog of everything in the wiki — each page listed with a link, a one-line summary, and optionally metadata like date or source count. Organized by category (entities, concepts, sources, etc.). The LLM updates it on every ingest. When answering a query, the LLM reads the index first to find relevant pages, then drills into them. This works surprisingly well at moderate scale (~100 sources, ~hundreds of pages) and avoids the need for embedding-based RAG infrastructure.

**log.md** is chronological. It's an append-only record of what happened and when — ingests, queries, lint passes. A useful tip: if each entry starts with a consistent prefix (e.g. `## [2026-04-02] ingest | Article Title`), the log becomes parseable with simple unix tools — `grep "^## \[" log.md | tail -5` gives you the last 5 entries. The log gives you a timeline of the wiki's evolution and helps the LLM understand what's been done recently.

## Optional: CLI tools

At some point you may want to build small tools that help the LLM operate on the wiki more efficiently. A search engine over the wiki pages is the most obvious one — at small scale the index file is enough, but as the wiki grows you want proper search. [qmd](https://github.com/tobi/qmd) is a good option: it's a local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking, all on-device. It has both a CLI (so the LLM can shell out to it) and an MCP server (so the LLM can use it as a native tool). You could also build something simpler yourself — the LLM can help you vibe-code a naive search script as the need arises.

## Tips and tricks

- **Obsidian Web Clipper** is a browser extension that converts web articles to markdown. Very useful for quickly getting sources into your raw collection.
- **Download images locally.** In Obsidian Settings → Files and links, set "Attachment folder path" to a fixed directory (e.g. `raw/assets/`). Then in Settings → Hotkeys, search for "Download" to find "Download attachments for current file" and bind it to a hotkey (e.g. Ctrl+Shift+D). After clipping an article, hit the hotkey and all images get downloaded to local disk. This is optional but useful — it lets the LLM view and reference images directly instead of relying on URLs that may break. Note that LLMs can't natively read markdown with inline images in one pass — the workaround is to have the LLM read the text first, then view some or all of the referenced images separately to gain additional context. It's a bit clunky but works well enough.
- **Obsidian's graph view** is the best way to see the shape of your wiki — what's connected to what, which pages are hubs, which are orphans.
- **Marp** is a markdown-based slide deck format. Obsidian has a plugin for it. Useful for generating presentations directly from wiki content.
- **Dataview** is an Obsidian plugin that runs queries over page frontmatter. If your LLM adds YAML frontmatter to wiki pages (tags, dates, source counts), Dataview can generate dynamic tables and lists.
- The wiki is just a git repo of markdown files. You get version history, branching, and collaboration for free.

## Why this works

The tedious part of maintaining a knowledge base is not the reading or the thinking — it's the bookkeeping. Updating cross-references, keeping summaries current, noting when new data contradicts old claims, maintaining consistency across dozens of pages. Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass. The wiki stays maintained because the cost of maintenance is near zero.

The human's job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM's job is everything else.

The idea is related in spirit to Vannevar Bush's Memex (1945) — a personal, curated knowledge store with associative trails between documents. Bush's vision was closer to this than to what the web became: private, actively curated, with the connections between documents as valuable as the documents themselves. The part he couldn't solve was who does the maintenance. The LLM handles that.


## Note

This document is intentionally abstract. It describes the idea, not a specific implementation. The exact directory structure, the schema conventions, the page formats, the tooling — all of that will depend on your domain, your preferences, and your LLM of choice. Everything mentioned above is optional and modular — pick what's useful, ignore what isn't. For example: your sources might be text-only, so you don't need image handling at all. Your wiki might be small enough that the index file is all you need, no search engine required. You might not care about slide decks and just want markdown pages. You might want a completely different set of output formats. The right way to use this is to share it with your LLM agent and work together to instantiate a version that fits your needs. The document's only job is to communicate the pattern. Your LLM can figure out the rest.

This will create the entire folder structure needed. You can also copy it from the following GitHub Gist.

Step 3: Ingest your first Knowledge Base

Add a new file under your raw/ folder and give the following prompt to the LLM:

I have ingested a new file under the raw folder. Ingest it.

And you will the LLM do the magic. It will go through the file, summarize it, find cross references, link them.

Step 4: MindMap created by Obsidian

Click on the mindmap button on obsidian to see the entire mind-map created by obsidian. This is how my mindmap looks like:

Most Useful way to understand LLM Wiki

LLM Wiki tends to be strongest when:

• materials keep accumulating

• similar questions recur

• links and context matter

• human-readable intermediate outputs are valuable

It is not something every team should adopt immediately.
If real-time data, high-stakes decisions, or large-scale freshness are the real problem, other retrieval systems may come first.

Still, the idea matters.
At minimum, Karpathy’s proposal suggests that alongside better models, we may also need better-maintained memory layers.

LLMs have become very powerful engines for document understanding and question answering. However, they are constrained by a fundamental architectural limit: context window which is the maximum number of tokens the model can process at once. This makes it challenging for LLMs to accurately interpret and reason over long, complex, domain-specific documents such as financial reports or legal filings.

To solve this, developers have heavily relied on Retrieval-Augmented Generation (RAG) powered by vector databases. But what if the very foundation of vector RAG is flawed for complex tasks ?
Enter PageIndex, a paradigm-shifting, “vectorless“ and reasoning-based RAG framework. Instead of relying on mathematical embeddings, PageIndex organizes documents into a hierarchical tree and uses the LLM’s own reasoning to navigate them -- just like a human expert would. Let us deep dive into how it works, why it matters, and how it’s redifining document retrieval.

The Problem with Traditional Vector RAG

In traditional RAG, a document is chopped up into fixed-size “chunks” (e.g., 512 tokens), converted into mathematical vectors, and stored in a database. When you ask a question, the system looks for chunks that are semantically similar to your query.

While this works for simple lookups, it falls apart in complex, domain-specific scenarios (like legal or financial analysis). Here is why:

1. Similarity ≠ Relevance: Vector databases assume that words with similar meanings contain the right answer. But if you ask for “inconsistencies across documents” or “direct quotes from John,” vector search fails because these concepts do not map well to embeddings.

2. Hard Chunking Destroys Context: Chopping a document into fixed blocks arbitrarily cuts through sentences and paragraphs, fragmenting the actual meaning of the text.

3. Blind to Cross-References: If a document says “see Appendix G,” a vector database will miss it because the text in Appendix G is not semantically similar to your original question.

4. No Conversational Memory: Vector searches treat every query in isolation, making it incredibly hard to maintain multi-turn chat contexts (e.g., asking “What are the assets?” followed by “What about the liabilities?”).

What is PageIndex ? The Core Concept

PageIndex takes a “subtraction” approach to innovation: it completely removes vector databases and chunking from the equation.

Instead of searching for semantic vibes, PageIndex relies on reasoning-based retrieval. In the preprocessing stage, it uses an LLM to read the document and generate a structured, JSON-based Table of Contents (ToC) tree.

This tree organizes the content into logical, natural sections—like chapters, subheadings, and pages—complete with node IDs, summaries, and metadata. This ToC is then fed directly into the LLM’s active context window, acting as an “in-context index”.

How PageIndex Works: The Human-Like Loop

When you ask PageIndex a question, it doesn’t just return a mathematical match. It dynamically “thinks” about where the answer might be, following a highly traceable, iterative loop:

1. Read the ToC: The AI reviews the document’s structure to understand the layout.

2. Select a Section: It infers which section is most relevant (e.g., “The user asked about deferred assets, let me check the Financial Summary section” ).

3. Extract Information: It pulls the raw data from that specific node.

4. Evaluate Sufficiency: It asks itself: “Did I find the complete answer?” If the answer references “Table 5.3 in Appendix G,” the AI loops back to the ToC, finds Appendix G, and reads it.

5. Answer the Question: Once all context is gathered, it delivers a precise, fully-informed response.

Trade-Offs: Accuracy vs Speed

No system is perfect, and the developer community has rightly pointed out the trade-offs of vectorless retrieval.

• The Pros: It is incredibly accurate. On the FinanceBench benchmark, PageIndex achieved a state-of-the-art 98.7% accuracy, significantly outperforming traditional vector systems. Furthermore, retrieval is completely transparent; the system leaves a trail of exactly which pages and sections it reasoned through.

• The Cons: It is computationally heavier and slower. Having an LLM iteratively read summaries and traverse a tree costs more time and money per query than a simple mathematical vector comparison.

Ultimately, PageIndex is built for quality maximalists. If you are building a quick chat app over a product catalog, use a vector database. If you are building an AI analyst to parse 200-page earnings reports where a hallucination could cost millions, PageIndex is the superior choice.

All of the applications rely on vector embeddings, which are a type of vector data representation that carries semantic information that is critical for AI to gain understanding and maintain a long-term memory they can draw upon executing complex tasks. Embeddings are generated by the LLMs and have many features, making their representation challenging to manage. These features represent different dimensions of data that are essential for understanding patterns, relationships and underlying structures.

This requires specialized database designed specifically for handling this data type. Vector Databases have the capabilities of traditional database that are absent in standalone vector indexes and the specialization of dealing with vector embeddings, which the traditional databases lack. These vector databases are intentionally designed to handle complex data and offer performance, scalability and flexibility you need to make the most out of your data. With a vector database, we as a developer can add our knowledge to our AIs, like semantic information retrieval, long-term memory and more**.**

How does a Vector Database Work ?

The traditional approach for database is that they store strings, numbers and other types of scaler data in rows and columns. Vector databases on the other hand operates on Vectors, so the way it is optimised and queried is quite different. In our traditional databases, we usually query for rows in the database where the value exactly matches our query, whereas in vector databases, we apply a similarity metric to find out a vector that is the most similar to our query.

In a vector database, there are combinations of different algorithms that participate in ANN Search Algorithm which is basically a technique to quickly find data points in large, high-dimensional datasets that are similar to a given query point, prioritizing speed and scalability over find single absolute closest match.

These algorithms are assembled into a pipeline that provides fast and accurate retrieval of neighbours of a queried vector. Below is a common pipeline:

1. Indexing: Indexing is about intelligently organising vector embeddings to optimise the retrieval process. It involves advanced algorithms to nearly arrange the high dimensional vectors in searchable and efficient manner. Some of the common techniques are Inverted File (IVF), Hierarchical Navigable Small World Algorithms (HNSW), Multi-Scale Tree Graph Algorithm (MSTG).

2. Querying: The vector database now compares the indexed query vector to indexed vectors in dataset to find the nearest neighbours.

3. Post Processing: Some cases where the vector database retrieves the final nearest neighbours from dataset, it then post-processes them to return the final results. This step can include reranking the nearest neighbours using a different similarity measure.

What are Vectors ?

Before moving forward, let us first get into the fundamentals first. Let us first understand what are Vectors. A vector is essentially a list (or array) of numbers that represents data. Think of a vector as a coordinate system for meaning.

Scalar vs. Vector: A “scalar” is a single number (e.g., today’s temperature is 85°F). A “vector” is a collection of numbers describing a complex object (e.g., today’s weather is representing the low, mean, and high temperatures)

Vector numbers can represent complex objects such as words, images, videos and audio generated by an ML model. This high-dimensional vector data, containing multiple features, is essential to machine learning, natural language processing (NLP) and other AI tasks.

What are Vector Embeddings ?

Now that we know what are vectors, let us now understand what are vector embeddings. Vector Embeddings are numerical representations of data points that converts various types of data including non-mathematical data such as words, audio or images - into array of numbers that ML models can process. Vector embedding is a way to convert an unstructured data point into an array of numbers that expresses that data’s original meaning.

Consider the example:

cat = [ 0.5, -0.7, 0.9]

dog = [0.9, 0.2, 0.9]

Here, each word is associated with a unique vector. The values in the vector represent the word’s position in a continuous 3-dimensional vector space. Embedding models are trained to convert data points into vectors. Vector databases store and index the outputs of these embedding models.

Summing up, this is how the vectors are displayed on a x-y-z scale:

Vector Indexing

Vectors created need to be indexed to accelerate searches within high-dimensional data spaces. Vector databases create indexes on vector embeddings for search functions. Indexing maps the vectors to new data structures that enables faster similarity such as nearest neighbor searches, between vectors. Vector can be indexed by using algorithms such as Hierarchical Navigable Small World (HNSW), Locality-Sensitive Hashing (LSH) or Product Quantization (PQ)

Semantic Search

Traditional search engines typically focus on matching keywords within a search query to corresponding keywords in indexed web pages. Semantic search is a data searching technique that focuses on understanding the contextual meaning and intent behind a user’s search query, rather than only matching keywords. Imagine searching for “best laptops for graphic design students.” A traditional search engine might focus solely on matching those keywords to web pages. Whereas, a semantic search engine would try to understand that you are looking for laptops with specific features like powerful graphics cards, ample RAM, and color-accurate displays. Then it would return the results that would recommend laptops for graphic design related tasks.

Here’s how it works:

Semantic search engines employ various techniques from natural language processing (NLP), knowledge representation, and machine learning to understand the semantics of search queries and web content. Here’s a breakdown of the process:

• Query analysis: The search engine analyzes the user’s query to identify keywords, phrases, and entities. It also attempts to interpret the user’s search intent by analyzing the relationships between these elements.

• Knowledge graph integration: Semantic search engines often leverage knowledge graphs, vast databases containing information about entities and their relationships. This information helps the engine understand the context of the search query.

• Content analysis: Similar to how a search engine analyzes queries, it also examines the content of web pages to determine their relevance to a particular search. This analysis goes beyond keyword matching and considers factors such as the overall topic, sentiment, and entities mentioned within the content.

• Result return and retrieval: Based on the analysis of the query and the content, the search engine could return web pages according to their relevance and semantic similarity to the search query. It then retrieves and displays the most relevant results to the user.

Use the links given below to understand more about vector databases:

• https://www.ibm.com/think/topics/vector-database

• https://thedataquarry.com/blog/vector-db-2/

• https://www.pinecone.io/learn/vector-database/?_sm_vck=MqVLq7tq4TJQHJg5gZqjMVZV0rNTTqHHvLHsNWnn00fq7tHvvrvn

• https://www.computer.org/publications/tech-news/trends/vector-database-deep-dive

Introduction: Why is AI Naturally “Forgetful” ?

Consider this very use case:

I created a fresh new account on ChatGPT and asked it Who am I ? The answer that I got is ‘I’m not sure yet - you tell me!‘. This means that it does not have any context or memory initially of who I am . Now, the next thing I tell it is to remember that ‘My name is Aditya Trivedi and I am from Mumbai’.

What do we see from the image above ? We get that Memory in ChatGPT has changed meaning now the AI has a memory of who I am and where do I live. It also tries to remember your recent chats but may forget over time.

Now when I ask the same question, ChatGPT will now be able to answer the question, why ? Because now it has the memory of my name and where do I live.

Tools like ChatGPT feel helpful until you find yourself repeating instructions or preferences, again and again. To build agents that can learn, evolve and collaborate, real memory isn’t just beneficial - it’s essential.

To understand memory, we first need to understand how tools like ChatGPT work under the hood.

1. The Stateless Problem:

   By default, AI models are stateless. This means that they do not have built-in memory of past requests. If you say “Hello” and AI replies, and then when you say “My name is Aditya”, the AI will process that specific message. The catch here is that if you send a new message without the previous context, the AI will treat it as a brand-new conversation. One way to keep the chat going could be that the developer actually has to send entire conversation history back to AI with every single message you type.

2. Context Window Limit:

   Considering the above solution , you may ask “Why now send the entire history forever ?”. Hehe, that’s not possible. AI models have a limit called the Context Window. In simple words you can think of it as AI’s short term-attention span.

What do we mean by Memory in AI Agents ?

Memory is the ability to retain and remember/recall relevant information from the context of the user’s input across time, tasks and multiple user interactions. It allows agents to remember what happened in the past and use the relevant information to improve behaviour in the future.
It is not about storing the chat history, it is about giving more and more context to the LLMs and building a persistent state which evolves.

Think of it like a human memory. If you meet a friend after say 15 years, it is obvious that you would not remember everything. You remember the important stuff like their name, where they live, what are their likings in food, etc. This is the exact same thing the Memory Layer does for AI.

The 4 Types of AI Memory:

Short Term Memory:

This is like a temporary memory. It will exist only for the current conversation or task and will be deleted once the task is finished. For eg, you are going to a restaurant and you order a coffee and a burger and you get the order number 132. After 10 minutes, the waiter calls for order number 132. Since you have memorized the number as your order number, you get the food. Once you complete eating, you leave, you forget the number. You don’t remember the order number for the rest of your life.

Long-Term Memory:

This memory lasts forever. It will stay across different chats, days months or even if you refresh your browser 😅. It is now divider into 3 subtypes:

1. Factual Memory: Retains user preferences, communication style, domain context etc. For eg “You prefer TypeScript output and detailed answers”.

2. Episodic Memory: Remembers specific past interactions or outcomes. For eg “Last time when i used this script, it destroyed my performance”.

3. Semantic Memory: Stores generalized, abstract knowledge acquired over time. For eg “Tasks involving JSON parsing usually stress you out, want a quick template?”

RAG ≠ Memory

While both RAG (Retrieval-Augmented-Generation) and memory systems retrieve information to support LLMs, they both solve very different problems.

RAG brings external knowledge into the prompt at inteference time. It is fundamentally stateless meaning it has no awareness of previous interactions, user identity, or how the current query relates to past conversations whereas

Memory, brings continuation. It will capture user preferences, past queries, decisions and failures and make them available in future interactions.

In a world where every agent has access to the same models and tools, memory will be the differentiator. Not just the agent that responds — the one that remembers, learns, and grows with you will win.

• Querying data to get the data from database

• Executing an API call to make a POST request over a server

• Having AI do the stuff for you

What is the Agent to Agent Protocol ?

A2A Protocol is a protocol launched by Google to maximise the benefits that we can get from Agentic AI. This is to make the agents be able to collaborate in a dynamic, multi-agent ecosystem across data systems and applications.

Here they have collaborated with more than 50 technology partners like Atlassian, Langchain, MongoDB, PayPal, CapGemini, KPMG, PwC, TCS, etc**.** This will now allow AI agents to communicate with each other, securely exchange information, coordinate actions on top of various enterprise platforms or applications. It is an open protocol that complements Anthropic’s Model Context Protocol (MCP), which provides helpful tools, and context to agents.

How Does it Work ?

Consider an example where there is an agent made by me. My agent has the task to get the resume, analyse it , and return the best fit candidate for based on the Job Description and Job Profile. Now consider Apple is the recruiting partner, and they have created their own agent to get the work done for them and get the best candidate for them. This would be the flow now:

• Agent 1 finds the relevant job role, description and sends it to Agent 2.

• Agent 2 now checks all the resumes which match the job role and description given by Agent 1.

• After analysing and finding out the best candidate, it sends it to Agent 1.

• Agent 1 now sends handles the next step.

With this both the agents will be able to talk internally within themselves.

Consider a real life example:

Dominos & Burger King have also started to create their own agent to make their work life easier for them. Zomato now as a platform which has both Dominos and Burger King will have their own agent which will take care for the work for them.

Consider a situation where the user says, ‘I want to eat a Pizza’, what will Zomato’s agent do ?. It needs to get the access to Domios’s Database to get the orders, but how will it get ? This will be done via the A2A Protocol. Same with the case when the user says, ‘ I want to eat a Whooper Burger Also’.

How Do These Agents Discover Each Other ?

Making it more simple, How does Zomato’s agent discover that Dominos Agent has the following capabilities ?

They connect via OpenID Connect.

What is OpenID Connect ?

Imagine you’re logging into a website, and instead of creating an account, you see a “Login with Google” button. You click it, and boom — you’re in. That is powered by OpenID Connect (OIDC).

📌 In Simple Terms:

OpenID Connect is a layer built on top of OAuth 2.0 that lets you log into apps using your existing account from providers like Google, Microsoft, Facebook,etc. It handles:

• ✅ Authentication (Who are you?)

• 🔐 Secure login without passwords

• 🔁 Passing user info between services

🔄 `/.well-known` in OIDC:

/.well-known is a standard path used by OIDC to automatically discover important configuration information about the identity provider like Google, AuthO, Okta,etc. In other terms, it’s where the identity provider publishes its OIDC configuration so that client apps know where to authenticate, get tokens, and fetch user info. Let’s take an example:

When an app wants to use Google as an identity provider, it goes to:

https://accounts.google.com/.well-known/openid-configuration

And gets back a JSON file like this:

For the agents, they connect via ./well-known/agent.json configuration in OpenID Connect.

This is just the start of something new, there’s more to come. Checkout the GitHub link below for a sample code using A2A Protocol.

Source Code: https://github.com/google/A2A