My favorite memory system for LLMs

Context is the ceiling for language model usefulness in a project, more so than model intelligence - the more complex a project becomes, the more tedious it gets to keep the model in the loop. The good news is: Language models are actually great at structuring and keeping track of information, they just have to be told to do so.

I have been using Karpathy’s “LLM wiki” approach for each of my projects for the past couple months, which works so well that I now use the models to help ME understand relevant context for projects rather than the other way around.

The system is quite simple and easy to understand: you try to gather as many relevant sources as possible, and have each one be “ingested” by the model to build a knowledge graph consisting of linked markdown files, similar to a fandom wiki.

The sources go in a sources directory, which is immutable to the model and serves as the ground truth. Each time a new source is ingested by the model, it updates the wiki with any new concepts, topics or people.

The wiki is treated as a first class priority, no question is answered without thorough investigation, any information gathered in the course of the project has to be persisted somewhere in it.

I can easily access and reference this information with a markdown reader like obsidian, although the wiki is meant to be largely untouched by the human and is maintained and read mostly by the LLM.

I wrapped Karpathy’s instructions in four small skills: /create-wiki to scaffold it, /ingest to fold in new sources, /query to answer from the wiki, and /lint to clean it up periodically, this allows me to set this up in any project in seconds.

These are available as reusable skills: install them with this command, npx skills add lukaskunhardt/skills, or grab them on GitHub