Autonomous agents need a record of what they tried

You go to bed. An agent runs a hundred experiments on a real language model training setup while you sleep. In the morning there’s a tab-separated file listing everything it tried and a git branch that has moved forward past every improvement it found.

That’s autoresearch, which Andrej Karpathy released in March 2026. It’s a small project and a very good demo, and I think it’s one of the clearest pictures so far of what autonomous agents look like when you actually let them run. It also shows, quite quickly, where the tooling around them runs out.

How it works

There are three files. prepare.py handles data loading and evaluation, and the agent isn’t allowed to touch it. train.py holds the whole GPT model, optimiser and training loop, and it’s the only file the agent can change. program.md is what Karpathy calls the research org code: the instructions for how the agent should behave.

The loop goes like this:

1. The agent reads train.py and decides on a change (architecture, hyperparameters, optimiser, batch size).
2. It commits the change to git.
3. It runs uv run train.py > run.log 2>&1.
4. Training runs for exactly five minutes of wall-clock time, not counting start-up and compilation.
5. The agent pulls val_bpb (validation bits per byte, a compression-based loss where lower is better) out of the log.
6. If it improved, the branch advances. If not, git reset.
7. The result goes into results.tsv.
8. Repeat. Never stop. Never ask the human.

The last step is the one that makes it different. program.md tells the agent, in capitals, “NEVER STOP… Do NOT pause to ask the human if you should continue.” The agent decides what to try, tries it, judges the result and moves on. The human’s job moves from writing Python to writing the document that sets the strategy and the limits. In Karpathy’s words: “you’re not touching any of the Python files like you normally would as a researcher. Instead, you are programming the program.md Markdown files.”

Two design choices are worth stealing.

The first is the fixed five-minute budget. Every experiment gets the same wall-clock time, so results are comparable whatever the agent changes. A smaller, faster architecture gets more training steps in five minutes and wins. A bloated one gets fewer and loses. The budget selects for efficiency without anyone having to say so. At eight to twelve experiments an hour, that’s somewhere around 70 to 100 configurations overnight on a single H100.

The second is that the number isn’t the whole objective. program.md says: “A 0.001 val_bpb improvement that adds 20 lines of hacky code? Probably not worth it.” Deleting code without losing performance counts as a win. That’s a judgement call written into the instructions, and it’s more thoughtful than most automated hyperparameter search.

If you want to try it and have an NVIDIA GPU:

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
uv run prepare.py

Then point your coding agent at the repo and tell it to follow program.md. It creates a dated branch and gets going. It’s tested on H100s; there are community forks for Apple Silicon, Windows and AMD, and on smaller GPUs you can shrink MAX_SEQ_LEN, VOCAB_SIZE and DEPTH in train.py. Give it a full night.

The pattern behind it

The obvious win is that one person runs a hundred experiments overnight instead of eight during the working day. For a small team without a cluster, that’s a lot.

The bigger point is that the shape generalises. Set a goal you can measure, give an agent the code, let it iterate without supervision, and keep track of what happened. That works for compiler flags, query tuning, infrastructure configuration, performance work, and plenty of everyday software tasks where feedback is quick and the space of options is large. I’d expect a lot of software to be built and maintained this way within a few years.

And it’s the last part, keeping track, where things get interesting.

What a log file can’t tell you

Autoresearch keeps its history in two places. Git holds the changes that survived; failed ones are reset and vanish from the branch. results.tsv holds one row per experiment, failures included.

That’s fine at a hundred experiments. Somewhere past a few hundred, ask it a few ordinary questions and see how it copes.

Which architecture changes helped after the switch to the Muon optimiser? The TSV has the scores and git has the surviving diffs, but the link between them lives in the code. You’d have to check out each commit, read train.py, work out what it was doing, and line it up against the metric.

Should I try twelve layers again? It failed three times. Did it fail on its own, or because it was bundled with something else each time? The log can’t separate them, and the failed diffs are gone from the branch.

Why did experiment 47 work? The agent knew at the time. It had a reason for trying it and an expectation of what would happen. None of that was written down. Git records what changed. It doesn’t record why, or what the agent expected, or how one result relates to another.

A human researcher carries this in their head: which ideas have been tested, which variables interact, which directions are dead ends. An agent working from a log and a repo rebuilds it from scratch every time, and at experiment 500 it’s deciding with the same flat context it had at experiment 5. Two agents working side by side have it worse, because neither can see why the other did what it did.

Keeping a proper record

What I think autonomous agents need is a record of their work, attached to the work itself. For every change: what was tried, what the agent intended, what it expected, what actually happened, and what rules it was working under. Kept whether the change survived or not. Readable by the next agent, and by a person who turns up later and asks why the system looks the way it does.

For a research loop, that’s about efficiency. The agent stops retrying things it already ruled out, and you can actually learn from a night’s work instead of squinting at a spreadsheet.

For production software it’s more than efficiency. Once agents are changing systems that people depend on, “why is it like this?” stops being curiosity and becomes something an auditor, an incident review or a customer can ask. A git log and a TSV are a reasonable answer for a weekend project. They aren’t for a payments system. I’ve spent enough time in trade finance to know how much of the job is being able to show your working.

That’s one of the things we’re building into Kizuna: every change an agent makes comes with a record of what it set out to do, what it did, and on what authority, so you can follow the reasoning later without reconstructing it. We’re still in development, but it’s the part I’d least want to bolt on afterwards.

Where this goes

Karpathy introduced autoresearch with a tongue-in-cheek note from the future: “Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures,” filed as documenting “how it all began.”

It’s a joke, and I think the direction is right. Agents that change code, judge the results and build on them without a person watching will become normal, in research and in ordinary engineering. Whether they can do the work is mostly settled. What’s still open is whether we’ll be able to tell, afterwards, what they did and why.

If you’re running agents like this and thinking about the same problem, we’d like to hear from you.