AI benchmarks ignore 90% of real work: test it with Bun

According to Mercor, 90% of the hours worked in the United States go to tasks that AI benchmarks don't cover well. You've seen the chart of the moment: a new model breaks a record on some test and everyone celebrates. But that test measures a tiny slice of what we do every day. This post is about that gap between AI benchmarks and real work, and about using Bun to measure what matters to you.

What Mercor measured in AI benchmarks

The research was published in Mercor's Labor Index. The idea is simple to explain: take the tasks people do at work, see how much time each one takes, and check which benchmarks test each of them.

The result is uncomfortable. Most work hours fall on tasks that no popular benchmark evaluates well. Tests cluster where grading is easy: multiple choice questions, math problems with a single answer, isolated functions with a ready-made unit test.

Real work lives somewhere else. It's in meetings, in decisions made with missing information, in reading other people's code, in negotiating scope, in understanding what the client meant when they wrote "make it more modern".

That doesn't mean AI is useless for those tasks. It means nobody is measuring. Those are very different things.

Why this matters if you write code

We think we're the exception. After all, programming is the most benchmarked field there is: SWE-bench, HumanEval, LiveCodeBench, Terminal-Bench, and another dozen that showed up last month.

But think about your last week. How much time did you spend writing new code from a clear description? And how much time went to:

• figuring out why the deploy works on staging and breaks in production;
• reading a 40-file PR and deciding whether to approve it;
• migrating a PostgreSQL table without taking anything down;
• explaining to product why that "quick change" takes two weeks.

Coding benchmarks usually test the part of the job that already comes with a well-written spec. In real life, writing the spec is half the job.

There's also the contamination problem. A lot of public benchmarks end up leaking into model training data. A model that does well on a test it has already seen is memorizing the exam, and you don't know how much of that happened.

A benchmark tells you what the model can do on the exam. Your backlog tells you what you need it to do.

What changes in your workflow

The practical takeaway isn't "ignore benchmarks". They work as a coarse filter: a model that does badly on all of them will probably do badly on your project too.

The takeaway is different: to decide which model to use on your code, you need your own test. A small eval, built from real tasks in your repository, that you run every time a new model comes out or when you change your agent's prompt.

It sounds like a job for a research team, but it isn't. If you have a test runner, you already have 80% of the infrastructure. And this is where Bun helps a lot: bun test starts fast, runs TypeScript with zero config, and ships Bun.spawn and Bun.file for everything else.

How to build your own eval with Bun

The structure I use is very simple. Each case is a JSON file with the prompt and the test file the generated code has to pass.

{
  "name": "cursor pagination on the orders endpoint",
  "prompt": "Implement cursor pagination in listOrders(). It currently uses OFFSET.",
  "file": "src/orders.ts",
  "test": "src/orders.test.ts"
}

The key is that the cases come from your own history. Pick tasks you've actually solved: that timezone bug, the payment service refactor, the query that was doing N+1. You already have the right answer and the test that proves it works.

Then, a test file that loops over the cases, asks the model for the code, applies it to a copy of the project, and runs the original tests:

// evals/code.test.ts
import { test, expect } from "bun:test";
import { Glob, $ } from "bun";
import { generateCode } from "./model";

const cases = await Array.fromAsync(new Glob("evals/cases/*.json").scan("."));

for (const path of cases) {
  const evalCase = await Bun.file(path).json();

  test(evalCase.name, async () => {
    const tmp = `/tmp/eval-${crypto.randomUUID()}`;
    await $`cp -r fixture ${tmp}`;

    const code = await generateCode(evalCase.prompt, evalCase.file);
    await Bun.write(`${tmp}/${evalCase.file}`, code);

    const proc = Bun.spawn(["bun", "test", evalCase.test], { cwd: tmp });
    expect(await proc.exited).toBe(0);
  }, 120_000);
}

generateCode is just a fetch to the API of the model you want to evaluate. Switch models with an environment variable and run:

MODEL=model-a bun test evals/
MODEL=model-b bun test evals/

In the end, you get a score like "17 out of 20 cases passed" for each model. That number is worth more for your decision than any public leaderboard, because it came from your code.

A few tips I learned the hard way:

• Start with 10 cases. Ten real cases beat a hundred made-up ones. You can expand later.
• Run each case more than once. Models aren't deterministic. If it passes 2 out of 3 times, write that down.
• Isolate the environment. Generated code might run rm in the wrong place. Use a temp directory, a container, or a sandbox. Don't run this in your main repository.
• Save the outputs. When a case fails, you want to read what the model wrote, not just see red in the terminal.

What about tasks that don't fit in a test?

That's the honest objection, and it's exactly the point of Mercor's research. Reviewing PRs, designing architecture, and deciding what not to do don't easily turn into an expect().toBe().

You can get close. For code review, build cases from old PRs that had a known bug and see if the model points out the problem. For architecture decisions, save the context of a decision you've already made and compare the model's reasoning with yours. Here the grading is manual: you read it and mark "helped", "didn't help", or "got in the way".

Is it work? Yes. But it's the same kind of work you'd do to evaluate a new junior dev on the team. Nobody hires someone just because they scored high on a multiple choice exam. Well, nobody should.

My take on AI benchmarks

I find Mercor's research more useful than half of this year's model launches, precisely because it isn't selling anything. It just shows where the ruler doesn't reach.

My read is that we outsourced evaluation to the people who have an interest in showing nice numbers. Whoever launches the model picks the benchmark, and the benchmark picks the task that's easy to measure. Nobody has to lie for the result to come out skewed.

The way out is within reach of any dev: an evals/ folder, twenty cases from your history, and a bun test. It takes an afternoon to set up and saves months of "I think this model is better". If you want to see how I apply this to real projects, take a look at my projects.

According to Mercor, 90% of work hours go to tasks that no AI benchmark measures well.

Tests live where grading is easy: multiple choice, math with a single answer, isolated functions.

But my real day is reading a 40-file PR, migrating a table without taking anything down, and figuring out what the client meant by "make it more modern".

So I stopped picking models by this week's leaderboard. I set up an evals/ folder with 20 tasks I've actually solved and run it all with bun test.

"17 out of 20 passed" on my code is worth more than any launch chart.

I wrote on the blog how to build yours in an afternoon. If you already do something similar, tell me how you measure it.

#ArtificialIntelligence #Bun #SoftwareDevelopment #LLM #TypeScript