Bun and LLMs: why opinions on AI are so split

In the same week, two devs I respect told me opposite things. The first said an agent wrote an entire feature for him in Bun, tests included, and all he did was review it. The second swore AI "can't even handle CRUD," because the model made up an import that doesn't exist. Both were telling the truth. And that's exactly what Andrej Karpathy complained about in a recent post on X: the gap between what people think LLMs can do is getting wider, not narrower.

This post is about that gap, using Bun as a thermometer. It makes sense. Bun is a runtime that moves fast, is now owned by Anthropic, and runs under one of the most widely used AI coding tools.

What Karpathy is pointing at

The core idea is simple. Back then, almost everyone used the same thing: one chat, one model, one prompt. Opinions varied, but the baseline experience was similar.

That's no longer the case. Some people use a free model in a browser tab, paste a snippet of code and copy the answer back. Others run an agent in the terminal, with access to the repo, running tests, reading errors and fixing things on their own for half an hour. These are different products that share the same name: "AI".

When those two people argue, each one is describing a different tool. AI debates in the company chat have turned into football arguments. Everyone watched a different game, and everyone is sure about the score.

If you tried AI once in 2024, you're arguing against a product that no longer exists.

Why perceptions of LLMs keep drifting apart

I don't think it's about someone being dumb or starstruck. The gap grows for very concrete reasons:

• The model matters a lot. On multi-file coding tasks, the difference between an entry-level model and a frontier one is huge. If you've only used the free one, your sample doesn't represent the rest.
• The environment matters even more. A good model in a chat with no context gets imports wrong. The same model inside an agent that reads package.json and runs bun test fixes the import on its own on the second try.
• Opinions age fast. A lot of people formed their view a year and a half ago and never tested again. In the meantime, the product changed about four times.
• The type of task changes everything. Generating an endpoint is one thing. Debugging a race condition in a worker is another. Both sides generalize from their own case.

The result is that the public conversation becomes useless. The hype says devs will be gone next year. The skeptic says it's glorified autocomplete. Neither one helps you decide whether to use this tomorrow at 9am on your project.

Bun is a good thermometer for this

Bun is an interesting case for three reasons.

First, it moves fast. APIs like Bun.serve with routes, bun:sqlite, Bun.password and the native S3 and Postgres clients arrived over several releases. A model with outdated knowledge, or without access to the docs, will suggest Express with better-sqlite3. It works, but it throws away the reason you picked Bun.

Second, Anthropic bought Bun in late 2025, and Claude Code ships as a Bun executable. So the runtime has become infrastructure for an AI tool. That doesn't mean models magically know Bun better. It means there are people with strong incentives to make Bun work well inside agent workflows.

Third, Bun has a very short feedback loop. bun test starts in milliseconds, with no Jest config and no separate TypeScript transpiling. For an agent that has to run tests, read the failure and try again twenty times, that makes a real difference in total time.

Here's the difference in practice. I asked for the same thing in two scenarios: "endpoint that fetches a user by id from SQLite." In the chat with no context, I got Express, better-sqlite3 and an app.listen. In the agent with access to a repo that already used Bun, I got this:

import { Database } from "bun:sqlite";

const db = new Database("app.db");
const findUser = db.query("SELECT id, name FROM users WHERE id = ?");

Bun.serve({
  routes: {
    "/api/users/:id": (req) => {
      const user = findUser.get(req.params.id);
      return user
        ? Response.json(user)
        : new Response("Not found", { status: 404 });
    },
  },
});

Zero new dependencies. The query is prepared only once. This is the kind of difference that makes one dev think AI is useless and another think it's great. The model may even have been the same. The context was different.

How to measure an LLM on your own code

The way to avoid both extremes is to stop giving opinions and start measuring. And you can do it with what you already have.

Build a homemade eval. You don't need a framework. Take 5 to 10 real tasks you've already solved in your repo: a bug that cost you an afternoon, a new endpoint, a schema migration. For each one, write the test that proves it's solved.

// evals/user-endpoint.test.ts
import { test, expect } from "bun:test";

test("returns 404 for a nonexistent user", async () => {
  const res = await fetch("http://localhost:3000/api/users/999999");
  expect(res.status).toBe(404);
});

test("returns the user with id and name", async () => {
  const res = await fetch("http://localhost:3000/api/users/1");
  const body = await res.json();
  expect(body).toMatchObject({ id: 1, name: expect.any(String) });
});

Then, for each task, create a clean branch, hand the description to the model or agent you want to evaluate, and run:

bun test evals/

Write down three things: whether it passed, how many times you had to step in, and how long it took. Repeat every time you switch models or tools. In one afternoon you'll have data worth more than a hundred threads on X.

Two tips I learned the hard way:

1. Use tasks you've already solved. That way you know what a good solution looks like, and you won't be impressed by code that only looks right.
2. Include at least one truly hard task. If they're all CRUD, you'll conclude AI solves everything. It doesn't.

What changes in your workflow

If the perception gap is real, the practical consequence is that you can't outsource your opinion on AI. Not to the excited influencer, not to the skeptical coworker. Both are describing their tool, not yours.

Day to day, this turns into a few small changes:

• Write code that verifies itself. Good tests are now input for the agent, not just a safety net for you. With Bun, running tests is so cheap there's no excuse.
• Give context instead of clever prompts. A CLAUDE.md or AGENTS.md saying "this project uses Bun, bun:sqlite and Bun.serve, don't install Express" fixes half the silly mistakes.
• Reassess every few months. Put it on your calendar. Seriously. Your opinion from March might be wrong in October.
• In team discussions, ask "which tool and which task?" before you even think about agreeing or disagreeing. That ends a lot of pointless fights.

And one thing doesn't change: review. The agent wrote it, you sign off. If code ships to production with concatenated SQL because "the AI did it," it's still your fault in the post-mortem.

My take

I think Karpathy is right, and I think the gap will keep growing for a while. Frontier tools are improving faster than the average perception can keep up. People who use the best setup every day keep pulling further away from those who tried once and gave up.

But the fix is boring and within anyone's reach: measure on your own code, with your own tests. Bun makes this easier because the loop is fast and the tooling comes built in. You don't have to believe me or anyone else. Run bun test and look at the result.

That's how I've been testing AI in my Bun and React Native projects. If you want to see where this is already running for real, take a look at my projects.

In the same week, one dev told me AI wrote an entire feature in Bun for them, and another swore it can't even handle CRUD.

Both were right. They were just using completely different tools with the same name: "AI".

Karpathy pointed this out: the gap between what people think LLMs can do keeps growing. A chat with no context gets imports wrong. An agent with repo access reads the error and fixes it on its own.

My way out was to stop giving opinions and start measuring. I take 5 to 10 real tasks I've already solved, write the tests, and run `bun test` every time I switch models.

One afternoon of homemade evals is worth more than a hundred threads on X. And the review is still on me: the agent writes, I sign off.

I wrote the full step-by-step on the blog. Have you ever measured AI on your own code?

#Bun #ArtificialIntelligence #LLM #SoftwareDevelopment #TypeScript