AI agents: the ranking has a new owner again

You approved an automation project in April based on the "best model on the market". In August, the best model is a different one. And probably Chinese.

Qwen3.8 Max took the top of the Artificial Analysis agent index, which measures how well a model can carry out multi step tasks on its own, not just answer questions nicely. It is the third change of leader this year. If you own a business and are trying to decide where to bet on AI agents, this news feels important. It matters less than it seems, and I will explain why.

What that ranking really measures

There is a big difference between a model that talks well and a model that works well.

Talking well is answering a question. Working well is getting "consolidate the orders in this spreadsheet, check which ones have a missing invoice and send me a summary", and doing all three things in the right order, without inventing data along the way.

The agent index tries to measure the second thing. It puts the model to work with tools, browsing, writing code, querying systems and completing long tasks. It is the test closest to what you actually want: someone who does things, not someone who has opinions.

And here is the curious part. In this kind of test, the gap between first place and fifth place is small. We are talking about a few percentage points. First place makes the headline. In practice, if your process works with the fifth, it works with the first, and the other way around.

Why switching models rarely fixes your problem

Let me be direct: in almost every automation project I have seen fail, the model was not the culprit.

The usual culprits are:

• Messy data. The agent reads the spreadsheet where the same client shows up as "Silva Ltd", "Silva ltd" and "silva corner shop". No model in the ranking fixes that on its own.
• Undefined process. Nobody can explain in writing what happens when an order arrives with no tax ID. If a person does not know, the agent will not know either.
• No access. The system holding the information has no integration, and the answer is "someone exports it by hand every Friday".

Those three problems cost more than any difference between models. And they are tedious to solve, which explains why so many people would rather argue about which AI is best.

Switching models is easy. Fixing the process is the real work.

The quote that used to take two days

A client of mine, a distributor, had a flow like this. The sales rep would get an order by WhatsApp, usually a photo of a handwritten list or a PDF from the client. He would type everything into the system, check stock, calculate shipping, put the quote together and send it back. Average of two days per quote on busy days.

We built an agent for the mechanical part: read the order, identify the items in the catalog, check stock and generate a draft quote. The sales rep reviews and approves. It went down to about twenty minutes.

What made that work was not the model. It was the month before, spent on three unglamorous things:

1. We standardized the product catalog, because the same item had four different names.
2. We defined in writing what the agent does when it is not sure (answer: it stops and asks, it never guesses).
3. We put the sales rep at the end of the flow, with veto power.

The model used there has already been swapped twice since then. The process is still the same, and the swap took one afternoon. That is the point.

So the ranking is useless?

It is useful. Just not for what most people think.

It is useful for understanding where the market is heading. And the direction is clear: open Chinese models, like Qwen, are catching up with the American ones and costing a fraction of the price. That changes the math on your project.

An agent running a thousand tasks a day with an expensive model can cost a few thousand a month. The same agent with a competitive open model can cost hundreds. When the quality gap is two points on a benchmark and the price gap is five times, the decision gets a lot less philosophical.

It is also useful for calibrating expectations. The best models today get around 60% to 70% of complex agentic tasks right in a test environment. Not 99%. If someone sells you an agent that "does everything on its own with no supervision", ask which benchmark backs that up. The answer will be interesting.

How to decide without becoming a hostage to headlines

A practical way to look at this, in order:

Pick the process, not the technology. Take a task that is repetitive, has clear rules and eats up expensive people's hours. Quotes, email triage, document checking, first line replies. If the task needs fine judgment or the cost of an error is too high, leave it for later.

Build it so the engine can be swapped. That is the builder's job, but you can hold them to it. The right question for your vendor is: "if a better or cheaper model comes out in three months, how much work is it to switch?". If the answer is "we redo everything", the project is badly designed.

Measure before you start. How long does this task take today? How many errors happen per month? Without those two numbers, you will not be able to prove the automation was worth it, and you will end up deciding by gut feeling.

Leave a human at the decision point. Not because the AI is dumb, but because the responsibility is still yours. The agent prepares, a person approves. It works better and you sleep better.

What I would do in your place

Ignore the ranking for six months. It will change again, probably sooner than that.

Use that time to pick a process, document how it really works (not how the manual says it works) and clean up the data it uses. When that is ready, plugging an agent on top becomes a matter of weeks, and you will be able to swap models as many times as you want without redoing anything.

The hard part of automation was never the AI. It was always understanding your own process well enough to explain it to someone who has never worked at your company. It just happens that this someone is now a machine, and a very literal one.

If you have a process in mind and want an honest opinion on whether it is worth automating, tell me what you do today. Sometimes the answer is a better spreadsheet, and I will say that too.

In April you approved an automation project with "the best model on the market". In August the best one is different, and probably Chinese.

Qwen3.8 Max took the top of the agent ranking. Third change of leader this year.

But in almost every automation project I have seen fail, the model was never the culprit. It was messy data, a process nobody can explain in writing and systems with no integration.

A client of mine cut quotes from 2 days to 20 minutes. What made it work was a month spent cleaning the catalog and defining rules, not the trendy AI. The model has changed twice since then and the switch took one afternoon.

Ignore the ranking for six months. Use that time to understand your own process.

If you have a repetitive task in mind and want an honest opinion on whether it is worth automating, message me. Sometimes the answer is a better spreadsheet, and I will say that too.

#ArtificialIntelligence #Automation #Productivity #Technology #Business