AI blind spots: the lesson from a Go master

Have you ever approved something the AI handed you without checking? A piece of text, a spreadsheet, a reply to a customer. I have. And most of the time it worked out, which is exactly the problem. Because when AI blind spots show up, they don't show up as an obvious error. They show up as a polished, confident, wrong answer.

In July, South Korea's Shin Jinseo, the world number 1 in Go, beat KataGo, one of the strongest programs out there, in a game with a two-stone handicap. For anyone not following: since 2016, when AlphaGo beat Lee Sedol, the score flipped. Machine beats human at Go. It's not a debate, it's settled. That's why the news traveled the world.

And I think it says more about your business than about a board.

The score isn't the interesting part

A two-stone edge is a lot in Go. A more skeptical reader will already say: "so he didn't really win". Fair. The handicap exists precisely because the program is stronger than any person alive.

But look at what that means in practice. Somebody measured the distance between human and machine, put a number on it, and found where the human can still win. That's valuable information. It's different from "AI is better" or "AI is a fraud". It's a map.

Most of the companies I talk to don't have that map. They either think AI solves everything, or think it solves nothing. Both positions cost money.

A strong machine isn't a machine without holes

There's a case from 2023 I like to tell because it's too concrete to ignore.

Researchers at FAR AI trained a program just to find flaws in KataGo. Not to play well. To find flaws. They found a pattern: if you build a big, loose group of stones and surround it from the outside in a specific way, the program simply doesn't see the threat. It rates the position as calm right up to the moment it has already lost.

Then came the scary part. An amateur player named Kellin Pelrine learned the maneuver and pulled it off with his own hands. He won 14 of 15 games against a superhuman-level bot. No computer helping during the game. An amateur beating what nobody was beating.

The program didn't get weak. It stayed better than everyone at the normal game. There was just a side door, and nobody had looked at it because everyone was busy admiring the brute force.

A system that's strong on average can be a disaster in a specific case. And it's always the specific case that shows up in front of the customer.

Where this happens in your company

Translating to your day to day, no board involved.

You put an AI on the website chat. In 95% of the questions it does fine. In the other 5%, it invents a delivery date that doesn't exist, or promises a payment term you don't offer. It does that with the same confidence as the other 95%. No shake in the voice.

You put an AI on invoice classification. It works beautifully until a supplier shows up issuing in a weird format, and it starts dumping everything into the wrong category. Silently. You find out at the quarterly close.

You put an AI on resume screening. It learns from history. If the history has bias, it reproduces the bias and still gives you an articulate justification for every choice.

The pattern is always the same. The error doesn't shout. It disguises itself as a correct answer.

What separates who profits from who gets burned

The difference, in practice, isn't the model you picked. It's who bothered to look for the side door before the customer found it.

When I build an AI automation for someone, I do three things that sound boring and are the ones that save the project:

• I define where it can't decide on its own. Amount above X, long-standing contract customer, cancellation request, complaint with an aggressive tone. That goes to a person. Always. The machine prepares, the human signs.
• I leave a trail of everything. Every generated answer gets logged with what went in and what came out. Without that you can't even find out you have a problem, let alone fix it.
• I test with the ugly cases on purpose. Not with the ten pretty examples. With the badly written question, the crooked PDF, the annoyed customer, the request with two contradictory things inside it.

That third item is the equivalent of hiring Pelrine to try to take you down. It costs a few hours. It avoids the kind of error that turns into a screenshot on Twitter.

"So AI isn't worth it?"

It is, very much so. I make a living building this.

It's just that the real gain is almost never in replacing a whole person. It's in clearing away the tasks nobody enjoys doing that eat up the day. Reading fifty emails and pulling out the three urgent ones. Turning a meeting recording into a summary with the open items. Filling in the first version of a proposal for someone to review in five minutes instead of writing it from scratch in an hour.

In those cases the cost of an error is low and the human reviewer is already there, naturally, no new process needed. That's where the math works out fast.

The problem shows up when the company jumps straight to "the AI answers the customer on its own" because it read in a post that it's doable. It is doable. It's just that then you need the boring part, and the boring part is what nobody posts.

Something I always say: if you can't explain in one sentence what happens when the AI gets it wrong, you're not ready to put it in front of your customer. That's not pessimism, it's the same question you'd ask before hiring an intern to answer the phone.

The human didn't become a decorative piece

Shin Jinseo didn't win because he's faster than a server. He won because he understands the game in a way that lets him pose a question the machine can't answer properly.

That's exactly your role now. You know your customer, your industry, the exceptions, the agreements that aren't written down anywhere. The AI knows none of that. It knows patterns. And your business is made of patterns as much as exceptions.

The person who only knows how to use the tool becomes a commodity fast, because the tool is available to everyone at the same price. Whoever knows where the tool fails and builds the process around that, that person stays valuable. The advantage isn't in the access. It's in the judgment.

That holds for a Go player and it holds for someone running a twelve-person company.

If you're thinking about putting AI somewhere in your process and want to do it without discovering the holes alongside your customer, tell me what you have on your hands. I usually start with the most useful question of all: what happens the day this gets it wrong?

Have you ever approved something the AI handed you without checking? I have.

And it worked out almost every time, which is exactly the problem. An AI error doesn't shout. It arrives polished, confident and wrong.

In 2023, researchers found a gap in KataGo, one of the strongest Go programs in the world. An amateur learned the maneuver and won 14 of 15 games against a superhuman bot. No computer helping.

The system didn't get weak. There was just a side door, and nobody had looked at it.

In your company it's the same: the AI gets 95% of the chat right and in the other 5% it invents a deadline you don't offer, with the same confidence as always.

If you can't explain in one sentence what happens when the AI gets it wrong, it isn't ready to sit in front of your customer yet.

Thinking about putting AI somewhere in your process? Tell me what you have on your hands. I always start with the same question: what happens the day this gets it wrong?

#ArtificialIntelligence #Automation #Technology #SmallBusiness #AIinBusiness