Trusting AI too much: the Pentagon lesson for your SaaS

You know that report the AI generated? The one you glanced at, thought looked good, and sent to the client without checking? Yeah. Trusting AI too much is a habit that starts small and grows without anyone noticing. This week brought the heaviest example you can imagine.

According to a Bloomberg report, the Pentagon admitted that excessive reliance on artificial intelligence contributed to a missile strike that hit a school in Iran. I won't get into the geopolitics. It's not my field and it's not the topic here. What interests me is the mechanism behind the mistake, because it's exactly the same one I see in companies that use AI every day.

What happened, in simple terms

An organization with the most expensive resources on the planet, with trained people and strict protocols, let a critical decision lean too heavily on what the system said. At some point in the chain, the machine's recommendation became truth without the questioning that should have come.

This has a name. Researchers call it automation bias: the tendency to accept what an automated system says, even when there are signs something is wrong. The more the system gets it right, the more we relax. And the more we relax, the more expensive the mistake is when it finally happens.

Here's the point I want you to take away: if this happens in a place with that level of control, it happens in your company too. Just with much smaller consequences. And that's exactly why nobody notices.

Why trusting AI too much is so easy

Modern AI has a presentation problem. It gets things wrong with the same confidence it gets them right.

An insecure intern says "I think this is it, but double-check." A language model delivers clean, formatted text with an expert tone. Even when it made up half of it.

Add three very human things to that:

• Rush. AI exists to save time. Reviewing feels like undoing the gain.
• Good track record. Got it right the last 50 times? The 51st goes straight through.
• Diluted responsibility. "The system suggested it." Nobody feels they own the decision.

None of this is stupidity. It's fatigue, routine and the wrong incentives. That's why the problem isn't solved by asking people to "pay more attention." It's solved with process.

AI doesn't take responsibility away from anyone. It just makes it easier to forget whose it is.

Where this shows up in a real SaaS

Let me bring it down to earth. These are cases I've seen or that come up all the time in conversations with clients.

Support triage. A management SaaS uses AI to sort tickets by urgency. It works well. Until a big client writes "I can't issue invoices" in a polite, calm way, and the AI marks it as low priority. The ticket sits in the queue for two days. The client cancels.

Credit or risk analysis. A billing system uses a model to suggest who should be blocked for late payment. The model learned from old data and blocks a client who had just renegotiated. The finance team approves in bulk, because there are 300 cases a week and "the system already filtered them."

Automated replies. A chatbot answers questions about plans and pricing. One day it invents a discount that doesn't exist. Now you have a screenshot going around and a customer demanding you honor the promise.

Do the quick math. If your AI is right 97% of the time and processes 2,000 items a month, that's 60 errors. Every month. The question isn't whether it makes mistakes. It's which of those 60 errors are expensive and whether anyone is looking at them.

When AI can decide on its own (and when it can't)

Here's my strong opinion: most companies put human review in the wrong place. They review what's cheap and let what's expensive slip through.

A simple way to sort it out is to ask two things about each task:

1. Is the error reversible? A marketing email with a misplaced comma, yes. A blocked account, a denied refund or a wrong charge, much less so.
2. Who feels the error? If it's just you, internally, you can tolerate more. If it's the customer, the cost goes up right away.

Reversible and internal error? Let the AI run on its own and review by sampling. Hard to undo or reaches the customer? The AI suggests, a person decides. Simple as that.

The Pentagon was in the worst possible quadrant: an irreversible error with real victims. Your SaaS will probably never get anywhere near that. But the "irreversible and reaches the customer" quadrant exists in every business. And it's usually exactly where automation went first, because that's where the volume was.

How to use AI in your SaaS without losing control

I'm not saying you should drop AI. I work with it every day and I think companies that ignore AI will fall behind. The point is to design its use properly. In practice, this is what I usually set up:

Let the AI show its doubt. Instead of just "urgent" or "not urgent," ask for a confidence level. Anything below a threshold goes to human review. That cuts review volume and focuses on what matters.

Put a brake on the expensive points. Blocking, refunds, sales promises, plan changes. At these points the AI prepares everything and a person presses the button. It takes 20 seconds. It saves weeks of headaches.

Review by sampling, even when everything looks fine. Pick 20 random cases a week and check them. It's the only way to notice quality has dropped before the customer does.

Record who approved. It looks like bureaucracy. It isn't. When every decision has a name next to it, people start reading what they approve again. Funny how that works.

Limit what the chatbot can promise. Price, deadlines, discounts and cancellation policy should come from a fixed source, not from the model's creativity. Creative AI is great for drafting a post. For a price list, go with the most boring AI possible.

The cost of doing nothing

One answer I hear a lot: "but it hasn't caused any problems so far." That's exactly what makes automation bias dangerous. It gives no warning. The system works, and works, and works. And the day it fails is the day nobody was looking, because everyone had learned they didn't need to look.

You don't need an AI ethics committee or a six-month consulting project. You need a few questions asked before you turn on the automation: where it gets things wrong, how much that error costs, and who checks. That fits in a one-hour meeting.

If you already have AI running in your product or your operation and you can't say where those brakes are, it's worth stopping to map them. It's the kind of work I do with companies that want to automate without becoming hostages to the system. If you want to understand how I work, take a look at who I am.

The Pentagon admitted that trusting AI too much contributed to a strike that hit a school in Iran.

I won't talk about geopolitics. What interests me is the mechanism, because it's the same one I see in companies every day: the AI gets it right 50 times and by the 51st nobody checks anymore.

AI gets things wrong with the same confidence it gets them right. And "the system suggested it" doesn't take responsibility away from anyone.

If it's 97% accurate on 2,000 items a month, that's 60 errors. The question is which ones are expensive and who's watching.

My rule: if the error is reversible and internal, the AI runs on its own. If it's hard to undo or it reaches the customer, the AI suggests and a person decides.

I wrote about how to put these brakes in your SaaS without losing the gains of automation. The link is in the comments.

#ArtificialIntelligence #SaaS #Automation #RiskManagement #Technology