Science fiction is not a SaaS manual

You have heard someone in a meeting say the product will "solve the problem once and for all". That big promise tone. That slide with a movie quote on it. And then, six months later, the system has become a machine nobody understands, nobody can fix, and that treats the customer like a row in a spreadsheet.

There is a name for this. Harvard historian Jill Lepore has been saying that a good part of the tech industry is made of "bad readers". People who read science fiction dystopias and took them as instruction manuals. The warning became a roadmap. And that way of thinking trickles down, all the way to your SaaS, your automation, your support flow. That is why this matters to anyone running a business, not just to people who write code.

What "reading science fiction badly" means in practice

Science fiction is almost always a warning. The robot that decides on its own, the system that optimizes everything and crushes people along the way, the company that knows too much about you. The author is saying "look at the danger here". Part of the industry read it and thought "what a great product idea".

In the daily life of a SaaS, it shows up like this:

• The system decides on its own and tells nobody.
• The person using it cannot undo anything.
• Nobody can explain why that happened.
• Dashboard efficiency matters more than the customer on the other side.

None of this needs a robot or movie AI. A dumb auto-billing algorithm does plenty of damage on its own.

The case of the customer who vanished for no reason

I have seen this happen more than once. A subscription company built a "smart" rule: if a payment failed twice, access was cut immediately and the account went into a recovery queue.

Sounds reasonable on a slide. In practice, an old customer, four years with them, had a card failure on the issuing bank's side. Two attempts in two days. Access cut on a Tuesday morning, with their team in the middle of a delivery. They sent an email. It hit a bot. The bot replied that the account was "in recovery". Nobody on the team saw it.

The customer churned. And took a nearly closed referral along with them.

The mistake was not technical. The code did exactly what it was written to do. The mistake was a reading error: someone thought automating means taking the person out of the loop. Good automation is the opposite. It takes the boring work off the person and keeps them where the decision carries weight.

The fix was cheap, by the way. One extra rule: a customer with more than twelve months and a clean history does not lose access automatically, it becomes an alert in the accounts team's Slack. Two hours of work. Zero churn from that cause in the next six months.

Good automation gives people time to decide. Bad automation decides for them and disappears.

The part nobody talks about: "frictionless" is an ideology

There is a word that became dogma in the SaaS market: friction. Everyone wants to remove it. Fewer clicks, fewer screens, fewer confirmations, fewer questions.

Here is my strong opinion: zero friction is a dangerous idea when applied without judgment. Friction is what gives people time to think. That "are you sure?" screen before deleting an entire project exists for a very good reason.

When you remove all the friction, you did not remove the decision. You just handed the decision to whoever wrote the software. And whoever wrote the software was not in the room when your customer needed help.

The real work is choosing where friction stays. Signup? Remove it. Permanent data deletion, contract cancellation, mass sending to your whole list? Keep it. And keep it visible.

Democracy, in your context: who gets to disagree with the system

Lepore talks about democracy being undermined. That sounds far away from someone running a thirty-person company. But there is a very direct translation.

Democracy, inside a system, means having a mechanism for disagreement. It means being able to say "this result is wrong" and having someone listen. Ask this about your product:

• If the system gets something wrong with a customer, is there a path for them to contest it?
• Does that path lead to a person or to another bot?
• Does anyone on the team review the cases where automation was contested?
• Can you explain, in plain language, why the system made that decision?

If the answer to any of these is "no", you have a product problem. And, depending on your industry, a legal problem waiting around the corner. Brazil's data protection law, in article 20, gives people the right to request review of automated decisions that affect their interests. That is not philosophy. It has been law since 2020.

And where does AI fit in?

The temptation today is huge. Drop a model in the middle of the flow and let it answer, classify, prioritize, decide. It works well for plenty of things. I use it every day.

But language models are wrong with impressive confidence. They are wrong with good grammar, which is worse, because it looks right. So the rule I use on projects is boring and simple:

AI suggests, humans confirm, when the cost of being wrong is high.

Classifying a support ticket by topic: let the AI run, a mistake costs one reroute. Deciding whether a customer goes to legal collections: not without human review. Drafting a proposal: AI drafts, you review. Sending that proposal to the client on its own: no.

The question is not "AI or no AI". It is "how much does it hurt if this is wrong, and who pays that bill". Usually the one paying is the one with the least power in the system. That is exactly the warning science fiction was giving, and that a lot of people read backwards.

A five-minute test for your system

Take the most automated flow you have today. Could be billing, onboarding, email sends, signup approval. Answer three questions, without asking anyone:

1. What is the worst thing this flow can do to a good customer by mistake?
2. How long until someone on the team notices it happened?
3. What does the customer do while they wait?

If the answer to the second one is "when they complain", you do not have automation. You have a bet.

The fix is usually small. A log an actual person reads. An alert when the rule fires on an unusual case. An undo button. A limit: "if this will affect more than X customers at once, stop and ask". None of that is expensive. It is just less glamorous than the slide version.

The product left over after you drop the fantasy

Good technology ages well because it was built by people who respect whoever will use it. It is less exciting than promising to revolutionize an entire industry. It also causes far fewer headaches at three in the morning.

If you are building or fixing a SaaS and you suspect the automation is deciding more than it should, that suspicion is a good one. It is worth sitting down and mapping where the system speaks and where a person needs to speak. That is the kind of work I do with the people who call me: take a look at how I work and tell me which flow is keeping you up at night.

Your system cut off a 4-year customer because their card failed twice. The code worked perfectly. The customer churned.

Good automation gives people time to decide. Bad automation decides for them and disappears.

Removing all friction became a trend. But zero friction does not remove the decision, it just hands it to whoever wrote the software. And that person was not in the room when your customer needed help.

My rule on projects: AI suggests, humans confirm, when the cost of being wrong is high.

Quick test on your most automated flow: what is the worst thing it can do to a good customer by mistake, and how long until someone notices? If the answer is "when they complain", you do not have automation. You have a bet.

If that doubt just hit you, tell me which flow is keeping you up at night.

#SaaS #Automation #DigitalProduct #AI #TechLeadership