JPMorgan Chase outage: a resilience lesson for devs
A technical problem at JPMorgan Chase slowed down online banking at other banks in the United States. The customers of those banks never opened an account at Chase. The JPMorgan Chase outage is banking news, but the lesson applies to anyone who writes software: another company's system can take yours down.
The AP report lists institutions like Wells Fargo and Truist among those affected. I won't speculate about the internal cause. Only the Chase team knows that. What matters here is the pattern. It shows up every week in projects much smaller than a bank.
What happened to Chase and the other banks
In short: one of the biggest banks in the world had a technical problem and the slowdown reached competing banks. For the end customer, the app opened, but transfers and balance checks took forever or froze.
This only happens because the banking system is a network of dependencies. Payments, clearing and transfers between institutions go through shared points. One big node chokes and its neighbors feel it.
Now swap "bank" for "payment API", "login provider" or "transactional email service". Same movie, smaller budget.
Why does a failure in one system take down another?
It's almost never the failure itself that takes your system down. It's the way your code waits for it.
The classic scenario:
- Your backend calls an external API.
- The API doesn't go down. It gets slow. It responds in 30 seconds instead of 300 ms.
- Each of your users' requests holds an open connection while it waits.
- The connection pool fills up, memory climbs, and endpoints that don't even use that API start failing.
Notice the external API never went down. It just got slow. And slow is worse than down, because an outage throws an error right away, while slowness holds resources for a long time.
Slowness spreads. Fast errors don't.
I've seen this happen with a payment gateway on an e-commerce site. The gateway started responding in 40 seconds on a Friday night, and the product page, which had nothing to do with payments, went down with it. The culprit was a fetch with no timeout in the checkout, holding every worker.
Timeout: the line of code nobody writes
fetch has no timeout by default. Not in Node, not in Bun, not in the browser. If the other side never responds, your call never ends either.
The fix fits in one line:
const res = await fetch('https://api.partner.com/balance', {
signal: AbortSignal.timeout(3000),
})
Three seconds is a reasonable guess to start with. The right number comes from your p99 in production. If the API usually responds in 400 ms, waiting 30 seconds helps no one.
I use Bun in my backends and the API is the same. AbortSignal.timeout works the same way. A modern runtime gives you the tool, but it won't write the line for you.
Retry without idempotency is a delayed bug
The reflex after a timeout is "try again". In banking, that's dangerous. If the first transfer request reached the server but the response got lost, the retry transfers twice.
That's why any serious financial API accepts an idempotency key:
const idempotencyKey = crypto.randomUUID()
async function transfer(payload: Transfer, attempts = 3) {
for (let i = 0; i < attempts; i++) {
try {
return await fetch('https://api.partner.com/transfers', {
method: 'POST',
headers: { 'Idempotency-Key': idempotencyKey },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(3000),
})
} catch {
await Bun.sleep(2 ** i * 200 + Math.random() * 100)
}
}
throw new Error('Partner unavailable')
}
Three details matter here:
- The key is generated only once, outside the loop. If you generate a key per attempt, it protects nothing.
- Exponential backoff with jitter. If a thousand clients retry in the same millisecond, you help take down a service that was already struggling.
- A limit on attempts. Infinite retry is a denial of service attack you run against your own vendor.
In PostgreSQL, the key becomes a UNIQUE constraint on the operations table. That way the database guarantees the operation happens only once, even if the code gets it wrong.
Circuit breaker: stop knocking on the door
If the dependency failed 20 times in a row, the 21st call will probably fail too. The circuit breaker stops calling for a while and returns an error right away.
The minimal version is short:
let failures = 0
let openUntil = 0
async function callPartner(url: string) {
if (Date.now() < openUntil) throw new Error('Circuit open')
try {
const res = await fetch(url, { signal: AbortSignal.timeout(3000) })
failures = 0
return res
} catch (err) {
if (++failures >= 5) openUntil = Date.now() + 30_000
throw err
}
}
This is in-memory state, per instance. If you run ten instances, each one learns on its own. For most projects that's enough. When it isn't, that's when it makes sense to keep the state in Redis or use a ready-made library.
You win twice. Your system stops wasting resources on waiting, and the vendor gets room to recover. Anyone who has kept hammering a dead API knows they're usually part of the problem.
What changes in your workflow
Code is half the conversation. The other half is product and operations.
Map your dependencies. List every external API your app calls. For each one, answer: if it gets slow for an hour, what does the user see? If the answer is "the whole screen freezes", you've found your next incident.
Degrade gracefully. When the balance service is slow, the app can show the last cached value with a note like "updated 5 minutes ago". That's much better than an endless spinner. Users accept slightly old data. What they won't accept is an app that looks dead.
Isolate resources. The checkout connection pool shouldn't be the same one the product page uses. When one compartment floods, the whole ship doesn't sink. This pattern is called a bulkhead.
Test failure on purpose. In staging, swap the partner URL for an endpoint that takes 60 seconds to respond. In ten minutes you'll find out whether your timeouts really exist or only exist in your head.
A slow test endpoint in Bun:
Bun.serve({
port: 4000,
async fetch() {
await Bun.sleep(60_000)
return new Response('finally')
},
})
Point the app to localhost:4000 and see what happens. It's the cheapest resilience test there is.
My take
The Chase episode is an uncomfortable reminder: you don't control your system's availability. You only control how it reacts when something you depend on fails. And something always fails. It could be a bank, a cloud, or that SMS SaaS you integrated one afternoon.
My strong opinion: timeouts should be mandatory on every fetch, and I would block any PR without one. It sounds extreme until the first Friday night you stay up watching workers stuck waiting on an API that never answers.
The good news is that none of this needs fancy architecture. Timeouts, retry with idempotency, a twenty-line circuit breaker and a cache to show stale data already cover most cases. If you want to see how I apply this in production, take a look at my projects.
LinkedIn summary
A slow API takes down more systems than an API that is down. JPMorgan Chase had an outage and online banking at other banks slowed down. The customers of those banks never opened an account at Chase. Your project has the same problem. The payment gateway starts responding in 40 seconds, every worker gets stuck waiting, and the product page goes down with it. fetch has no timeout by default. One line with AbortSignal.timeout already prevents a lot of the damage. Add retry with an idempotency key, a simple circuit breaker and a cache to show the last known data, and you cover most cases. I wrote the step by step with code on the blog. If you've ever spent a Friday night watching stuck workers, you'll relate. #Resilience #SoftwareDevelopment #Backend #TypeScript #Engineering