Every software company has incidents. When you move tens of millions of transactions each month, things will break. Some problems are simple. Others are deep or strange. What matters is not perfection. What matters is how you respond.
We use a simple and disciplined process. It has served us since the first days of the company. It keeps us calm. It keeps us fast. It also mirrors the way strong logistics teams solve operational failures.
Here is how it works.
The Cultural Foundation: Blamelessness
Incidents create fear when blame enters the room. Fear slows people down. Fear makes them hide facts. Fear breaks teams.
We do not blame during an incident. We focus only on the facts and on the fix. Nothing else matters.
If a shipment was not booked, that is a fact. We fix the missed booking. We move forward. We do not shame the person who made the mistake. When people know they will not be blamed, they speak up early. The whole team becomes stronger.
A Low Bar for Declaring Incidents
We declare incidents quickly. Even small issues count. If a problem affects one customer and we are not certain it is isolated, we treat it as an incident.
This helps us catch larger issues early. It also gives us constant practice. We run the playbook often. The team knows the steps by heart. When a major issue appears, the group moves in sync. We do not panic. We execute.
If you do not practice, you do not play. We choose to practice.
The Incident Manager
Every incident has one leader. We call this person the Incident Manager.
They do not fix the system. They manage the situation. They track actions. They keep notes. They ask clear questions. They pull the team back when the discussion drifts. They protect the process.
This role creates order. It frees the technical team to focus on the work.
Work Only From Facts
Hunches waste time. Facts save time.
When someone has a theory, they test it. They check the logs. They confirm the timeline. They reproduce the behavior. When the theory becomes a fact, the team acts. Until then, it sits on the side.
We run incidents with everyone present. The facts are visible to all. Anyone can ask for proof. Anyone can challenge a detail. This keeps us honest. It keeps us fast.
Document Everything
The log is our anchor. We record every action, every test, every result, and every fix.
This prevents confusion. It prevents people from repeating work. It helps us track patterns. It gives the next person a clear view when they join the call. It forms the story we use in the post-incident review.
Before we built this habit, long incidents became chaos. People tried the same commands twice. People forgot what they changed. Hours were wasted. Good documentation ended that.
Why This Applies to Supply Chains
These rules come from software. They also fit logistics.
A missed booking, a misrouted container, or a late handoff is not a moment for blame. It is a moment for clear thinking. Strong leaders pull the group together. They look at facts. They document everything. They act with discipline. They improve the system with every incident.
Supply chains and software share the same trait. Both are large and fragile. Both break in unexpected ways. A good incident process makes them stable again.
Final Thoughts
Incidents happen. Disorder does not have to follow.
A blameless culture builds trust. A low bar for declaring incidents creates practice. A clear incident leader creates structure. A fact-driven process creates speed. Strong documentation creates memory.
This is how we work at Chain.io. It is why we stay calm when systems fail. It is why our customers trust us to keep their supply chains moving.
