Key Takeaways

  1. You know an automation broke by opening the booking, the invoice or the message it should have produced and confirming it exists. The log only says it started.
  2. How often you check comes from what a miss costs. Anything carrying revenue gets checked several times a day.
  3. A pressure test is the only thing that finds a failure that reports success, because your reporting is built out of those same success messages.
  4. Agents run the evaluation. It costs little to nothing to have a system in place that always verifies your process is still active.
  5. A business that can’t see its system failing loses profit slowly and puts the blame on the wrong cause. Usually on its own people.

You know an automation broke by opening the booking, the invoice or the message it should have produced and confirming it exists. The log only says it started.

How often you check comes from what a miss costs. Anything carrying revenue gets checked several times a day.

A pressure test is the only thing that finds a failure that reports success, because your reporting is built out of those same success messages.

Agents run the evaluation. It costs little to nothing to have a system in place that always verifies your process is still active.

A business that can’t see its system failing loses profit slowly and puts the blame on the wrong cause. Usually on its own people.

What should you check

Check the output, and put a separate agent on it that has no reason to say it went well. The thing to look at is what the automation produced, not whether it ran, because those are different questions and only one of them costs you money. An automation reporting on its own health is the same process grading its own work, and it passes itself every time.

We run agents whose only job is checking the other agents, and what they look for is the same list any owner should look at:

  • An agent that has stopped reporting at all
  • A login that expired
  • A run that started, produced nothing, and closed
  • A lookup that has matched nobody for days
  • A message that was written and never sent
  • A booking the agent confirmed on the call that never landed on a calendar

Every agent has scheduled check-in times, and a missed check-in is the alarm. We also push manufactured traffic through the system during the day, so we see what an agent does with a live input instead of hearing about it from a customer.

I. A failing agent reports its own failure

An agent that hits a problem sends a text, says it’s urgent, describes what failed, and asks for next steps. If the failure touches money it calls instead of texting.

“If I see my agent’s name flash across my screen, I know something is out that requires an immediate decision.”

Jon Jack, Founder, Overtime GO

That decision could be handed to a protocol that resolves it without a person. We don’t allow it. Anything at that threshold gets a human.

II. Whatever fails, another agent is still reporting

Build the checking so no single failure takes out both the work and your view of the work. Checking that shares a dependency with the thing it checks goes quiet at the exact moment you need it most, and that silence reads as everything being fine.

III. The escalation matches the person who has to act

For us it’s a phone call, because a phone call gets answered and a dashboard gets opened tomorrow. Yours might be different. Some owners never pick up and read every text. Some want it in a thread their whole team can see. Pick the channel the person actually responds to. An alert sent to a channel nobody opens is the same as no alert.

How often should you check your system

Check it as often as a miss would cost you. Anything touching money or a customer gets looked at several times a day and some of it hourly. Everything client facing gets a pressure test weekly. The whole system gets an extensive round monthly.

Frequency comes from what a miss costs, never from a calendar.

How oftenWhat
Several times a day, some hourlyAnything touching money or a customer
Weekly pressure testsEverything client facing
MonthlyExtensive pressure testing across the whole system

A missed booking is the difference between a three figure client and a four or five figure one. Anything carrying revenue gets looked at several times a day.

The monthly audit drives into the quiet lanes. The parts nobody hears from are usually the customized ones, the piece of the process that was tailored to how this business actually works, and that piece is often the make or break. When it stops running properly the system keeps going and the work stops meeting the standard the company expects. Nothing announces that. You go find it, at the level of every individual detail.

What a pressure test is, and how to run one

A pressure test is a real transaction you push through your own system on purpose to see what comes out the other end. A call, a form, a booking, a message. Not a check that the system is up.

We place a real call to our own line as a customer would, with the scenario written in advance. We let the agent handle it. Then we open the calendar, the customer record and the message log and confirm everything that call should have produced is actually there, with the right name and the right time on it. Anything missing is a finding, whatever the transcript says.

That’s how we found one of our phone agents treating every caller as a stranger. It was supposed to know a returning customer and it never knew one, on any call, for a month. Every log said the check ran and succeeded. Only pushing a real call through and reading the result showed it.

What does a flawed system look like

A healthcare operation came to us. When we came aboard the owner was overwhelmed, because either they did everything themselves or nothing got done properly.

They had no system. They had a stack of separate tools bought at different times, expected to work together, and a staff who didn’t talk to each other. Inside that stack, calls from caregivers ringing to say they couldn’t make a shift were being answered and never passed on.

Shifts went uncovered. Clients were left waiting for care that never arrived. It happened again and again and nobody could trace it back to the tools.

They concluded the caregivers were the problem. Their own people took the blame for a failure none of them could see, and because everyone was buried in the day to day, it stayed that way until we came in and ran an audit.

We rebuilt it. A call out now goes into a decision protocol instead of a voicemail box. A text goes to available staff from the operator’s own number, so the caregiver is answering what looks like the scheduling team. Replies update the schedule automatically. When a client cancels for the day, the system finds who was assigned to them, tells them they’re off that shift, and asks them to stand by in case another case calls out.

That’s one healthcare operation. The same shape fits any business where a person is booked to be at a job and the schedule has to move when they can’t make it.

When should you trust your system

A new system gets tested, broken, fixed, and tested again. That stretch matters more than the build, because it’s where the problems cost you nothing to find. A team that shows up expecting to find problems finds them. A team that shows up expecting it to work hears about them from a customer instead.

The system earns your trust once it has run right long enough that you stop finding things. Until then you check it, and you check it more often than feels necessary.

How to evaluate your automations

You evaluate your automations by writing down every task your business repeats, sorting each one into work an agent can take and work that has to stay with you, then checking the output of everything in the first pile. The sort is the part most owners skip, and nothing downstream of it works until it exists.

Pile one, this can be handed off.

An AI agent runs it, or an assistant runs it, and you check the result it produced. Most administrative work belongs here, and it’s the pile that buys back your week.

Pile two, this has to be you.

Judgement, relationships, pricing an unusual job, anything where being wrong costs more than the hour.

Owners tell us the same thing in different words. If the administrative work went away, they could go be good at selling, at marketing, at getting in front of clients and prospects. The sort is what makes that real, because you can’t hand off what you’ve never written down.

Everything in pile one gets four questions:

  1. What should this produce? Name the booking, the invoice, the message. If you can’t name it, you can’t check it.
  2. Did it produce that last week? Open the calendar or the inbox and count.
  3. When it looks a customer up, is it finding them? A lookup that matched nobody thirty times running is broken.
  4. When does its login expire? Expired credentials kill more automations than bad logic.

Action item: write the list of repeated tasks this week and sort it into the two piles. Nothing else on this page works until that list exists.