Back

The order that cost more to fix than it did to fulfil

Manual provisioning errors are rarely catastrophic on their own. A missed field, a wrong product code, a duplicate order. Each one looks manageable in isolation. It’s only when you add up what fixing them actually costs — engineer time, customer service, rebilling, goodwill credits — that the scale of the problem becomes visible.

There’s an order sitting in a queue somewhere right now that was entered twice. Same customer, same address, slightly different product code because two different sales agents took the call on different days. One will provision successfully. The other will fail partway through, leave an orphaned VLAN assignment in the network and generate a support ticket that three people will touch before anyone works out it’s a duplicate. The customer will have a slightly degraded experience for reasons they can never quite explain. Eventually it’ll get sorted.

Total cost of the failed order: probably somewhere between £80 and £150, depending on how long the support ticket lives. Cost of the original successful order: around £12 in processing and provisioning time. The error cost ten times more to fix than the original transaction cost to fulfil. And because it didn’t produce a single dramatic incident — it was just a slow, annoying, hard-to-attribute drain on several people’s time — it will never appear in any report as a provisioning error. It will just be absorbed into “operations.”

This is the real shape of the manual provisioning problem. Not one catastrophic failure, but a continuous low-level bleed of errors that each look minor in isolation and collectively cost a significant amount of money and margin.

Why errors happen where they do

Provisioning errors cluster around handoffs. The moment an order moves from one system to another, from one team to another, or from one format to another, is the moment something goes wrong. Not always. Not even usually. But often enough that at any meaningful scale, the errors are a predictable feature of the process rather than an occasional aberration.

The classic handoff points in a manual provisioning operation look something like this. An order is taken by a sales agent and entered into the CRM. Someone in provisioning picks it up and re-enters the relevant details into the provisioning system — because the two systems don’t talk to each other directly. The provisioning system generates a job for a field engineer. The engineer completes the job and records the outcome on a paper job sheet or a mobile app that doesn’t integrate with the OSS. Someone transcribes the completion back into the system. The billing team picks up the completed order and raises the first invoice.

That’s four or five handoffs for a routine residential installation. Each one is an opportunity for a field to be missed, a product code to be wrong, a completion to be recorded against the wrong address, or a job to be marked done when it wasn’t quite. Most of the time, nothing goes wrong. But “most of the time” isn’t good enough at scale.

What a provisioning error actually costs

Most operators have a rough sense that errors are expensive, but the actual cost per error is rarely calculated. When you break it down, the numbers are usually higher than the informal estimate.

Take a failed installation — the engineer arrives, something in the provisioning data is wrong, the job can’t be completed. Here’s a realistic cost breakdown:

Cost componentNotesEst. cost
Failed engineer visitTravel + time, no productive outcome£95–£140
Investigation timeNOC or back-office identifying root cause£25–£40
Data correctionFixing the underlying provisioning error£15–£25
Rescheduled visitSecond engineer dispatch£85–£120
Customer service contactCall or chat handling, apology, explanation£8–£15
Goodwill credit Often applied on delayed installs£10–£30
Total per failed installvs. ~£12 for a clean install£238–£370

That’s a single failed installation. Not a systemic crisis. Just one order where something was entered incorrectly and an engineer turned up unprepared. At an operator processing five hundred installations a month with a 3% error rate — which is not an unusual figure for manual operations — that’s fifteen failed installs a month, somewhere between £3,500 and £5,500 in avoidable costs, every single month.

Annualised, that’s £42,000 to £66,000 in waste from a single error type at a single operator. Before you count the billing errors, the duplicate orders, the wrong product activations and the misattributed completions.

The errors you know about and the ones you don’t

Failed installations are visible. The engineer reports the failure, a ticket is raised, someone investigates. The error is known and counted, even if the full cost isn’t always attributed correctly.

The more insidious errors are the ones that don’t produce an immediate visible failure. The order that provisioned the wrong speed tier — the customer got 80Mbps when they ordered 160Mbps and they haven’t noticed yet because they don’t speed-test their connection. The billing record with the wrong start date, generating a small discrepancy that accumulates slowly until someone raises a query six months later. The ONT registration with a transposed serial number that will cause problems if the device ever needs to be remotely rebooted.

These errors live in the system undetected for weeks or months. When they eventually surface — through a customer complaint, a billing audit, or a field engineer who can’t find the device they’re supposed to be working on — the cost of resolution is significantly higher than it would have been if the error had been caught at the point of entry.

Why “it’s just human error” is the wrong diagnosis

The standard response to provisioning errors in manually-operated environments is to improve training, tighten checklists and exhort people to be more careful. This is understandable, but it fundamentally misdiagnoses the problem.

People make errors at a predictable rate when performing repetitive data entry tasks. This is not a character flaw. It is a well-documented feature of human cognition. Asking people to be more careful reduces the error rate slightly, temporarily, until the pressure of volume or the passage of time returns it to baseline. Training has the same trajectory.

The correct diagnosis is that the system is designed in a way that requires humans to transcribe information between systems, and transcription at volume produces errors. The fix is not better transcription. The fix is no transcription — an integrated system where the order flows from entry to provisioning to billing without being re-entered anywhere.

This seems obvious when stated plainly. Yet an enormous number of operators are still running provisioning operations built around exactly this pattern — because the systems were built at different times, by different teams, and the integration work has always been “next quarter’s project.”

The second-order costs that don’t appear in any report

The direct cost of a provisioning error is measurable, if someone bothers to measure it. The second-order costs are harder to see and usually don’t appear anywhere in management reporting.

The churn contribution. A customer whose installation failed, who waited two weeks for a rescheduled visit, who received a goodwill credit that acknowledged something went wrong — that customer is statistically more likely to leave at contract end than a customer whose installation went smoothly. The provisioning error doesn’t appear in the churn data. It happened months ago. But it contributed.

The engineering team capacity drain. Field engineers who spend a significant proportion of their time on failed-and-rescheduled jobs are less productive than engineers whose jobs are correctly specified first time. The capacity loss doesn’t show up as a provisioning cost. It shows up as “we need to hire another engineer” or “we can’t hit the installation SLA.”

The management time overhead. Provisioning errors generate noise — tickets, queries, escalations, exception handling. Someone senior enough to resolve them has to touch each one. That time has a cost that’s almost never tracked back to the original error.

What the fixed version looks like

An automated provisioning operation doesn’t eliminate errors entirely — nothing does. But it changes where errors happen and dramatically reduces their frequency and cost.

When an order is entered once and flows automatically from order management through to provisioning and billing without being re-keyed, the transcription errors disappear. When the provisioning system validates the order against live network inventory before accepting it, the category of errors where an engineer is dispatched to a job that can’t be done disappears. When the completion of a field job automatically closes the order in the OSS and triggers the billing record, the manual completion transcription errors disappear.

The errors that remain in an automated system are structural — wrong product in the catalogue, wrong address geocoded, customer gave an incorrect serial number. These are much harder to eliminate entirely, but they’re also much easier to detect quickly because the system is processing them consistently rather than inconsistently.

The error rate doesn’t go to zero. But a 3% error rate becomes a 0.3% error rate. And the cost per error goes down because detection happens earlier, before the engineer visit, before the customer notices, before the billing discrepancy has been accumulating for three months.

Measuring before fixing

The operators who make the most progress on provisioning error rates are the ones who start by measuring properly. Not just counting the visible failures, but auditing for the silent ones. Checking a sample of completed orders for accuracy against what was actually provisioned. Running a billing audit against the provisioning records. Looking at the gap between what the system says was delivered and what the network inventory shows is actually there.

The measurement exercise is uncomfortable, but it’s essential. You cannot scope the fix until you know the scale of the problem. And the scale of the problem, for most operators running manual or semi-manual provisioning operations, is larger than the informal estimate suggests.