AI plans count tickets. Measuring work starts here.

Tickets aren’t work

Most AI business cases in support are built on tickets: AI will take 20% of them, so cut 20% of the team. But a ticket isn’t a unit of work. A password reset and a billing dispute both count as one.

To measure work instead of tickets, you first have to know what kind of work each contact was, consistently, across thousands of them. That’s labeling, and it’s the step most teams skip or trust blindly.

Proof of Work tests that argument on real data, one piece at a time. (The full argument is in the manifesto.) This first issue tests the foundation: can AI label support conversations reliably enough to measure work? Mostly yes. And the biggest source of error wasn’t the model. It was a rule I wrote.

What I did

I took 200 customer-service chats and labeled every one for seven things: why the customer reached out, whether the contact could have been avoided, who would have to change something to avoid it, how hard the customer worked, how hard the agent worked, how it ended, and how the customer’s mood moved.

The chats come from ABCD, a public dataset of online retail support conversations released by ASAPP Research. Names and account details were redacted before anything read them. Every label followed a written guide, and every number below carries how accurate it is.

The mechanism: one bad rule cost a third of the errors

The labels for why customers reached out matched the dataset’s own answer key on 177 of 200 chats: 88%, with a 95% range of 83 to 92%. That key was written by the dataset’s authors, not by me, so it’s an honest check.

The other six labels have no answer key. I checked them against a separate set of 48 chats, labeled by a second AI model and then reviewed and corrected by me. Agreement ran from 77% to 85% per label. On effort and agent difficulty, the misses were almost always one step apart, like medium versus high.

The interesting part is where the cause misses came from. 8 of the 23 traced back to a single rule I wrote: I filed shipping-cost questions under billing. The dataset files them under order status and delivery. One rule, a third of the errors.

That is the mechanism. Label accuracy is mostly a guide problem, not a model problem. You find it by measuring against something you didn’t write, then you fix the rule, not the model. The rule is fixed in version 1.2.

This part transfers to any company. Whether people or AI do the labeling, the rules decide what gets counted where, and an error in one rule repeats across every contact it touches. The numbers in the next section don’t transfer the same way.

What this demo set shows, and why yours will differ

The same labels produce the numbers people ask about first. In this demo set, they’re big:

• Avoidable: 173 of 200 (87%). Another 23 were partly avoidable. 4 needed a person no matter what.
• Self-service: 120 of 200 (60%) could have been handled by self-service that works.
• The work itself: 91% was routine or followed a known path. 10% needed real judgment: an exception, a “no” the agent had to handle, an upset customer.
• How it ended: 88% resolved in the chat. 7 were escalated, 8 not resolved.

Don’t carry these to your own queue. ABCD is role-played support: crowdworkers following scripted flows for an online retailer with no self-service tools at all, so every easy question landed with an agent. That inflates both the avoidable share and the self-service share.

Real queues look different. In B2B support, much more of the work is configuration, integrations and judgment, and the avoidable share is usually far smaller. Many consumer and ecommerce companies already send their easiest contacts to self-service, so what reaches a person is the harder remainder. Your number will almost certainly be lower than this one. That’s why it’s worth an hour to measure it.

What it means for your team

If you run support for an online retailer, the money sits in each order. The same labels tell you contacts per order, how much is “where’s my order,” and how much goes out in refunds and goodwill credits. With peak season six weeks out, that is the list of contacts to hand to self-service or AI before volume doubles, not after.

If you run support for a B2B company, the money sits in accounts. The same labels, joined to account and renewal data, show which customers carry the avoidable work and what it costs to serve them. That is the evidence a headcount or AI decision needs before it reaches finance.

The method is the same either way. The only thing that changes is what the labels get joined to.