If nearly everyone says they want a human, why do almost all of us choose a machine? The latest surge in customer service AI is an efficiency story on the surface and a risk story underneath. When 94 percent of customers opt for an AI agent, and usage multiplies 22 times in months, investors see scale. Systems thinkers should see load, boundary conditions, and the illusion of competence. Complex systems do not fail where they are strong. They fail at the seams.
Salesforce reports triple-digit growth in created agents, month-over-month expansion in agent actions, and an explosion in AI-led conversations. This is a classic momentum curve built on small bases and high frequency behavior. Markets have a habit of misreading these curves as inevitability. Base rate neglect thrives when dashboards look like hockey sticks. The temptation is to extrapolate growth as proof of robustness. That is how brittle bridges get built: beautiful span, underestimated wind shear. The deeper question is not how fast usage rises, but how resilient outcomes are when the task stops being templated. Adoption is not the same as reliability. Users fleeing a phone queue for instant responses create a stampede dynamic. Herd behavior makes systems look stable right up to the point of buckling.
There is a striking contradiction: a large survey finds 90 percent of people say they prefer humans for service and 61 percent believe humans understand them better. Yet when a button offers instant AI versus a hold tone, 94 percent click the bot. That is not love. That is friction. In queueing terms, the shortest line wins until the job is atypical. Most inquiries are low stakes and repetitive; the fat tail of customer anger is not. A single mishandled refund or misrouted claim can erase dozens of fast interactions. Power laws govern brand damage: a few incidents drive disproportionate churn, complaints, and regulatory attention. Convenience extracts goodwill early and risks it late. When companies optimize average handle time, they often degrade tail outcomes. Goodhart’s Law applies. Measure speed and you may get speed at the cost of judgment.
Escalations to humans rising from 22 percent to 32 percent is being framed as progress; robots are taking on harder problems and asking for help at the right time. Perhaps. It could also be a leading indicator of load creeping past the edge of the system’s map. As domain complexity rises, so does the chance that error modes overlap. The Swiss cheese model explains many industrial accidents: multiple thin safeguards, aligned holes. AI routing mistakes, partial context windows, stale knowledge bases, and vague customer prompts are thin slices. Stack them and the miss routes itself. Normal accident theory says that in tightly coupled systems, small errors propagate fast. Customer support is now tightly coupled across data, workflow, and brand voice. When the system escalates more, it admits its boundary conditions more often. That is healthy if the humans are sharp and resourced. It is hazardous if those humans have atrophied.
The law lags. Liability for an AI agent misquoting terms, mishandling a warranty, or booking the wrong appointment is not settled in many jurisdictions. When this goes to court, transcripts and logs will be exhibit A. So will training data and prompts. Regulators look at accountability chains. Policy analysts studying misfires, from military standoffs to industrial accidents, warn against blaming only the last visible step. The causal chain is longer. Sales or service choices made upstream by product and compliance teams create the conditions for errors downstream. Today’s AI agent strategy carries unpriced legal options embedded within it. Most are out of the money—until they are not. Legal uncertainty is volatility, and volatility carries a cost. Firms that scale chatbots without a clear liability framework are writing insurance for unknown risks at unknown premiums.
Automation always promises leverage. It also threatens muscle loss. A 2022 industry report found 65 percent of service managers worry their teams will lose problem-solving skills as bots absorb routine tasks. This is cockpit complacency applied to contact centers. If humans become the last line of defense, they must train on edge cases, not only review pleasant summaries from the agent. Resilience is built under stress. Antifragile systems improve when tested. If managers move every solvable task to a bot to hit short-term numbers, they starve the team of practice. Then, when a black swan lands in the queue—a product recall, a data breach, a viral complaint—the humans are cold. The time to drill is before the emergency. The cost of maintaining skill looks like drag in a dashboard. It is actually insurance against compounding failures.
Most enterprises will not build their own stack. They will rent an agent platform, let it interface with CRM, and call it transformation. That creates concentration risk. Vendor models update. Guardrails shift. Optimization for aggregate performance can push your brand tone off center or degrade nuanced flows that worked for your customers. Data drift is subtle; feedback loops are slow. Switching costs rise with every workflow automated and every knowledge article embedded in prompts. Pricing power shifts to the platform over time. In zero interest rate days, vendor risk was hand-waved. Today, margin matters. If your entire service perimeter depends on a single provider’s black box, you have replaced operational complexity with contractual complexity. That is not a reduction in risk. It is a transformation of it.
The sales pitch is a hybrid human-AI team. The execution needs real option value built into the process. Design for graceful degradation when the bot is wrong. That means circuit breakers on low-confidence responses, conservative fallbacks for regulated topics, and fast lanes to well-trained humans who have authority to fix problems. Measure tail metrics, not just averages: false resolution rate, P95 time to make right, escalation quality, complaint velocity after bot contact. Randomize human-only days to keep skills alive. Rotate staff through complex cases by design. Build audit trails that a regulator or judge could follow. Set clear consent and disclosure rules for customers who think they are talking to a human. These are not tips. They are the difference between a system that fails quietly and one that fails loudly with evidence.
The current narrative reads like a one-way bet: huge adoption, lower costs, happier customers, higher margins. Travel, retail, and finance show triple-digit growth in agent actions. That can all be true in fair weather. Markets, however, price the world as it is until they do not. When a low-probability, high-impact event hits service—say a cross-platform outage or a data access bug that corrupts CRM context across agents—the same efficiency that made the system sing will amplify the error. The feedback loop is bi-directional. As with any leverage, your upside accelerates and so does your downside. The real edge is not deploying the fastest bot. It is building a service function that benefits from volatility, learns from stress, and keeps human judgment sharp. Efficiency wins the quarter. Resilience wins the decade.
