Person
1 episodes · 21 quoted claims
CVS Health hosted 25 small language models in its data center alongside large language models, which Yogesh Bhagat says cut the cost of its IVR system by five to six times.
“We hosted twenty five S L M s in our data center, working in conjunction with the L L M s. What it helped us, it reduced our cost by five to six x.”
Yogesh Bhagat says CVS Health's interactive voice response system handles more than a million calls every day.
“when we looked at our I V R calls, which is million plus call every day.”
Bhagat says the small language models also let CVS Health meet the sub-second response time that an IVR system requires.
“In addition, it also helped us to meet the sub second response time, which any I V R system would require.”
CVS Health built an enterprise AI development platform that provides security, guardrails and observability by default for every agent built on it.
“we actually build an enterprise A I development platform, which brings security, guardrails and observability by default for creating any agent.”
Bhagat says CVS Health validates its shared development platform once rather than reviewing each AI agent separately.
“So you validate the platform, not every agent which goes.”
CVS Health's approach, as described by Bhagat, is that no AI agent may launch as autonomous and each must earn autonomy by proving consistent behavior.
“Nobody will have a right to launch an autonomous agent. You have to launch an agent and earn the respect.”
Once a task such as a date-of-birth check is shown to work consistently, Bhagat says CVS Health removes the judge model and the human reviewer from that task.
“We remove the larger model out of the context, judge model out of a context, we remove human in the loop, and we know that it is always working fine.”
Bhagat cites a report attributed to MIT saying that 95% of AI projects fail.
“I'll say everyone tells me M I T has published a report where 95% of A I project fails.”
CVS Health's developer platform automatically creates a testing agent for each AI agent that is built.
“What we have done is every agent we build, our developer platform also builds you an agent to test it.”
CVS Health's testing agents are prompted to act as older patients with health issues to generate test questions and show frustration.
“We say, hey, agent, you are an O A P S person, old age person, and you have X number of basically healthcare issues.”
Bhagat says traditional software fails the same way each time, while probabilistic AI agents can fail in several different ways.
“Nothing fails in the same way. If you go in a traditional software, it fails in the same way every time.”
In CVS Health's setup, smaller models give the main answers and larger models cross-check them as judge models.
“your main models, which are generally a smaller models, are giving an answer. And then you are cross validating the output just by a larger models.”
Bhagat expects judge models to be retired over time as the smaller models learn from them.
“As you see maturity, these judge models will go out of life because your smaller models would have learned everything.”
At CVS Health, a repeated question or caller frustration triggers escalation from the small model to a larger model, and then to a human if needed.
“if the question is getting repeated twice, then I need to escalate to a larger model and validate my response.”
Bhagat says the human in the loop at CVS Health is the contact center agent who takes escalated calls, and those calls become feedback for training.
“Human in loop is the agents who are finally picking up the customer calls.”
Bhagat says a chatbot can tolerate responses of about two seconds, unlike a voice system, because users are used to tools like ChatGPT and Claude.
“Chatbot is a place where you can get two seconds.”
Bhagat says a system must be able to replay a transaction and locate which of many agents answered and where it went wrong, within seconds.
“You need to know out of hundred agent, which agent answered, which particular task was deterministic in nature, where the path went wrong.”
CVS Health replaced a legacy tree-structured IVR menu with a conversational AI system.
“We were replacing the old legacy I V R, which used to say press one. It's a tree structure.”
Bhagat uses a 2% customer launch as an example and says business users usually want AI containment to beat the old IVR, but that learning should come first.
“when you launch with a let's say 2% of your customers, you want the containment of that A I system to be better than the traditional I V R.”
Bhagat says phone callers will assume the network has failed and hang up if an IVR pauses for several seconds.
“Because in I V R, as we are talking even if I don't speak for another five seconds, you will feel like network is not working and you will hang up on me.”
Bhagat says running frontier models for the IVR would have kept the project at the pilot stage.
“Frontier models, if I would have run frontier model, then I would have not been able to make it live. It would have only be a pilot project, would have never gone live.”