Armilla Assurance verifies and guarantees the performance of AI models for enterprises.
“what we do at Armilla is essentially verify and guarantee the performance of AI models.”
Person
Karthik Ramakrishnan is the CEO and co-founder of Armilla Assurance, and on The Leadership in Insurance Podcast he discussed insuring artificial intelligence (AI) and the risks of AI models for businesses. Armilla verifies and guarantees the performance of AI models for enterprises.
Ramakrishnan said AI moves decisions from fixed rules to probabilities, which adds uncertainty to every outcome. He explained data drift and concept drift, and pointed to the Dutch tax department's fraud model as an example of harm. He said insurance is an old solution repeatedly applied to new technologies, and that Armilla's guarantee is backed by Swiss Re, Greenlight Re and Chaucer.
1 episodes · 21 quoted claims
Armilla Assurance verifies and guarantees the performance of AI models for enterprises.
“what we do at Armilla is essentially verify and guarantee the performance of AI models.”
Armilla Assurance says it will reimburse an enterprise if an assured AI model's performance degrades on the agreed KPI.
“should the performance of the model degrade and whatever KPI that we are assuring degrade, we will reimburse the enterprise as a protection of their return on investment.”
The same report found 55% of the AI issues companies see come from third-party models.
“And 55 % of the issues that they see with AI come from these third-party models, right?”
Karthik Ramakrishnan says AI moves decision-making from deterministic systems to probabilistic ones, which introduces uncertainty into every outcome.
“we're moving from a world of deterministic decision-making to probabilistic decision-making.”
Karthik Ramakrishnan defines data drift as incoming real-world data differing from the data a model was trained on.
“So data drift means the incoming data that's now in the wild may be continuously deferring from the data that the model was trained on.”
Karthik Ramakrishnan defines concept drift as the concept a model has learned shifting away from its baseline.
“concept drift is that the the concept that the model has learned is also drifting from the baseline”
Karthik Ramakrishnan says Samsung employees reportedly put confidential company information into ChatGPT, though he is not certain it was Samsung.
“you've had issues where this turned into an issue, I think, for Samsung or one of the companies.”
Karthik Ramakrishnan says the majority of the Dutch government resigned because of an AI model used in the tax department.
“the majority of Dutch government had to resign because of an AI model.”
The Dutch model was used by the tax department to detect fraudulent claims, particularly child welfare reimbursements.
“This was a model that was used in the tax department that was detecting fraudulent claims.”
Karthik Ramakrishnan says the Dutch tax model ended up targeting socioeconomically disadvantaged people.
“it ended up targeting sort of socioeconomically challenged segment of the population.”
Karthik Ramakrishnan says humans at one manufacturing plant Armilla spoke with believe they inspect parts with about 96% accuracy.
“humans in one of the plants that we were talking to these they think they have an accuracy rate of 96”
Karthik Ramakrishnan says even deterministic software systems will eventually fail.
“it's not a matter of if it will fail, it's a matter of when.”
Karthik Ramakrishnan describes insurance as an old solution that has repeatedly been applied to new technologies.
“this is a 400 or 4 ,000 year old solution, insurance is not new to a brand new technology.”
Karthik Ramakrishnan says he and his co-founder have worked in AI for 15 years and built companies in the field.
“My co-founder and I have been in the AI space for the last 15 years.”
Karthik Ramakrishnan says his wife is a chief compliance officer at a large Canadian insurer, though his own insurance knowledge was superficial before Lloyd's Lab.
“even though my wife is a chief compliance officer at a large insurer here in Canada, my understanding was superficial.”
Armilla spent three months in the Lloyd's Lab accelerator presenting its problem to mentors and refining the idea.
“So we spent three months presenting the problem, getting their feedback and really workshopping, is this real?”
Armilla launched its first product, a product guarantee for AI model performance, shortly before this episode.
“And we exited with creating our first product, which we launched a couple of weeks ago, which is the product guarantee.”
Armilla's guarantee is backed by Swiss Re, Greenlight Re and Chaucer.
“in partnership with Swiss Re, Greenlight Re, and Chaucer, the Lloyd syndicates, who really innovative companies”
Greenlight Re and Chaucer were Armilla's mentors in the Lloyd's Lab and then provided capacity for its product.
“So Greenlight Re and Chaucer were our mentors in the lab.”
Karthik Ramakrishnan cites a report estimating 15 to 30 trillion in economic activity will be affected by AI.
“the largest one I saw was 15 or 30 trillion in economic activity that is going to get impacted by AI, right?”
Karthik Ramakrishnan suggests mutual structures, shared risk pools and alternative capital as possible ways to fund AI risk.
“Is there a mutual way of doing this, right? A shared risk pool. Is there alternative sources of capital?”