CaseText operated for about ten years before it was acquired.
“we were in business for ten years before our acquisition.”
Person
Jake Heller
1 episodes · 23 quoted claims
Thomson Reuters acquired CaseText for $650 million in cash, a deal that Heller says closed less than a year before the interview.
“an acquisition by Thomson Reuters for 650 million in cash, which took place, you know, a little less than a year ago.”
CaseText's original vision was to crowdsource legal information from attorneys and build a search interface on top of it.
“The original vision was really around crowdsourcing and enabling, uh, attorneys to help build a really good source of information and build a great kind of search UI on top of that.”
CaseText's crowdsourcing plan failed because attorneys did not want to contribute their time and information.
“long story short, attorneys didn't wanna contribute their time and information.”
CaseText began working with large language models around the time of the BERT paper, in 2017 or 2018, to improve search.
“we started working with, you know, large language models, basically since the BERT paper in was that 2017, 2018,”
CaseText received early access to GPT-4 from OpenAI before its public release.
“we got really early access to GPT four”
Before GPT-4, CaseText's research product covered over a billion pages of legal material, including cases, rules, regulations and statutes, updated daily.
“over a billion pages of legal information updating daily. Uh, cases, rules, regulations and statutes.”
Heller says semantic search suited legal research because judges describe the same ideas in varied language, so keyword search used by competitors missed important opinions.
“keyword based searching, which is what most of our competitors offered at the time, would miss really critical, uh uh, legal, you know, legal opinions and precedents.”
Heller says the California Innocence Project had a three-to-four-year backlog of case files to review, which CoCounsel could help address.
“The California Innocence Project had a three or four year backlog of files to review”
Heller says CaseText had roughly 10,000 customers when it entered 2023.
“We were growing fine as a company, you know, we had a good product, maybe even 10,000 customers at that point.”
Heller says CaseText grew about 50 to 70 percent a year in the years before the GPT-4 launch.
“we've been growing at call like 50 to 70 year over year for the years before then.”
Heller says the surge of attention around ChatGPT beginning in November 2022 helped drive CaseText's growth by making law firm leaders eager to learn about AI.
“to the extent there was like a hype moment around ChatGPT starting in November 2022, that really like played into, I'd say, our our growth.”
Heller says OpenAI was coy about the model it gave CaseText early on and described it as a new version of GPT-3, which he thinks was called Davinci three internally.
“they called it Davinci three internally or something like that.”
Heller says CaseText ran an unpublished bar exam on GPT-4, and it did better than at least 70 to 90 percent of human test takers.
“So we knew it wasn't part of the training data and it like, did better than at least like 70 to 90 of of human test takers”
Jake Heller personally built the first demo of CoCounsel and showed it to all of CaseText's employees.
“Like I personally built the 1st demo of CoCounsel, and we demoed it in front of the whole company.”
CaseText shut down all of its offices during the Covid pandemic.
“we shut down all of our offices during Covid.”
CoCounsel's legal research skill turned a single user question into about 20 different search queries before reviewing the results.
“taking that one query and turning it to like 20 different search, search questions or queries, right?”
Heller says CaseText had the model output a relevance score before its reasoning, which they found improved performance.
“you can have it output the number 1st and then provide its reasoning.”
Heller says CaseText could cut the model's output off after the first token, a score, so the result could be tested objectively.
“But you cut it off after the 1st token produced, so that's very objectively testable.”
Heller says about 20 customer support and success staff, mostly former attorneys, regularly graded CoCounsel's test results by hand.
“a lot of our customer support and success people, maybe 20 or so professionals, constantly checking these tests”
CaseText used a copy of GPT-4 that had not been fine-tuned for most of its tasks.
“we've been using a, uh, not fine tuned copy of, uh, GPT four for the vast majority of the tasks that we do.”
Heller says CaseText had not found Meta's Llama 70B model able to outperform GPT-4 on its tasks, though it came close.
“we haven't seen, you know, uh, say, a Llama 70B outperform GPT four in a task, get close, but not outperform GPT four.”
Heller says one guardrail against hallucination is a simple non-AI check that a quoted passage actually exists in the source document.
“if you ask it to quote from a document, you can do a non AI kind of.”