Harnessing AI for legal practice with CoCounsel’s Jake Heller

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. Today I'm talking with Jake Heller, CEO and cofounder of CaseText, one of the most innovative tech companies in the legal space. It's also one of the very 1st companies to make a really working, real world application of GPT, and last year it sold to Thomson Reuters for $650 million. Jake really gets into the weeds on how he made GPT work for his use case.

And I think anyone around GPT or LLMs will find this episode particularly useful. I hope you enjoy it. All right, well, this might be a cliche place to start, but I need to do it because I know this is a good story. Can can you start by telling us the story of CaseText? From from what you were thinking at the beginning to the um, the large acquisition, where it ended?

Yeah, totally, so, so, first of all, it's a long s history, uh, Take your in that we we time, are. Yeah, well, we, we were in business for ten years before our acquisition. And so to to kind of jump to the end, it's it's one of those classic, uh, you know, quote unquote ten year overnight success stories that you, you, you hear about in Silicon Valley. Sometimes, where you have to work for ten years to make something that, um, really, really works.

So we we began. I was a lawyer, um, and so are my co founders, and I think we all shared the feeling that, uh, as we're with, uh, you know, ten years ago. And and still is true today. When you're using like consumer technology, regular, regular technology, if you wanted to find something really easily, or, you know, have use a a very kind of simple interface. Um, you could do that very simply and and and quickly and easily with tools like Google or your iPhone or whatever.

But as soon as you step into your law office and you need to find a piece of evidence that might, you know, exonerate your client, you know, get them out of jail. Or enable you to win a a billion dollar lawsuit on behalf of a client, which may make the difference between the business existing or not, and other such things. Uh, it might take you many days to find a piece of information, and, uh, you know, or, or.

You may be struggling against the technology. And so we thought, well, if we just bring consumer level, um, you know, natural language processing and user interfaces to law, uh, we would, you know, enable this. This kind of l non trivial, really important for a lot of people kind of use cases for for lawyers. And so that was the idea behind the company, and we're now, we're known now as an AI company. That actually wasn't the original vision.

The original vision was really around crowdsourcing and enabling, uh, attorneys to help build a really good source of information and build a great kind of search UI on top of that. Um, long story short, attorneys didn't wanna contribute their time and information. The same way that Wikipedia editors or GitHub contributors or Stack Overflow question answers were were just a very different personality. But we, you know, we found that out kind of the same time that, um, natural language processing and kind of very early versions of artificial intelligence.

Nothing that even resembles what we have today. But we saw that that potential of providing major value to our customers, even without the crowdsourcing of information. And and so we leaned more and more heavily into that, and we started working with, you know, large language models, basically since the BERT paper in was that 2017, 2018, uh, and really early on, that bad bandwagon. We saw immediately its applications to make search better in our case.

And I I think in part because we were so deeply invested in that technology, um, we were really tracking closely the work at the major labs, including OpenAI, we had a pretty good relationship with. And so so fast forward from the kind of founding to closer to the end, we we got really early access to GPT four. Um, and immediately saw an opportunity to go way beyond anything we were developing for our customers at the time, which is more of like a search based tool.

And, uh, we saw this opportunity to instead create, using this technology, the world's 1st AI legal assistant. Uh, an assistant where you, as a lawyer, could just ask it to do, um, you know, major legal tasks, like doing research for you, or reviewing thousands of documents, or, uh, editing contracts. It's similar kinds of activities and, um, a almost unlimited quantity, you know, research, dozens of questions, review thousands of documents and get answers. Both at a human level of quality, you know, rivaling that of, like, a 1st, 2nd or 3rd year associate at a law firm, but at superhuman speeds.

And so that that product, that new product direction and, uh, new capabilities, really, I think launched us towards the end of our of CaseText's like kind of history as an independent company. Um, culminating in in, uh, an acquisition by Thomson Reuters for 650 million in cash, which took place, you know, a little less than a year ago.

but I guess GPT four, even if you had early access, it happened, you know, maybe nine or ten years into the company. So, you know, for example, like, um, you know, 2017, when you're starting to look at these, um, you know, these early large language models, what was the product doing and and how were the models helping then?

so, the the main challenge that we were trying to solve for our customers is searching for information. And the real focus of our product at the time was helping them find, you know, do a task called legal research, where we had, um, over a billion pages of legal information updating daily. Uh, cases, rules, regulations and statutes. And, you know, finding the right say precedent to help make your your point in court was really, really important job for lawyers, and so we were helping search and find the right information.

The same thing is true of people would upload their own databases of information. So you're doing an internal investigation as a lawyer for a company and you're trying to figure out, uh, whether or not fraud, committed or insider trading has been committed. You're looking over all the emails and teams messages and so on. And so they'd upload, you know, sometimes millions of documents to a database, um, you know. And many of them are like ESPN, fantasy news alerts or whatever, like.

It's just so much stuff that that is irrelevant, and there's oftentimes needle in a haystack kind of challenge. And so that those are the those are the big challenge we're solving for customers at the time. And I see the way that, so yeah,

you're probably using like embeddings, like you're trying to find like kind of semantic search, yeah,

exactly the way that this this ended up being really important is semantic search was, um, a really, really good fit for, uh, for legal in particular. Because, you know, sometimes, for example, as I do, looking for precedent, judges just write these like legal opinions that become precedent, become like the law, basically. But they use this, you know, pretty diverse set of language to describe the exact same things. So keyword based searching, which is what most of our competitors offered at the time, would miss really critical, uh uh, legal, you know, legal opinions and precedents.

And sometimes it would bubble to the top stuff that is like, very obviously irrelevant. You know, the same words used in a different context mean totally different things, but embedding like the sentence or paragraph level or both, um, you know, it's. It's kind of commonplace now, you know, honestly, in, uh, in. In most text stacks for, you know, not just in legal, but across, you know, basically every vertical at this point. But at the time, it was like, really novel and gave us a real edge over, um, you know, why use us instead of others?

the and I guess be before we get like, super deep into the the technology. B Before this company, you were working with the Innocent Innocence Project, weren't you? I mean, you, you kind of come from an interesting legal background. And in fact, did, did that, did that influence, um, this, this product, in any way?

Yeah, I mean, I, I, I, you know, if you look at my LinkedIn, for example, I, I would look just kind of like a normal lawyer before starting this company. Um, and so I think there there are a lot of personal influences on on how the company and product evolved. Like, I grew up out here in Silicon Valley and I've been coding since I was a kid. So, on the one hand, I, I really fell in love with technology, on the other hand, I fell in love for a variety of reasons with with law and policy and justice.

And had worked on, um, before starting CaseText, I worked, um, at nonprofits, for the government, um, you know, including, uh, just very short stints, like an intern, basically for the White House Counsel's office and for the governor of Massachusetts at the time. Um, and I, uh, you know, really saw, I guess, firsthand, through a lot of those experiences, that without great technology. If you have more money and resources to throw at a legal problem, the odds of you doing like, succeeding, um, go up substantially.

Even if you're not on the side of of the right answer or justice. And I always, I think, I think a lot of lawyers practicing see that and they have kind of a pit in their stomach about it. Um, and so we saw from day one the opportunity for technology to be a a equalizer, to to level the playing field. And kind of fast forward, you know, W back into, uh, CaseText days, like when, you know, we had, we had a relationship with the Innocence Project.

I'd informally done some work for them, but that was renewed again when we launched CoCounsel. And it turns out that they have these like immense problems, um, where even just getting your case reviewed, you know, if you're in the Innocence Project. The way it works is, um, people in jail will say, Hey, actually, I'm, I'm innocent of this crime, and I think we can prove it. It's like DNA evidence, like something that's like hard, you know, uh?

And a lot of these people folks have been in in jail since before. There was like good DNA testing used in trials, but the evidence has been preserved. So, um, so they apply and say, Hey, I think that, you know, my murder or rape or whatever case, it's like, it's, it's not on me, it's on somebody else. And I've been in jail for like 20 or 30 years at this point. Um, and the Innocence Project will review that case, just the reviewing process.

They had a a. The California Innocence Project had a three or four year backlog of files to review, thousands of for each each case. It's like thousands of pages of police reports and, you know, witness testimony and statements and so on. And they they like to review these things to decide which cases they dig deeper into a four year backlog on that. And feeding that into, like, AI, for example, um, you know, when? Instead of the two folks running the show and like the the four interns or whatever they had at the office, right?

Spend it by by by years to help make them make determinations on which cases to take or not. Um, so, you know, it was really edifying to see, although it took a very long time to get there. Some of the causes I cared about and worked on, even before starting CaseText, were the kinds of things where, um, the technology we were building could really make an impact.

Uh, and now my my impression of your company is that it was kind of, you know, really kind of struggling or not working for a long time. And then sort of started to, like, really work as the technology ad advanced. Is that like an accurate understanding? Like, was there one moment where it it was like, hey, like, now the technology, like, works so well that it's really like, powerful for for customers? And and if so, like, what point was that?

Or, or am I misinformed that? It's just sort of this, this kind of gradual, um, increasing of capability that eventually kind of, you know, got to this point where, um, it was working super well. Or maybe it's just like, kind of GPT hype, like, what, how, how, what, what do you I, I would think? Uh, if say, I would you feel up to say, it's similar it?

to the 1st. Two things that you said, which is we, we, you know, we're doing pretty well. Um, we entered 2023 before launching our CoCounsel product to the public. We had already been developing it for a number of months and and had it in beta with a number of customers. But we entered it in a pretty strong position. We were growing fine as a company, you know, we had a good product, maybe even 10,000 customers at that point.

Um, so it's not like we were, um, you know, on the ropes or whatever. There was never the, you know, you, you hear about some of these companies, um, maybe including, uh, some that you've run. Where they have these like massive takeoffs and and they're, you know, raising these enormous rounds, and they're got being constantly reported on, um, you know, in the news and and TechCrunch and so on. That was not us, right? We were kind of under, like behind the scenes, slowly growing, getting better gradually.

And then, um, for us, GPT four is a moment where, uh, the value that we could provide to our customers kind of exploded overnight. And we could take some of that those pre existing assets, like all the the billions of pages of legal contact content we had developed, great search technology, 10,000 customers. And you add on top of that, you know, an inc, an incredible increase of, uh, of capability and utility. Like, all of a sudden, this, this AI assistant does a lot more, just searching, it reads for you, it edits it, does all this amazing stuff.

You add that on to a thing that's already, you know, kind of working pretty well, and, uh, and that's where you hit. Like, what felt like a real inflection point for us? So we've been growing at call like 50 to 70 year over year for the years before then. And then in the 1st like few months of 2023, we almost doubled or tripled our revenue. This is crazy, you know? Um, and that's just the numerically, that's one way to put it.

But like, also, it just felt very different being at the company. Like you could feel the velocity, um, and feel the impact. And and and you know, there's this great essay by Marc Andreessen that I think about a lot these days. And it's people, people credit. It as like the 1st time the word product market fit was like, coined. I think it's, I think the essay is called something like the only thing that matters or something like that.

yeah, I know exactly what you're talking about. Yeah,

you know what I'm talking about? Yeah, so there's this paragraph in it and he's like, talks about what product market fit feels like, and, uh, he says stuff like, well, your servers will go down because so many people will be using it at the same time. You'll have. You can't hire support and salespeople fast enough. You are gonna be hounded by the press. Um, they're gonna be staking out in front of your, of your house, you know, just to get an interview with you.

They're gonna be you could eat your eat for a year free at Bob's. Is one of the things he says. That Bob's is like this Woodside, you know, cafe or or diner out here. Um, that that, you know, a lot of VCs live near here, so they wanna have you meet there. And then none of that was happening before, you know, CoCounsel, and before that product relaunched, and literally all of that happened after CoCounsel, almost like verbatim.

It was kind of freaky. And and so we had thought we had product market fit before we had some some amount of product market fit. We had a lot of happy customers, um, but nothing felt quite like that. Essay like, you know, that that I like, literally had to buy the press. Servers going down, couldn't hire sales and support people fast enough. Uh, and I ate a lot of bucks, you know? And and for every other previous fundraise, for example, we we would always get like one or two term sheets if we were lucky after like months of of, and then we're getting term sheets thrown at us, you know, in this and and that.

So that must have been GPT hype. It also is like a reality of of, you know, people are doing these like market maps and and and trying to figure out what's going on in legal tech. And everyone's saying, like, you gotta check out CaseText. And that was the 1st time it's ever happened to us. We had investors coming to our door, you know, um, you know, and and and that that was like a very, very different, you know, night and day kind of experience for us, for sure.

it's funny. I feel like some cynical people watching this, you know, might might view that as kind of like a like, a hype story. I mean, certainly, like, you know well, 1st of all, fifty to seventy growth is fantastic from my perspective. So I don't want anyone to think that that I consider that, you know, not good. Um, but, you know, I, I guess, you know, VC interest in in meeting, and, um, you know, growth is probably connected, but not always the same, and I feel like it disconnects the most.

Um, in the AI realm, so, I mean, uh, but it sounds like the from your perspective, the product was just really like, palpably different and and more powerful. And and were you using, um, like GPT three also, and it was like the switch to four that really was the change. Or, yeah, I mean, you yeah, just,

know, I, I, so, so, 1st of all, I do think there's the that. At least in our case, the v. The increase in value is incredibly real and palpable to our customers. Uh, going from like a basic search tool to this, um, uh, AI assistant, you can ask it to do basically anything, and it does it like, really, really well and really fast. And and it expanded us, I mean, like, very tangibly, into parts of the workflow that we just weren't even touching.

You know, for those kind of lawyers, or those no lawyers listening, you know, you go from just legal research to reviewing documents, reviewing contracts, um, reviewing contracts for compliance to the playbook, for example, uh, or or a set of policies of the company. Um, you know, searching over vast databases of of information and summarizing those results in a little paragraph. All that stuff is like, you know, like five or six x more valuable than we were offering before.

Um, and and so I do think, you know, I think that that the indicators that are better, much more than VC interest. Because I agree that, you know, not always the same thing as as what's happening in reality, sometimes it's, but, you know, kind of loosely related to reali reality. Um, is just the increase in in revenue and retention and usage. And, uh, and you know, people are just publicly saying about us, you know, everywhere, uh, and and so it.

It was very like, I say this with humility, because we know what it's like to not have that for like nine, a half years. And to experience all that for your customers, um, over, you know, the last kind of six to nine months of our company before before exiting. And that, you know, we're still building the product, still hear about this kind of constantly. It was. It was definitely a step change in in value. And I think the other thing too, by the way, in terms of the hype thing, that is is not investor related.

But I think it's an interesting learning for for me in our industry, and I think this is true of many industries. We were fighting an uphill battle to get them to even care about AI before GPT four, and really before ChatGPT, um, and what ChatGPT did is it created this moment where all of a sudden everybody was talking about AI. Within law firms, they're like, Oh, this is gonna replace us potentially. I don't really know how or why, or what, or maybe not replace us, but it's gonna change the way I practice.

Again. I'm not sure exactly how or why or when, but I gotta know about this, I gotta keep informed about this. And so for the 1st time in our history, instead of saying like, we're gonna make you more efficient. And that's scaring lawyers because they bill by the hour. And you know, it is a pretty bad story, like, Hey, we're gonna make you more efficient with negative ROI, but you, you ought to do it cause it's good for your customers.

Um, you know, all of a sudden they're like, Well, we. We acknowledge the world is changing because of AI, and it's gonna change in really substantial ways, and so I better be on top of it. And we went, you know, from having conversations from still like important people at our law firms, like chief knowledge officers, chief information officers, to like the managing partner of the firm. The person who's basically the CEO of the law firm, who needed to to meet with us personally to really understand how this is gonna impact their business.

So, to the extent there was like a hype moment around ChatGPT starting in November 2022, that really like played into, I'd say, our our growth. In that, we went from kind of a kind of trying to push our way into being pulled into these deals. Um, and that was a really, really big change. And that that's persisted. In fact, it's accelerated in legal. Um, as I think, more and more applications are coming on board, and and the utility's more and more proven out.

And and if I were to give any advice to other founders, if you find that moment, you know, don't try, don't try to push against what customers want or are looking for. Even if you're right, you know, even if you can prove that you're right, they really ought to be doing this. Um, try to find the moments where they're really pulling, like, they're really looking for the solution right now. Um, the the difference in in how, how difficult it is to sell, and how, um, you know, how much coverage you're gonna get.

In the local trade press, all the way up to the national press, everything will will will depend upon what the zeitgeist is and what where people are pulling from. You know, to answer your question about GPT three and so on, um, we, we have a relationship with OpenAI, and it's it's funny to remember, like OpenAI before ChatGPT was obviously an impressive organization, they had some of the most impressive people working on this. Um, but I think that, like they were not commonly, you know, they were not a household name, let's put it that way.

And tools like like GPT three were, um, you know, had some real utility, like especially in the marketing copies. A few companies took off with GPT three with that, but definitely not in like other fields like legal. And and so our relationship with them, to some degree, was they would come to us and say, Hey, is this work for you? Let me show you a snapshot of the model, or show you. And we'd be like, No, this is, this is not good, like we cannot use this.

It is hallucinating wildly, you know, it's. It's coherent sounding English, but it's not really useful for our purposes. And then when we got access to GPT four, um, we tried some of the same stuff we've been trying for years, kind of assuming we didn't even know it was GPT four, by the way. They they just said it was a new version of GPT three or something like that. They were pretty coy, so I think I think they called it Davinci three internally or something like that.

So it was like, you know, just like the next, and there was already Davinci two, like, as if it was the next version of of Davinci, right? Um, which is kind of what we assumed, so we'd just started playing with it and we're like, okay, whatever. Until it started just nailing everything, we're getting it, you know, I mean, very popularly. We. We were the ones who ran the bar exam on it for the 1st time, an unpublished, yet yet unpublished bar exam.

So we knew it wasn't part of the training data and it like, did better than at least like 70 to 90 of of human test takers, you know? So we're like, Okay, this is super different. And and you're curious and interesting here. And I think it took us like a weekend of not sleeping and just constantly just working with this thing. To to be like, Okay, there's something real here and and Dave, this is a turning point for its utility in legal.

Um, you know, it's still not perfect, as everybody knows, it's not a perfect model. Uh, you have to do a lot of work sometimes to mitigate some of the dumb things it does. Or to. You know, you only build certain capabilities based around its its limits, current limitations. But the what it could offer was such utility that we we knew we had to lean in.

Yeah, that makes sense. Um, you know, I think something that people might not appreciate, but I do, is the the challenge you might have. You know, nine years in to kind of hop on the trend like that, like, I I think, like, you know, you would have a lot of advantage to like existing customers. Um, but I, I could imagine, you know, most organizations would have a little bit of inertia to, you know, kind of. Maybe like the way that things were, that they were doing things.

And and the change from, you know, running your own models to using, um, uh, 3rd party models is is a big one, right? Because it, it, you know, can, in some cases, you know, make a lot of work irrelevant, and and you know, people's jobs irrelevant. And we watch, you know, uh, our customers, um, wrestle with the these changes all the time, often, you know, n not very well. But it seems like you did a really great job of really leaping on a a different way to accomplish the the goals that you had.

I mean, w What did you account for? That? Um, that ability to navigate that change

You know, what's funny is is that, um, so, so, it was a really big change. Um, I remember getting up in front of. We then, like, we had 100 employees and be like, everybody, we're stopping what we're doing, um, and we're all gonna work on this. And by the way, we don't even know when this is launching publicly, and therefore we can't launch it publicly. So, on the one hand, like clock starts now, we may have just three months and we gotta build something great before then.

Uh, we may have six months, in which case, we did noth nothing public for six months. And that's gonna look weird to our customers, but it doesn't matter, we're doing this right now. And and so, um, I think that so on the one hand, like, yeah, it was like a very like, in retrospect, correct move. And retrospect. It was actually probably more difficult than I felt at the time. At the time, I was like, this is the almost obvious thing I've ever seen in my life.

And and in some ways, I'm surprised that more companies haven't done this kind of in our space and others. Where you get handed this like, absolutely breathtakingly new and different technology. Um, that, you know, your customers have been begging you for years. If you only could do this, I would totally buy your product. They're like, Well, that's impossible, that's, that's insane, you know, there's no way we can possibly. We have to develop a custom model just to do this one tiny part of whatever that doesn't make any sense.

And now, all of a sudden, this one model handled all those educations, did everything the customers were asking for. And so for S to to me and my co founders, who were the 1st ones to kind of have access to it, we're like, this is a slam dunk, obvious move, you know? Um, but you know it, it actually was kind of controversial within the company for exactly the reasons you said, in in ways that we in some ways at 1st, kind of somewhat underestimated.

And there was some change management needed to happen. People were skeptical, people were like, Oh, here we go again, some new other tech techno, you know, you know, uh, kind of craze, like, I think here's what. What ended up making it successful in inside our organization was a few things. The 1st is that we talked about the executives 1st, and all the executives after, like a offsite that we did, were like, Okay, yeah, this is, this is, this is awesome, we should obviously do this.

The 2nd thing is we made it very tangible. Like I personally built the 1st demo of CoCounsel, and we demoed it in front of the whole company. And then gave everybody access and let them play around with it. And they could see exactly how legal research was gonna work and exactly how reviewing documents was gonna work. And it worked pretty darn well. It wasn't as good as what we eventually released, but um, having that kind of something very tangible to get people excited about, I think went really far.

Um, the 3rd thing that really helped was that we talked to customers pretty early. We brought a few in under NDA, um, with a very early beta. And for our sales and success, and for people seeing our customers react and the way that they did, um, I think really made a huge impact. And then finally, like, culture as a company, make a huge difference. And we actually specifically look for people who are flexible. It's like one of our, not a company value, but we had these like traits and characteristics of folks that we like to hire, and one of them was about flexibility.

And we look for people who are, you know, kind of this kind of great startup mentality of like, Okay, I was doing that yesterday, that's it is what it was. Um, and and you know, th that comes in handy in a lot of different ways like th. This big change was a, you know, obviously one of the biggest, maybe not the biggest. Kind of, I won't call it a pivot per se, but like biggest new product approach you've taken, or whatever you want to call it, um, but we also, you know, we shut down all of our offices during Covid.

And, uh, you know, there are times where we thought we were gonna raise around and we didn't. We had to kind of change our approach to find finance and all this other stuff. Just happens in startups like it's. It's a volatile, crazy kind of world, as you know, running a company and we look for, we look for people, um, who not only could handle it, but did really well through those kinds of transitions. And so having that group of people there already kind of preselected for that, who had gone through some stuff together already.

Um, you know, uh, uh, I think really made a big difference there, too. Uh, the kind of right people in the room to, you know, and and being frankly small enough, just 100 people, uh, all that I think made an impact. So the culture of the company was in in a place receptive to that as well. But I, I can't emphasize this enough. People listening here, in your, in our industry, are at similar ones. I can't believe more people have not done this.

If you're in like a vertical and you're servicing with something like standard SaaS application, you have the opportunity to create like an AI coworker for your clients. Now, like, the value has gone up from being just I make your workflow slightly more efficient to I will do part of your work or your, your colleagues work for you. And we know what the value is that people get paid to do this work is, it's like their salaries.

Not saying that we're gonna replace a bunch of people's salaries. It's it's, you know, for other reasons, it's kind of been the opposite. But, you know, typically pay a lot more for just like one person that you usually pay, or like five people that you pay for most software. So if you can get that kind of value out of software, sell it to your clients and scale that. It's like, huge. And I I can't believe more people haven't thought about that a little bit more, you know?

And kind of some people have added in AI. This kind of sort of lightweight, or, you know, it's kind of on top of the product a little bit, um, to make their their workflow applications a little more efficient. But if you're like servicing like the insurance industry or doctors, or yeah, other aspects of legal work or whatever. Like, there's so many opportunities, like millions of opportunities to be like, Oh, well, what part of your job, you know, can we do with AI?

Um, faster, better, cheaper, and enable you to do more important stuff. And how much should you value that act? And I think that's, that's where, um, that's where the big opportunity is.

Well, totally. I mean, you know, I think that's an incredible message to send to people and people should listen. I also wanted to have you on this podcast because I think you kind of underestimate the skill that you have in navigating that change. Because I think it it comes easily to you, but clearly is is difficult, um, for a lot of people. And certainly there's like an inertia step, but my feeling is that there's other things that you were doing to actually make this a good product.

Like, for example, you talk about, you know, making a demo over a weekend. And I think a lot of people have that same story of making an exciting, you know, demo over a weekend. But then kind of have trouble, even after getting a company excited about moving in that direction, have trouble getting that into a product that's that's really useful for customers. So maybe you could talk a little bit about what the journey was like there, because I think that's where a lot of the people listening to this show get stuck.

yeah, yeah, and and I actually think to that point, the the kind of weird place you are with technology is. It's trivially easy to build a really fantastic demo, and and then after that, building a great product is like, super hard, super hard with this technology. And I can talk about what we did. And then not to say it's like a formula for everybody else, but there might be some analogies. Um, f 1st things 1st. It was really helpful within our company that that we had so many domain experts, um, you know, from the top to the bottom.

A lot of folks who had practiced law, um, really knew it deeply. And so when it came time to design, like the user interface and the um, the the prompts to, you know what, what does a good look like? Uh, and I'm happy to go into this a little bit more in detail, like what does it look like? To create like a great prompt, or even a g great set of prompts to achieve the the kind of end result.

Um, we were already kind of a step ahead there. I'd say like, we're not perfect, but like, we started from a place where, um, we intuitively knew what we think our customers would want, and intuitively knew how to achieve it. Using this new technology, how we'd express the the, you know, the the objectives to be done in English, and we're starting from there. And I think that that a certain type of company right now is gonna be really advantaged, which is gonna be those like, deeply, you know, if if you're, if you're making these like AI coworkers, um, then being having been one yourself.

Whether you're the founder or have the great product, people around you, or great designers around you, that kind of experience, I think, is gonna put you in like, the 1st, 1st good step, the 2nd step that, um, I think is really important, is around testing. Like, ironically, I was never like a a formal engineer, I've just always kind of built stuff for fun. I never really got a tester to develop it in that world, right? Um, but I'd say the opposite for field folks who are not driven by test driven development.

In the pre AI world, I'd say it's the opposite. Now. I think you need to almost write your test 1st, what is what is like success look like? And in AI, you know, for these kinds of like generative AI or or large language model applications. We're doing things like review one zero documents and tell me which of the which of them have any evidence of fraud, um, or what have you? And so you can create these like test sets that look exactly like that, right?

X number of documents, x number of questions. Here's the right answer. We objectively know it for whatever reason. Um, go and see how well it measures up. And we would literally put together hundreds and then later thousands of tests against each of what we called our our s, the skills that CoCounsel has. If it's gonna review documents, then it needs to be able to review. You know, here's an example in Japanese, and here's what the answer is, you know, it's gonna review this and has to cite to this page where the answer is located, or this this paragraph where the answer is located.

Um, you know, if you're just gonna be, you know, a contract, it's gonna have to understand that. The definition in page one and the the clause in page 72. H can be used together to create a kind of a novel and weird meaning of it, but it has to get that weird meaning. And you, so you set up these like test criteria, and and there are ways to do this, um, with like. If the answer includes this word or includes this number, then it's like fine, if it doesn't, it doesn't, right?

So the kind of basic tests, but also to use AI to grade kind of more conceptually difficult tests, where the words may not all exactly be there. Um, so you set up these tests, um, we built our own framework, but now there are many frameworks that are being sold as kind of like this kind of AI gold rush. Um, many of them are very good, uh, and and that's what we built 1st, and then the prompters and product developers, what we call skill developers, would would work against those tests.

And what what ends up usually being the case? You know, the 3rd kind of thing to think about here is when you're actually building these like capabilities or skills for the AI. Um, they often mirror almost exactly what the best person in the world who's doing that job would do if they just had an infinite amount of time. Basically, so, like we did legal research with CoCounsel. And you know, the way it would work is customer would ask a question behind the scenes, it would take that question and and maybe have a back and forth conversation, make sure it really understands after it understands.

So that's like one set of prompts, right? Understanding the uh, getting a firm understanding of the of the query. 2nd set of prompts is taking that one query and turning it to like 20 different search, search questions or queries, right? Um, you, if you're a great legal researcher, you don't just try one search and call it a day. You try different keywords, different approaches to to researching your question, different, um, different search engines, different databases. So we have prompts that basically make the AI do that, and then a list of results come back.

You do some like deduplication or whatever behind the scenes. And then we gave every result of the to GPT four to review. Um, and make a crooked determination relevant, not relevant. If it was relevant, it would start to making like notes, basically that it'd start compiling of its understanding of the answer based on the things that are found that were relevant. And then finally, set of prompts that would take together all of the notes it would make, um, and turn that into a memo responding to, uh, the user's original query.

All those steps could vastly resemble the kind of stuff that I would be doing if I was a researcher. I would hear the question from the partner, try 20 different searches, get hundreds of results, review each of them. Some of them I'd quickly be able to say no to many of them. I'd say, Okay, there's something interesting here. Let me underline this part, pull out these quotes, take these notes, right? And then I compile that into an answer based on the notes, et cetera I compiled.

That that's what I would do if I had an infinite time as a legal researcher. And so, um, so in that, you know, I described like a half dozen prompts or so to get to. And other kind of technical processes, hitting a search engine, deduplication of of results, et cetera. Some of it's kind of boring, but needs to be done. Um, each of those needs to be tested as well, because each of those may be the area where everything falls apart.

So, uh, you know, you can have, uh, uh, a a result, you know, fall apart. Because step three or whatever in the in the chain of the processes, um, just does not get the right answer. So, between, like, having the right people in the room to make the right decisions, having a very extensive testing and then developing skills with the mind frame of like, what are all like? Can I break down into steps? All the things that an insurance adjuster or support professional, or a lawyer or paralegal, or whatever would do in this circumstance, whatever, whoever you're building for, um, and then test each of those steps very rigorously.

I think you're in a pretty good starting position. I would say,

I think, um, well, 1st of all, this is exactly what we tell our customers to do. So, um, I'm totally aligned with with the rigorous evaluation and and starting with evaluations and kind of making that, um, I think, Desi designing the evaluation is the most important part of making these things. Um, work, so totally, totally agree there. But, you know, I'm I'm thinking about why, you know, people don't, um, build these evaluation systems very well. And I I think one place where people get stuck is it's actually hard to do the evaluation automatically.

And especially when you're generating things, you know, you really want like a person to look at it, but it's probably not feasible for a person to look at and grade every time your system runs. So maybe you get a little specific. And in your case, it sounds like you really are generating text here. You need to automatically evaluate if that's the text you want. Certainly, it's not gonna be the exact same string every time. So how did?

how did you make that work?

so so there's two, two approaches here, maybe three that are worth noting. 1st is to the extent possible, especially for these like micro steps along the way. Make the make the out output like numeric or objective or something like that. Like, here's like a a tip that, so far as more people don't adopt. Um, the, uh, say, instead of making a decision about relevance, right? It's a generative AI model, so we all think like, Oh, it should like, write a whole like essay about whether this is relevant or not.

But actually, like, I think people just put too much, you know, emphasis on the generative and not enough emphasis on the intelligence. Actually, a very intelligent model. So if you, for example, be like number one through five, five being the most relevant, one being not relevant at all. How relevant do you think it is? That's gonna compare very favorably to previous AI models. It'll be a little slower, but if you're looking for like, raw intelligence, you know, GPT four to be like five, that's probably gonna be a very relevant thing.

And you can do little tricks like you can have it output the number 1st and then provide its reasoning. 2nd, you don't get all the benefits of chain of thought prompting, but you do get, um, it, we, we've. We've seen, we think, our hypothesis is. It thinks ahead about what it has to write for its reasoning, and therefore does a better job of throwing out the number. Either way, we've seen an increase in performance, I'm not sure why.

And if you use stop words to break off its reasoning, so add or, or, or, you know, total, total max tokens. So it, it thinks it's about to write you a whole essay about why this is the most relevant document ever. But you cut it off after the 1st token produced, so that's very objectively testable. Is this? Should this have been graded? A five? If yes, pass, if no, we gotta change our prompt for this, this test, right?

Um, for the things that's not objective, yeah, you, you have to do kind of like AI evaluation, um, for scaled stuff. But we also had, like, um, a lot of our customer support and success people, maybe 20 or so professionals, constantly checking these tests, like at least once a week, if not in the very early days, once a day. They'd be going over and hand grading some of this stuff as well, uh, so in the early days before I figured out AI evaluation that well, we, we, we would just took, you know.

And and kudos to them, they were. They were supporting our customers and making us successful people by day. And like, kind of like Batman or whatever, like, would turn on at night and like, check these tests. It was a lot of work, um, to to make happen. But, like, I would encourage folks to consider, like, is there an ROI? Um, to including, uh, some people who are, you know, good domain experts, in our case, our success people, almost entirely, if not maybe not entirely.

Former attorneys with deep domain expertise in their in their particular fields. They can apply to the tests, they created the tests and also were great at grading the tests. Um, like, maybe reallocate some people there, you know, it's, it's a. It's a really good use of resources if you can get a trustworthy, accurate product. And I think maybe the the biggest thing, by the way, is in terms of the people we were talking about earlier, the challenges that people have.

In terms of going from like, cool demo to actually is like. We just have this confidence, based on the intelligence of the model. That with enough prompting and work and guardrails and s breaking down steps, that you can actually get products that are are very, very, very accurate. And a very high number of times, um, and only fail on. Like the kinds of things that, frankly, young attorneys would also fail on. Not like, just kind of randomly or just hallucinating wildly or whatever.

There's a lot to be gained through prompting, a lot to be gained through, um, working against a good test set. A lot to be gained by decomposing tasks into smaller pieces when necessary, um, and a lot to be gained by just trying a number of different approaches. Like the the try, just having it output a number or a word, a single word, like true or false, or what have you, depending on the task? And and um, what?

What we've found through our experience, and I I doubt it's any different in any other field, and I think it'll be the same in most other fields, is. If you put in a ton of effort into those kinds of activities, you'll come out the other end with like a thing that you can with reasonably say. Yeah, this is operating at a level that that if it was a person, I would hire this person. It's not maybe not perfect, but like, yeah, I mean, like, you know, it's.

And and maybe when GPT five comes out, it will be perfect, who knows? Like, I, who knows what's happened, this next kind of turn of technology? But. But the technology today has a ten year roadmap of stuff we can build against it because it is good enough to basically be like, Yeah, I'd hire, I'd hire this guy if it were a person. So I should be willing to onboard this technology to to support me.

How much of your improvement do you think came from? Um, like tweaking the prompts versus decomposing the steps in the right way versus maybe other things you haven't talked about, which I would love for you to to mention, like, I don't know, for example, fine tuning or or other techniques.

Well, the the fine tuning stuff, um, and this is just us, this is not, I think, universally true. We have not found there to be a ton of value to fine tuning for our models. Um, and to be specific, we are doing a lot of experimentation with smaller models, and I think it's a huge benefit. If you can get a small model working for you, it's local, it is fast, it is cheap. Um, and and uh, I think that that the the frontier is gonna be these open source small models.

Don't get me wrong, we just haven't been able to make it successful for our use cases yet. Um, we've been using a, uh, not fine tuned copy of, uh, GPT four for the vast majority of the tasks that we do. And so, um, again, I'm actually a big believer, kind of iron, ironically, even though we're not using it right now, a big believer in the direction of that. We've seen some very promising stuff kind of in the lab, so to speak, and I think that's that'll be coming in the coming months and years from us and from from many.

Many others have already brought that into production. Just for whatever reason, like, we haven't seen, you know, uh, say, a Llama 70B outperform GPT four in a task, get close, but not outperform GPT four. Play with us, even if it's particularly fine tuned trained for, um, for that particular task for us, right? Um, so for us, that that isn't as big of, uh, a component as I know it is for other businesses. And and again, I'm very optimistic about where that's going.

I would say that maybe equal weight to to better prompting and um, and decomposable tasks. I, I would say that like, um, you'd be shocked about if you were willing to like, you know, have a have. An interface almost where you have all your tests in one hand and the screen, and your prompt in the other hand of the screen. And you tweak your, you tweak your prompt, and you see it's failing at three use cases, or ten use cases or whatever.

Do all those share something in common? Add an instruction, pass those, get a beta version, get it into the customers hands. And they're like, Hey, I failed this other edge case. Add to the test set, you know, tweak your prompt to make sure it covers that. You can get so far with that process alone, like, like, really, really far, just like seeing where is it failing? is it failing consistently? And like any good test set, you change one thing over here, then something breaks, something over there.

So you have to kind of accommodate for that, like all that happens. But um, you know, I just I said the main characteristic right now, and this this will probably change with GPT five and six and Gemini two or whatever. But the models aren't perfect yet, so they need a lot of hand holding and explanation. And and kind of sometimes frustrating, you know, working against this, like kind of models, you know, post training or whatever, to get it to really do what you want it to do.

Uh, but you can get there through prompting. And and the number one qualifier I have for some of my prompters is, are they willing to stay up all night for four nights in a row to, um, just make it work? Because you, you, you will find that the people who are able to kind of grind through and and test test, change the prompt, test test. After a certain time, like, Oh my God, thousands of thousands of tests are are passing, like, that's crazy, you know?

Uh, and once a thousand are passing, um, you can feel pretty darn good about the next 100,000, you know, and pretty good even about the next million. There's there's some amount of like, you know, tran, like, even though it'll be hitting new experiences, new documents, new questions, new whatever. Um, it's gonna do pretty well, like, if you're, if you're at that kind of level of testing, so I'd say, I'd say that's it. Prompting gets you really far, farther than than.

I think is is discussed or acknowledged. Cause I think most people just aren't willing to put in the like five sleepless nights, like, you know, grinding their way through, uh, prompting it. But once you get there, it's it's you. It's really amazing, like, what you know how, how specific and good you can get your models to be.

And so this is like, not this. This is really looking at like specific examples where the model's failing and trying to modify the prompt so it fixes those examples. And then it's kind of looking at what's regressing.

Yeah, yeah, exactly. And and and like, using the simplified example again of like, is this case relevant to this question? Um, you know, on a scale of, say, one to five or whatever, there there's kind of an objective answer there. You can debate between a three or four, maybe, you know, so, and sometimes sometimes you have to. When you're developing these prompts, give the model some like, Okay, that's an error from my grade set, but like, we'll let it slide.

You know, it's like, fine, like a human could totally just do the same thing. It's a reason, reasonable expectation. Um, but like, uh, uh, you know, we, we've, you know, basically seen that. Like, yeah, you can get it to say five when it should be a five, and four when it should be a four, and one when it should be a one. Um, and again, having the model output its reasoning, even if you don't use it in in in practice, I think is very helpful.

You can cut it off after a certain number of stop words, or, um, uh, you know, stop words, or or max tokens, or what have you? Um, ends up being really helpful. Uh, you know, just to see, like, why it thinks that this thing that's obviously not correct is correct? And then you kind of, you know, it's. It's not exactly like this, cause these models are not, you know, they're not brains like like we have. But oftentimes it will say something that's actually kind of reasonable sounding.

You're like, Oh, I see where it got the mistake, let me edit the prompt and then all of a sudden it's fixed. So a lot of it is like trying to find as as close to an objective ground truth as you can. As to what you know if, if, if the application developing warrants it, um, build an objective ground truth and then, um, and build against that, basically,

the and now is it the same process for hallucinations? I mean, this is one of the problems that we have with an internal bot, you know, that we made. I think you can iterate on the prompt, um, to kind of fix, fix the errors, but the hallucinations have felt a little bit different and and tougher, um, at least for us to remove. Did. Did you find any different strategies worked for you there?

yeah, I mean, I think I think that, and this is evolving. But I think the strategies to deal with hallucinations are are there are a few within within any prompt dealing with like source material. So, so, 1st of all, like, maybe the obvious one is, you know, nothing that we do for legal professionals ever is just from like the AI's like memory of the world, you know? The the analogy that you probably probably heard before, but but maybe bears repeating is.

It's like a student taking a closed book exam. And yeah, GPT four is like a really smart student. So it's read a lot, it's memorized a lot, like it's gonna. And by the way, the closed book exam was like partial credit for every answer, so it's gonna answer, you know, um, but if you don't give it that context, um, it's a closed book exam, not an open book exam. It's gonna, like, give a partial credit answer, as best as it possibly can.

That's where it's especially gonna make up stuff. So step one is add in information, say, read this 1st and base your answer on this, and no extraneous information, and and only based on this information, you know, provide your answer. Now it's an open book exam to some degree anyways, it's like it's, it has source material to refer to, and it's a very smart student. It's a very good reader, very good understander, even of difficult concepts and and language.

And so it's much more likely to get the right answer there. And that will. That will eliminate a lot of the problems that people have with hallucinations. When you're just S, for example, chatting with Chatgpt and like, made up who the 3rd president of some country was, or whatever it's like, you know, or what color his hair was, or what have you like? That's not gonna happen if if it has a Wikipedia page about the presence of that country.

Fully in context, as as reading it. 2nd thing is like, sometimes give it a less less is more, um, and this is this is evolving. Like, Gemini 1.5's million context window is like, I wouldn't necessarily use all a million and expect perfect results, but it can handle a lot of information and still get it the answer pretty right. So this is evolving. But at the very early days of GPT four, providing a smaller amount of information, as small as you can, can without corrupting the actual meaning.

Um, much more likely for it to get the right answer based on that. Um, the prompting itself matters greatly. H How exactly are you instructing it to to hang on to the context and not come up with stuff from extraneous information? Um, and then there are a lot of checks also, by the way, making cited sources, maybe put a number against every line in the document and be like, which line specifically are you getting this from?

You know, it may still hallucinate, but maybe a little less so. Um, and then then guardrail checks, like, you know, if you ask it to quote from a document, you can do a non AI kind of. Just like, is this quote exist in the document? kind of check, or even a fuzzy check. And if it doesn't, then you have to maybe have a a a set of processes to clean up process. Um, so, so a lot of kind of approaches here.

We we feel really happy about where we got CoCounsel to. Um, and it's still like a really difficult challenge, right? And and to test against also, and that's where the human testers also sometimes come in. Even the automated tests can sometimes be like, if it's according to a P, not this paragraph, but some other paragraph, it's hallucinating. So we automatically flag those wrong, but sometimes the human human graders to go in. It's very subtle, sometimes it's like, as in the word, not that changes the entire meaning, you know, um, and sometimes it's smack in your face, obviously it's citing a case that never existed.

Um, you know, and and and sometimes you need people to grade that. Right now, I'd say, to, to really get the the full answer. So it's not, it's not easy, but again, it's doable, um, and and doable to a degree. That's like, you can, you can stamp out, you know, for your customer's experience, like the vast majority of hallucinations.

Well, look, I really appreciate all the practical detail and I'm I'm like, loving this. I also want to use some of our time to ask you some questions, some broader questions about, um, you know, LLMs and legal issues, if that's all right, since you're kind of writing the intersection of it. I mean, I guess the one place I would would love to start is, you know, um, you know, because you're obviously a fan of, um, of GPT, and, um, you know, a lawyer.

I'm. I'm curious what you make of the kind of, um, you know, the fair use issues around, um, LLM training? Is that? Is that something you think about?

So I have to preface this with I have not practiced law in a decade, and um, maybe more than a decade at this point. I was not a copyright lawyer, so everything I'm about to say also doesn't represent, I'd say, the viewpoints of my company that I work for right now. Like, um, but so I doesn't necessarily disagree, but like, uh, I think it's a really tricky issue, like it's a super tricky issue. Because on the one hand, you have all this copyrighted material that's used to train it.

And there's some things that are like, pretty obviously copyright violations. For example, when, uh, an AI produces verbatim, you know, an image or a uh, text from S, from something else. If I did that as a person, like if I like verbatim copied out, like the text and like, passed it off as my own, that would also be a copyright violation. It's like, literally, you know, it's almost in the name, it's like, I can't copy, you know, the right is to prevent people from copying, right?

And then there's a lot of like, really gray area stuff, like the fair use question is something super interesting, uh, because again, if, as a human, if I sat down and read all these books and then maybe be smarter and then I answer questions based on that, I'm not violating any law, right? And so the question becomes like, at massive scale, um, you know, and for for computer processes, if it's any, if that, if that should be treated any different than like a person sitting there and reading and summarizing.

The whole point of fair use is like, you know, again, kind of like the name implies, uh, you should have some capacity to, um, e, even if it's just people. Sometimes it's businesses to read things, get influenced by them, transform them into your own words, your own thoughts, you know. Use the kind of facts that are out there in the world to, you know, inform your opinion. And and enable you to say, write another book that is also say, like, say.

I read all the Winston Churchill biographies and then wrote my own Winston Churchill biography. And like, cited to them and didn't copy any of the sentences or any of the expression, any of the ways that they they wrote it, um, you know, but. But it came with my own novel kind of interpretation of his legacy, like that, that's protected. And the question is like, when GPT four does the same thing about Winston Churchill, you know, um, whether it's kind of doing the same thing, and I guess I think also gets kind of thorny.

Like, you know, when? Um, it's a very different scenario when I'm just talking to ChatGPT, at least I feel as a different scenario. I'm just talking to ChatGPT. And it's using everything it's read from all these books and articles and so on, and sometimes kind of parroting back what it's heard versus when I'm asked to do like a novel task. Like, read this document from an internal to company document management system that's never been seen before by anybody, you know, and answer questions about this.

It's clearly engaging in in in a activity there, that activity is probably not covered by protection. The question is whether it's training was, and yeah, it's, it's. It's maybe the best I can say, and this is a very lawyer kind of answer is it's thorny. It's. It kind of depends on some of the circumstances. Um, and uh, and I think it's. It's a tough question for the courts to figure out, you know, I think it's especially tough, frankly.

We have these really bad facts, where some companies kind of like, took stuff, like stole stuff, basically, like scraped illicitly, or, you know, and that's part of the training set. I don't know if that changes the legality one way or the other. I'm not this close to a look at it, but they're really bad facts. For for the side of fair use and the side of, you know that that side of things, it's like, oh, well, you know, you kind of like, like, stole a bunch of stuff, basically.

And then are TR are trained a model on it and they're passing it off for free to a bunch of customers, like that doesn't feel right. So I, I guess Another area, there's also maybe, by the way, too, is like, Yeah, yeah, please go ahead.

oh, I was just I was just thinking that there are kind of policy implications, like if summarization gets way easier, right? Like, y, you know, if if most the way I consume the internet is now from, you know, a summarization company versus going to individual websites, it's really easy to imagine that. M changing the incentives to to make any original content.

Yeah, yeah, I mean, I think that's, I think it's absolutely correct. Uh, but that the caveat being like, um, you know, it depends if, like, the next activity after you read the summary is to read the the kind of source where it came from and citing its sources. And you're like, Okay, I need a deeper dive in this, um, or not? And also kind of maybe depends on the the monetization model, like, right, the way we all monetize today in MO.

For most internet content, is ads. So, yeah, you need to click on the website for this to be worthwhile. I can imagine an alternative world where, like, if I'm a great content creator, then I explicitly say to the Googles and OpenAis and stuff of the world. Like, you are not allowed to use this as part of your training. It's probably not like a copyright issue, it's like a terms of service issue or whatever. And if you want this, you gotta pay me for it, and you do want this, cause it's high quality information that make your model smarter.

And that might be the new, you know, the new business model. Instead of, you know, getting your ads through Google, you're getting your monthly kind of royalty check or what have you? So I do think it it, you know, can potentially influence the business model of of how content's created. Um, but. But I agree that under the current business model of advertising, it's it's gonna put a lot of pressure on people who, you know, um, create a content and made a living off of that.

And by the way, that that cuts both ways. Google recently, you know, last four or five years, if you Google search for, like, a for for many topics, and the thing that comes to the top is stuff that's like geared towards number one in SEO, right? Like the classic example is you search for a recipe, and the ones that, for whatever reason, Google's ranking highest includes a person's like frigging life story before they get to, like, how to make, you know, grilled cheese.

Um, it's like how grandma used to make really grilled cheese, now, important to their family, that they make the same. It's like, I don't care about any of that stuff, but Google upranked that over shorter, more to the point, kind of recipe books. Um, so people started doing it, and it made it kind of like a terrible place to be. Cause everybody's optimizing not for, like, content, information, intelligence, but instead for, you know, what gets ranked higher in SEO.

And and you can imagine a world, a different world, where, you know, this stuff actually aligns incentives again. And the, you know, s somehow determinations are made that this content is good content. It'll make my model smarter. It's not just like fluff. To get to the top of Google for advertising revenue and and great content creators, um, and even the content creators who are now kind of forced into making this stuff for SEO purposes, start making much better content.

Um, you know, I guess there's a possibility that it goes that way as well. So, um, so I think there, there, there could be a positive dynamic comes that comes out of that.

That makes sense. Um, well, look, I, I'd love to, I'd love to end with a question from our head of legal. Uh, that heard that I was interviewing you. And she, she asked something I think really interesting, which is, um, you know, how do you see what your work and others work around AI is influencing the education of future lawyers? Like, how how do you expect the profession to change? And would you have any advice for someone in law school right now on on what they should be learning?

You know, it's interesting, cause I think, um, what's gonna happen? And whether we do it or other companies do it, or the industry will, or, or, honestly, even opening eyes models just get better and handle some of this be themselves. Is a lot of the tasks that lawyers used to do as young attorneys, um, are no longer done by young attorneys. They're, they're more and more done by AI, and we're talking about tasks that are like, there's a million documents, review in this case, each of you get 100,000.

You have like seven weeks to click through every single document and tell me if any of them seem relevant to the case. Or, um, you know, basic document creation or comparing documents, or taking that that older version of the contract and then kind of brushing it up to make it make sense with the new term sheet. Um, that we just got, you know, for for this matter. I think those are the kinds of things that AI will will, you know, be increasingly capable of handling, um, better and better and better, until the point that that young associates don't do that anymore.

And I think for a lot of people, that's really scary. Because, um, that's how they were trained when they s entered practice is like they s they 1st did the grunt work, so to speak. What, um, one of the managing partners of the one of the largest firms in the country I met with him, he's like, what will we do if these guys don't have to dig ditches anymore? He's like, dig ditching is the uh, uh, uh, ditch digging was the word he used, right?

What do we do if, um, uh, if that's, if that's the case? And, uh, I guess here's, here's my feeling about it. My feeling is, um, al right now today, uh, we currently do things with technology that, like 20 years ago and 40 years ago and 50 years ago, would be like, unthinkable. Like, literally, lawyers used to use typewriters and go to the library to do research by opening up books. And now you just there's no control F or no search function on the book, you just flip through the book.

And then maybe a reference to another book, which you go pull from the library and read that other book. We are way more efficient and do things that we never did before. Um, thanks to simple changes like like word processing, email, the internet, you know, uh, and and that that didn't like, destroy jobs or make it like harder to train attorneys, like, be able to start doing more valuable stuff with their time. And from day one.

And and right now today, if you ask a lot of young attorneys, um, how it's like, li, uh, working at a law firm? I think a lot of them will say like, I am not doing the work I went to law school to do. In fact, I can't believe that someone with this much debt is being paid this much money is doing this task. Why can't it be done by somebody else already today? So I think that, like, AI is gonna put pressure on those kinds of jobs, um, and those kind of tasks, as I was saying, not jobs, but tasks.

And and continue to uplevel people as they have been with the advent of like, internet and word processing and so on, right? I think it's gonna continue that kind of directional trend. And the truth is that that law school as an education is a really good preparation for the world, for for law students entering the the profession. Um, in a world after AI has automated a lot of the BS, you don't actually learn any of the BS in law school.

That's actually the great trick of law school is you spend all your time in law school reading cases, reading contracts and having these like kind of amazing debates about strategy and policy. And then you show up at the law firm and they're like, read these a million documents, get this all read these a million documents, you know, and like, don't come out until until you've read a million documents. All the emails being exchanged from this company you've never heard of about the the most boring topic you can possibly imagine, right?

You're like, that's why you have a lot of turnover in this profession. Um, and so I think, I think that that, um, law students probably don't need to change much, but what they're doing in the classic legal education. I do think that they have to probably start experimenting with AI. And and because I think, maybe to expand on this kind of thought, I think this is true, not just in law, but everywhere. As soon as you have these AI assistants that are super capable, um, the activity, like, the kind of the.

I'd say that what makes lawyers great today pre AI is like grind, like, am I able to grind through more time, more documents, more whatever? That's how you, like, prove your value almost as a as a lawyer. And I think that that delegation will become the most important skill set, right? Um, I now have this AI who will grind for me. Um, am I expressing what I need to do clearly enough that my little employee, my little AI employee, will do it well?

Am I getting feedback about the work that it's doing? Am I delegating the right things and holding the re the other, the other things for myself, right? Um, the camp management as a as a kind of art and science, I think, is gonna be start to be applied to younger and younger employees across all businesses. Um, and like other great managers, if you can figure out how to get leverage by delegating more and more to your, to your subordinates, in this case, your subordinate AI.

Um, that's gonna really go far for you, and so I think that's that's gonna be one of the main skills that will distinguish. Um, like, as as in, say, the 2010s and before, 2020s and before, lawyers who distinguish themselves by billing more hours, working harder. And just like, you know, still reading documents accurately at four, zero am. as they're clicking through the next million, million documents. That will become less important, and what will become more important is people who are able to deploy, um, the the tools they have accessible to them and delegate effectively.

And I think it's actually a bright future. I think it's much more fun, interesting, challenging, intellectually, you know, intellectually challenging. Um, and I think people will will ultimately like that. It'll be a it will be a difference. And I think that that's maybe the one thing in law school that you only really get experience through trying it, you know, a and get like knowledge around it by trying it. So I, I encourage people to start just pick up ChatGPT today and and see what it's like having this AI assistant.

And and can you leverage yourself? How little work can you do now that you have this AI assistant? And what are the things you still have to hold onto yourself? Where is your delegating capable? where are you capable? I think those are the kinds of, uh, I think, interesting questions for for law students in the future.

All right, Jake, thanks so much for your time, really appreciate it.

Thanks for having me, it's fun talking.

Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.