4 Jan 2024 · 57m · Lukas Biewald, Jerry Liu · 15 quoted facts
Jerry Liu argues that the practical way to make large language models useful on private company data is retrieval augmented generation (RAG). The model stays fixed, and the system fetches only the most relevant pieces of data and puts them in the prompt. He presents LlamaIndex as a vendor-neutral toolkit for building this, and says measurement and tuning matter more than the choice of vector database.
The main tension is between simple, reliable systems and ambitious ones. Liu says basic RAG works today and can be improved step by step with evaluation sets, chunking and retrieval tweaks. Fine-tuning and fully autonomous agents are harder, slower, or still unreliable. He also expects bigger context windows and OpenAI's own retrieval tools to change the field, but not to remove the need for retrieval.
Liu recommends collecting at least 30 to 50 question-and-answer pairs so every change produces a number you can compare to a baseline. A stronger model such as GPT-4 can grade answers against the ideal ones, though it is noisy and costly.
Liu says swapping vector databases rarely changes retrieval quality, so the choice mostly follows deployment needs like self-hosting, cost and speed. The language model and the retrieval strategy affect answer quality far more.
The easy first steps are tuning how many chunks are retrieved, chunk size and parsing. Too little context leaves questions unanswerable, while too much raises cost and speed problems and the model misses information buried in the middle.
Liu says fine-tuning needs more expertise and takes at least ten to twenty minutes to show results, while RAG gives results in about a minute. He sees it as worth trying later, for example to reduce hallucination when cost rules out GPT-4.
With two or three tools, an agent can usually figure out what to do. Liu says that in open-ended settings, as with AutoGPT, current models are not reliable even with an unlimited budget.
Jerry Liu describes LlamaIndex as an open source data framework for building applications on top of large language models. Its mission is to connect those models to a company's own data, such as PDFs, spreadsheets, databases and APIs.
RAG (retrieval augmented generation) keeps the AI model unchanged and instead fetches the most relevant pieces of your data and places them in the prompt so the model can answer from them. Liu says this is faster and easier than fine-tuning, giving results in about a minute.
Liu says to write a set of at least 30 to 50 questions with ideal answers, then score the bot's answers against them. This can be done by hand or by asking a strong model like GPT-4 to grade each answer from 1 to 5.
Liu suggests using a better model, giving stern prompt instructions, and designing the product so users can click through to the source passages. Later on, some teams fine-tune the model to say it doesn't know when the context lacks an answer.
Liu doubts it for now. He says models still miss information in the middle of very long inputs, and sending a gigabyte of data on every call would be slow and costly, so retrieval stays useful for large datasets.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Jerry Liu made the first commit of the LlamaIndex project on November 1, 2022 and announced it on Twitter on November 8, 2022.
“the 1st commit was on November 1st of 2022. Uh, I posted this on Twitter November 8th of 2022, and I started hacking on LLMs”
LlamaIndex was incorporated in April, raised funding at the end of April, and announced the company in June, according to Jerry Liu.
“it was officially incorporated April, um, and then, uh, did the fundraising like end of April, announced it in June”
Jerry Liu did AI research for about two and a half years at Uber before founding LlamaIndex.
“did AI research for like two and a half years at Uber, um, have worked in ML, uh, for most of my career”
Jerry Liu names LangChain and Haystack as other frameworks for building applications on large language models.
“there, besides LlamaIndex, there's also LangChain, there's Haystack, there's a few others”
Jerry Liu estimates that over 75 to 80 percent of LlamaIndex users focus on retrieval augmented generation.
“I would say probably like over 75 to 80 of our users are focused, um, on the RAG use case with LlamaIndex”
RAG stands for retrieval augmented generation, a term that Jerry Liu says came out of a paper a few years earlier.
“RAG stands for retrieval augmented generation.”
Jerry Liu says the LlamaIndex documentation contains over 300 guides.
“Like, for instance, we have over 300 guides in the LlamaIndex”
Jerry Liu describes evaluating answers by prompting GPT-4 to score a generated response from 1 to 5 against a ground-truth response.
“you can basically prompt, uh, GPT-4 to output a score between 1 to 5 on how much your predicted response, like the generated response, matches the ground truth response”
Jerry Liu cites Zephyr-7B, a Hugging Face model fine-tuned from Mistral, as an open source model showing more advanced abilities.
“a recent model like Zephyr-7B by Hugging Face, which was, I I think, fine-tuned on top of Mistral”
Jerry Liu says even the best fine-tuning services take at least ten to twenty minutes on a very small dataset, while RAG gives results in about a minute.
“even the best fine-tuning solutions these days take at least like ten to twenty minutes over a very small dataset to give you back anything”
Jerry Liu says OpenAI's Assistants API, released a few weeks before the recording, is essentially an agent execution loop with tool use.
“if you look at the assistance API that OpenAI released a few weeks ago, um, it's basically this”
LlamaIndex data agents let developers choose an execution loop such as the ReAct reasoning loop or OpenAI function calling.
“you have your choice of, uh, React, like the, you know, general chain of thought reasoning loop, um, open AI function calling, like a while loop over OpenAI”
The LlamaIndex team refactored and rewrote much of the codebase in April, a rewrite led by co-founder Simon.
“we ended up having to kind of, like, refactor, rewrite a bunch of stuff in April”
Jerry Liu says LlamaIndex had almost 500 community contributions to its open source project and almost 200 to its hub.
“I think we're almost at, like, 500 contributions on on open source, a and then, uh, like almost 200 on on the hub”
Jerry Liu cites context windows of 200K for Anthropic's newest Claude model, 128K for GPT-4 Turbo and 100K for Claude 2.
“and and Anthropic has a 200K context window. So, like, you know, the GPT-4 Turbo has 128K, uh, Claude 2 has 100K”
Liu explains the project's mission of connecting language models to private data and how it differs from other frameworks.
“is a data framework for building LLM applications”
Defines retrieval augmented generation and why the model is left unchanged while data is fetched into the prompt.
“you don't train the model anymore, you fix the model.”
Explains why you need a small curated test set and how another model can act as the judge.
“define some sort of like, uh, eval benchmark”
Covers tuning retrieval size, chunk size and parsing, and matching fixes to specific pain points.
“tuning your top k, your chunk size, um, your parsing strategy”
Walks through summarizing each page and retrieving by summary first, which suits large collections of short documents.
“you can basically do what we call hierarchical retrieval.”
Lists model choice, prompting, interface design and fine-tuning as ways to reduce made-up answers.
“some models just hallucinate more than others”
Shows how to pull structured fields out of text and why agents are powerful but unreliable beyond a few tools.
“do retrieval, then do, um, structured data outputs”
Liu recounts the first commit, the viral launch and the company's incorporation.
“the 1st commit was on November 1st of 2022.”
Liu argues that models still lose information in the middle of long inputs, so retrieval will remain needed.
“There's still a lot of red in the middle”
You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lucas Bewell. Jerry Liu is co-founder and CEO of LlamaIndex. He started his career in machine learning and autonomous vehicles, and since then has built one of the most popular frameworks for building text segmentation and text retrieval applications. This is a super practical conversation, and I hope you really enjoy it. All right, Jerry, good to good to talk.
Yeah, thanks for Good to see you.
having me on, Lucas.
Yeah, thanks for thanks for joining us. Um, I feel like where this needs to start is what is LlamaIndex and how could I use it?
Okay, yeah, getting straight getting straight to it. Um, so so LlamaIndex, uh, uh, for for those of you who don't know, uh, is a data framework for building LLM applications. Um, our goal, our company mission statement, is to connect the power of LLMs to your company's org, uh, data, right? Whether it's like personal data or organizational data. Um, this includes everything from PDFs to CSVs to PowerPoints, Excel sheets, uh, to APIs, databases, pretty much anything. Um, and we kind of realized this all the way back before the ChatGPT hype, uh, when people were just starting to hack around on GPT3.
And I think as people were developing with it, they realized that it had a lot of potential, but it had a lot of potential. Not just for memorization, but also for reasoning. And so as people tried to actually feed it their own data and try to stuff the prompts, there were certain patterns that emerged in terms of the best practices for how do you actually orchestrate data for use within an LLM. And so we kind of found ourselves at the center of that by launching a project very related to this, uh, around November of last year.
So it's been a crazy ride. It's been about a year, uh, since we started the project, uh, and now we're basically a very mature framework to help users build LLM apps over their data for a wide variety of, uh, different types of use cases.
So you're a framework for helping build LLM apps, and, you know, there's like a lot of, um, a lot of projects that might call themselves that. So, like, how do you fit into the ecosystem? Like, what's your, like, really sweet spot?
Yeah, so there's a few frameworks for helping build LLM apps. Uh, you know, there, besides LlamaIndex, there's also LangChain, there's Haystack, there's a few others. Our focus on, like, certain niches or areas. I think there's certainly overlaps with a few of these, but our focus has always been on, um, connecting LLMs with data. And so, if you actually take a look at the development of the framework, we've kind of there's certain things that we haven't really done.
Um, so, for instance, things that we haven't really done include, um, kind of, uh, a deep focus on, like, very strong, like, prompt abstractions, um, or a deep focus on, for instance, like, agent simulations. Um, a lot of our abstractions have tailored or been tailored around stuff like RAG, retrieval augmented generation, agents, and the use case of data analysis, search and retrieval, and being able to act upon information, uh, conversational memory, those types of things.
Yeah, I guess these days, when I think about data applications with LLMs, I I'm thinking RAG, like, how, um, is that like 100% of your use case or or most of your use cases? Like, how do you think about that?
I'd say it's the majority of our use case. I I think a lot of users are using the framework for stuff like, uh, structured data extraction or adding, like, agentic capabilities to RAG. It also kind of depends how you define RAG, because I think RAG in its very simple form is just stuffing stuff into a vector database and doing some sort of like, top K retrieval and search. Um, so you can certainly do that, you can index like a large document corpus.
I would say probably like over 75 to 80 of our users are focused, um, on the RAG use case with LlamaIndex. There are also users using LlamaIndex for, uh, structured data extraction, uh, agentic stuff, summarization, and basically kind of using some of the modules in more of an ad hoc manner.
So maybe we should break down what RAG is for folks who who might not know, like, can you, can you kinda walk me through how RAG works?
Of course, I think if you've been on, like, the Twitter or AI space or listened to podcasts recently, uh, I'm sure the term RAG has been thrown around a few times. And I'd say it's actually grown in popularity, probably since around, like, middle or earlier this year. Uh, RAG stands for retrieval augmented generation. Um, and it's kind of a term that's evolved over time. Um, it was a term that I think came out of a paper a few years ago.
Um, the high level idea is actually pretty simple. It's basically you don't train the model anymore, you fix the model. And then all you do is figure out how do you import data from wherever your data storage system is, whether it's a SQL database or a vector database. And you figure out how to stuff it into the prompts, right? Because once you actually stuff stuff into the context window of an LLM, then you can ask questions over the stuff that you you put in.
Um, and it's basically a a very common trick people figure out very early on with LLMs, especially with GPT three, 4,000 context window. You can just put in relevant context, and then, given the relevant context, all of a sudden, the LLM can reason over new information. So the trick is, how do you actually do that? If you have, for instance, like gigabytes or terabytes of data, you need to do some sort of retrieval mechanism first, so that you fetch just the most salient pieces of context.
And then, given the most salient pieces of context, you feed it to the LLM to answer the question.
And I guess, like, a key thing here is actually finding the most relevant bits of information in your in your database. Like, when I when I think of that, I'm always thinking vector database. But you said SQL is also possible, like, how does how does that work?
Yeah, I think that goes into the part where maybe different people have different definitions of RAG. And so, um, uh, for a lot of people, SQL probably doesn't actually fall into that definition. Typically, that lives in a separate stack, like a text to SQL type stack. Um, I think for, uh, for us, our our general interest is kind of imagine this like hypothetical scenario, right? Maybe taking a step back just in terms of the end user use case.
You have a chatgpt like uh interface, and then you have, like, just a mess of data sources, and somehow, just magically, you connect all these data sources to ChatGPT, regardless of the diversity or or size of this data. And then you can ask any question you want and get back the answer that you want. Um, and so there's a lot of complexity involved in actually setting up a system that can do that. But that, to us, is kind of this like holy grail of of RAG that we aim to achieve.
And I guess when you, um, when you set this up, like, h- how does it work? Like, do you are you opinionated on which vector database to use? Do you think it it matters?
Um, that's a good question. I don't think we're opinionated on the vector database themselves. We have a rich ecosystem of vector database providers. We have 35 plus vector databases, uh, integrations, and so pretty much any of the the ones out there you can use with LlamaIndex. We are starting to develop opinions on the best algorithms, though, depending on different use cases for ingestion as well as retrieval. Like, for instance, like, there's a lot of techniques floating out there, each with different, uh, costs in terms of or or trade-offs in terms of cost, latency, and performance.
Um, so, for instance, do you do like top k hybrid search on top of your data? Do you do, um, proper, like, what type of chunking do you do for like code files, uh, for SEC documents, for like legal filings, those types of things? And so that's something we're still actively working on, but we haven't, uh, fully formed the opinions yet. I think my typical my idea here, especially, uh, as an open source project, is that it's nice when you start from the bottoms up.
Because at least you give, uh, users the tools to kind of, like, hack around on things them themselves, like the developers, to build whatever custom solution they need. And once you become a bit more opinionated up the stack, you consolidate some of the practices into these abstractions that you can provide, uh, for for users.
Okay, so maybe let's get specific. Um, at Weights and Biases, we have, like, a whole bunch of different LLM use cases, um, but one that I think really fits fits your model is something called WANBot, which is actually in our Discord server, where it searches our documentation and answers questions about how to use, um, weights and biases. So if you're me and you're kind of just starting out, or you wanna maybe, um, improve this this WANBot.
What would be the kinds of things, um, that you'd that you'd think about? Like, what would be kind of maybe your 1st recommendation of a vector database? What would be kind of things to try? What would be the the right way to chunk, um, the stuff? So that we're answering questions well,
Yeah, that's a good question. Also, if I'm not mistaken, I think WANBot actually uses LlamaIndex, I don't know, though. So so we,
it does.
uh, we did, we did promote it. It's a it's a great project. Um, I think there's a few things to keep in mind. So let's say you're a developer, um, you follow the str- uh, quick start guide of LlamaIndex or watch a few YouTube video tutorials. And you've set up, like, a RAG pipeline over your system. And so you've imported some data, and now you want to, you know, you've asked some questions over the data, and then you realize that for some questions, it's not working.
So now, what do you do? Um, the 1st step that I typically recommend is to, um, define some sort of like, uh, eval benchmark, like, uh, basically, collect some data set, and it could be like 50 data points, um, you know, like 30, 40 is probably the minimum, but just like some curated collection of data, uh, uh, of of data points. And the reason for this is, like, you, you want, like, whenever you're trying or tuning something, you want to, you know, get a number out of it and make sure that whatever you're trying has a higher number than the baseline, right?
And so this is a good way to kind of, like, benchmark different strategies. Once you actually so this is actually have,
this is kinda tricky, right? Because, um, you know, for for weights and biases, there's there's lots of, like, possible answers that could be okay, right? So how would I, how would I set up an evaluation data set that I could reuse, uh, with all the different strategies that I might try?
I mean, yeah, the so so good good question, I think, uh, to be fair, evaluation is something that I don't think everybody does. And so this is more just a kind of like principled setting, uh, uh, the the ideal. And, of course, there's technical challenges in how to do it. Um, honestly, the simplest is probably just get a human to do it, just get a human, uh, like, uh, go through like 50 questions that you ask this bot and just write down the questions and answers.
Um, there are synthetic ways of generating this too, and we have some tooling in LlamaIndex to actually synthetically generate an eval data set for you.
Okay, but before we get to synthetic, just say, say, I've like, you know, I've, I've labeled some questions and answers. Are you So, saying, like, look at answers and mark if they're they're good or bad, or literally, like, pose a question and be like, this is my ideal answer here?
yeah, so, um, I guess, uh, for for just a very basic data set for evaluating RAG, could just be you have an input question and then the output answer. And so basically, you come up with the input and then you also give what the ground truth output should look like. Um,
But, of course, it's not gonna exactly match the output.
of course, and so that's why LLM-based evals is kind of, um, there there's a lot of companies that I'm sure, I'm sure, you know, uh, as well, and and uh, also efforts going on in in weights and biases, I'm sure, towards, like, analyzing this. Um, a typical way that you can basically compare, um, or or do evaluation of a predicted response from the LM via, uh, and compare it against, like, a human, uh, ground truth label is just use another LLM as a judge.
And so there's a lot of, uh, uh, papers out there, as well as, like, kind of, uh, works on, like, fine-tuning this stuff. To basically use, for instance, like, GPT-4, to measure the quality of a response given a predicted one. Um, and this is still a work in progress. There is definitely imperfections, there's noise, um, but as these models get better, they typically exhibit a bit less stochasticity and more consistency in in being able to, uh, compare different responses.
Okay, so now, say I did it, and I have done it, and our our, uh, this is a true story. Our 1st accuracy we measured was around 60. Doesn't But really feel good enough to put in front of customers. Like, where, where do we look to, um, improve our our flow? What's what's like the the typical lowest hanging fruit?
uh, yeah, so, um, I've given actually talks on this in a variety of different settings. Uh, and I think, to be honest, like, our own sense of best practices is constantly evolving. And so, like, you know, it's very possible that a month from now, after the time of this recording. Like, I might have slightly different opinions on, you know, what are the best things you should try first? But I think typically the 1st things you try, and and I have this like diagram, but that basically outlines, um, there's kind of easy things, like table stakes, things you should try, and then there's harder things that you should try.
Basically, easy things will be more incremental and maybe, uh, nudge up performance by a few percentage points. Harder things will basically just unlock new capabilities for your entire system. Um, easy things to try include, like, tuning your top k, your chunk size, um, your parsing strategy. Uh, typically, those are probably some of the main culprits. Um, if you have, like, for instance, uh, your total retrieve chunks is too small, you're not gonna have enough context to actually answer a lot of questions.
If your total context is too big, you start running into speed, cost issues, as well as like, loss in the middle problems. Uh, so low precision, um, and the thing is, this typically depends on the on the use case as well, and and the types of questions you want to ask. Um, the other thing, the other kind of way of looking at this, actually. And I I have like a slide that I haven't really shared publicly, but maybe just like, uh, a mental map of how to think about.
This is basically like a table where, like, the left column is the set of pain points that you're running into, and the right column is like the set of solutions that that solve some of these pain points. Um, so instead of looking at it in terms of, like, oh, easy stuff to like hard stuff, like a pain point, for instance, could be very specific. It could be like, Oh, this retrieval is actually not giving me back the right results.
So if the pain point really is around, like, retrieval, then what you wanna do is you want to invest in better retrieval algorithms, right? You want to use something to actually, given a question, give you back the most relevant context, uh, to answer this question. If the pain point is the LLM is the retrieval's fine, but it's the LLM is hallucinating, then you wanna invest in potentially more advanced prompt engineering, right? And and um, and and also the model itself, uh, and Well, let then the other,
let can we, oh, sorry, go ahead,
Yeah, um, oh. And then the last thing I'll just say is like, there's also just maybe a a one way of looking at this is. There's a huge long tail of questions, uh, that are very complex, that a user might want to ask. Just think about, like, the spectrum, right? Like the distribution of all types of questions a user might want to ask. Um, the simple, simple RAG pipeline that you set up might answer like 20 or 30 of them.
Um, but there's just like what you can in invest in, for instance, like agentic strategies or things that basically make the entire capabilities of this pipeline more advanced to answer some of the more advanced stuff.
okay, so let's let's talk about let's get into all those. So, you know, 1st of all, this sort of like chunking, what do you think like for our for our use case, right? We're trying to answer questions about documentation, just, you know, questions about how to use weights and biases. We're looking through docs. What? Where would we start with chunk size and and how would we do it? Would we kind of slide a window across our documents and and take like every token and and put in our database, or should we, like, go in steps or blocks, or how?
How? What do you recommend there?
so let's let's talk about documentation, actually. I think documentation is is quite interesting, cause we we've done a few of these, uh, uh, case studies as well as tutorials as well. What are some properties of documentation? Um, documentation, there tends to be one a lot of different files. Uh, if you look at the octree of like a, you know, an entire, or, sorry, like the table contents of like an entire, uh, uh, project's documentation tends to be very large.
Like, for instance, we have over 300 guides in the LlamaIndex. Um, each page itself tends to be not super long, it's usually not like a book, it tends to be like relatively concise, and also the format of each page tends to be like in Markdown. Um, and so given this, um, one way of thinking about, like, documentation specifically, is that when you ask a question, like, and and this reflects human behavior too. You first wanna find whether or not you're you're actually f- uh, heading the right pages, and then once you actually hit the right page, you wanna make sure you're heading the relevant context around the page.
That actually answers this question, right? Like, for instance, if you have a question about, like, the arguments passed into the specific class, you probably wanna fetch the API reference of of, like, the the project. Um, if you have a question about, like, the quick start tutorial, like, or any tutorial, you wanna fetch that page. Um, and so like, basically, based on the data that you have, and in this case documentation, it's, you know, large corpus, small documents.
You can kind of, like, have create, like, basically create intuitions around, like, the types of retrieval strategies to use to actually query this document corpus. Um, one general strategy you can do when, for instance, you have a large multi-document setting and each document tends to be, like, somewhat diverse, is you can basically do what we call hierarchical retrieval. So you can, um, index every document by, like, a summary, right of that document. So that when you first ask a question, you first retrieve the relevant, um, summaries, right, uh, corresponding to the question.
Each summary then maps to the underlying document source. Um, so the final context that's passed to the LLM could be basically the entire document corresponding to that summary, or you could also do top K within that document as well. In the case of documentation, what we typically found is it's like short enough that sometimes you can actually just stuff the entire thing into the context window, right? And so really, the key issue is finding the right pages to begin with.
Uh, but that's just an example of, like, you know, given different kind of settings, you can kind of craft your intuitions to develop the right retrieval strategies here.
so let me just make sure I'm I'm laying this out. So so you summarize each document using some kind of LLM, maybe probably GPT-4 or something like that. Could be even be an expensive one, cause you do it once, then you compute the embedding of each summarization, then you match the embedding of the question to the summarizations. And then you look within the document that has the best summary match for the kind of top-k snippets that match the query.
yeah, so that last step is optional, but every everything you said up until that point, uh, is is spot on. So you you summarize documents you retrieve first based on summaries, um, and then once you actually fetch a document, you could just return all th- that entire document as a contiguous chunk, right? You don't really need to chunk up the document, uh, but if you really want to, if it's a long document, you could go down and and retrieve more from it.
I see, I see. And now I I feel like sometimes people probably feel overwhelmed by the number of choices of LLMs, the number of choices of, like, vector databases. Like you said, you had to go with, like, 35, um, uh, vector databases. There's probably, like, more LLMs out there these days, and it gets confu- more confusing every day. Like, where do you start? Like, is there sort of, like, a basic setup that's, like, you know, kind of best practice here in November 2023?
Yeah, I think, um, vector databases, I'm kind of playing Switzerland right now, and so I I, I don't know, like, I I think, you know, I I think the the storage layer is also kind of evolving pretty rapidly as well. As as every vector database adds new features, you know, at the beginning of the year, there there wasn't really, like, good metadata filtering. Now I think every kind of, um, vector database has some sort of like structured information storage, as along with unstructured information.
Um, I think, really, for vector databases, it comes down to some aspect of your existing deployment environment, like if you're already using existing systems, um, stuff like cost, latency. It's really not performance related. I think what I s typically spend most of my time thinking about is like performance related stuff as well as like, uh, how that trades off with cost, uh, and and speed. And And when you so that,
say performance, you mean the quality like, accuracy, of the matching, accuracy of the matching. Yeah, gotcha,
yeah, so so this includes accuracy of, uh, the final retrieval, uh, retrieved results, and then the accuracy of the final generated output. Um, typically, like, alternating the vector DB isn't gonna ever affect the quality of retrieval, unless, you know, maybe one uses a slightly different, like, nearest neighbors algorithm for approximate nearest neighbor search. But like, you know, it's, it's all roughly the same, right? And and I feel like for most data volumes, typically people deal with small data anyways, so you don't really need to worry about approximate nearest neighbors.
Um, the LLM, though, does impact the quality, um, and so we actually have this LLM like full, uh, list, right of different LLMs, um, from paid to open source, as well as how well they do on what we call easy to hard tasks in LlamaIndex. An easy task is naive, like RAG, uh, so the easiest RAG setup possible in LlamaIndex, a hard task is, uh, agents. So, basically, how do you, uh, get an LLM to actually do tool use, interact with different services, and actually do stuff like chain of thought reasoning?
So, um, what we found is that currently, like, for paid LLMs, OpenAI, like GPT-4, is still the best at, like, agentic reasoning. And I mean, that's a pretty, like, common, uh, sentiment. I think most people use GPT-4 when they're building agents for, uh, RAG. Most LLMs can do RAG these days, like, um, open source, like Llama 2, 7B, um, all the way to, you know, the Anthropic, Cohere, and and OpenAI models. One thing that I'm very interested in seeing for the LLM side is the open source, um, LLM capabilities evolving into the more advanced stuff, so agentic reasoning, structured data outputs, and and, um, stuff like text to SQL.
And we're actually seeing, um, like a recent model like Zephyr-7B by Hugging Face, which was, I I think, fine-tuned on top of Mistral, uh, is actually exhibiting some of those properties. And so as these open source models get better, then they can actually start doing some stuff that, uh, previously only proprietary, like GPT-4, uh, like models could do.
is there ever any reason to use a different model for the embedding versus the actual, like, QA?
Um, uh, if the question is kind of like, uh, cause, cause, well, right now there's like embedding models and there's LLMs, right? So, like, for the embedding model, it's like, typically, you know, like, uh, ada 002 by OpenAI, like a Hugging Face embedding model, and then the LLM is typically like a generation, uh, based model. Um, but I think, like, may maybe kind of a broad question is, is there any reason to just, like, have a mix of different LLMs within your um, uh, within your app?
Uh, I think that's certainly possible. I think, for instance, like, um, one example architecture, uh, that I think is pretty interesting, is for a lot of use cases, actually. And I've talked to a few users, uh, some of them want, like, routing, like, and and so, for instance, like, given, uh, a question or a task, route this task to an underlying choice. So it's just a, uh, a choice selection problem. Um, and if you use GPT-4 for that, yeah, it can do it.
I mean, it'll be a little expensive and slow, but you could also just, like, fine-tune, like, a very fast, small model that that's very capable of doing this, right? And and so that's something that, uh, feasibly could happen, especially if, you know, you want this LLM to do a specific task very well. Um, I my my take is, I think, to start with, most people just use the same model, um, just because it's easier. And then as they start thinking about cost and trying to optimize that, then, uh, the performance, once they hit the performance threshold, they try to optimize the rest of stuff.
And and maybe then they bring in different models.
and so you're saying you wanna play Switzerland for the, um, the vector databases, but could you just say, like, what? Some popular choices might be?
Yeah, sure, I mean, I can list off like five-ish, um, five, five to ten, and and by the way, if you're a vector database and you're not on this list, like, please, please don't, please don't be offended. Uh, the the um, there's, there's Pinecone, there's Weaviate, there's Chroma, there's Milvus, there's Qdrant, there's ActivateLoop, um, there is, uh, Redis, MongoDB, um, and PGVector, and I think there's way more. But, yeah, I just listed off a few, um, Nice, thank you, thank yeah,
you. Um, and do any any of them stand out with like different functionality? I guess some of that would also, like, function as your database.
I mean, I I think again, it's less like performance and more just like system-related stuff. Like, for instance, um, Pinecone is a SaaS, right? I or I think I don't know if they deploy on cloud now, but you can't. It's not like self-hosted. Uh, if you wanna self-host something, you you use something that, like, uh, some people use Chroma, some people use like Qdrant, um, some people use like PGVector, um, some are like other databases that add it on vector database, uh, capabilities, so Mongo, um, as well as like Postgres PGVector.
Um, but like, roughly speaking, I think it's more just like, um, more a function of, like, your kind of deployment setting and and system requirements, as opposed to, like, any sort of performance gain.
Got it, got it, um, and now what do you do? And this happened to us, I think you alluded to it. If the if the model starts hallucinating, like, we were pretty concerned about the model, just like giving totally wrong, um, answers and and documentation to our users, like, like, how do you? how do you work on that? how do you fix that?
Yeah, it's a good question. I think there's a few off the top of my head. And again, going from, um, like, like, the initial things you should try to stuff that's maybe a bit more involved. Um, the initial things you should try is probably just use a better model, um, if you can afford it. I think, uh, typically, honestly, like, some models just hallucinate more than others, and unless you fine-tune the model, um, it's just hard to prompt engineer your way out of it.
The 2nd is like prompt engineering. Um, I think sometimes if you make a very stern statement that you must return this, if it doesn't actually know, then it'll do it. But sometimes, you know again, this is a property of the model. Um, the 3rd is is like some sort of, um, like, it's the product UX, actually, that you design quite, uh, matters quite a bit. Um, for instance, like, I think people have moved on beyond the basic expectation of, like, you know, what you see is what you get in terms of the response, and more like you can actually click into the response to look at different sources.
Um, there's novel UXs you can define too, like, maybe, um, this. There's some arrow drawing from like different parts of the response to like a highlight in the source document. That's basically what we did for, uh, some of the full stack apps that we've built, like SCC Insights. We have a full PDF viewer, so you can actually just, you know, for every response, you can go in, click the source and actually just look at the highlights.
So it's a very direct way to kind of see if they correspond to each other. Um, and then in terms of the model capability side, um, I think one thing that I think pretty much everyone that's maybe a bit more advanced and far ahead in the LLM space is doing is fine-tuning the LLM. Um, to to varying degrees of success, like basically trying to fine-tune it to be better at not hallucinating. And by that, I mean, like, somehow you get it to bypass, you know, its original instruction tuning, um, given enough data and human feedback to basically tell it, you know, for this, like, kind of given that you have this question and this context, you clearly don't know the
right answer. So stop making stuff up, like, return, like, you don't know, or something. And I have talked to a few users doing that, um, but I think that typically happens, uh, a bit later, you know, once, once, once you really need to.
It does seem like fine-tuning gets easier and easier every day. Like, do you do you feel like it's best practice at this point? Or, um, at what point do you do you fine-tune? H How much data do you need to really get a meaningful improvement?
I think right now there hasn't been. I think it's been a gradual change, but not a step change in the capabilities of fine-tuning. And so I think fine-tuning, typically, you know, um, is still seen as optimization. Um, so tuning something that exists and making it better, uh, is just ha- It just happens that it tends to be more accessible now to everybody, whether you're fine-tuning OpenAI or fine-tuning Llama 2. Um, a lot of companies popping out that allow you to, like, just fine-tune Llama 2 out of the box.
Here's the main issues why fine-tuning is not as popular as just building RAG these days. Like, just, you know, taking a pre-trained model and stitching together some stuff. One is, um, the like conceptual understanding, uh, it tends to take more to actually understand how to do fine-tuning, like, are you gonna? How do you format the data? If you're not using an API, you have to write like PyTorch code, which, um, most people honestly don't know how to do.
And then the 2nd is, um, the the time to value. Uh, even the best fine-tuning solutions these days take at least like ten to twenty minutes over a very small dataset to give you back anything. Whereas, um, RAG will take you like a minute if you know what you're doing, like, it'll, it'll just give you results, uh, very fast. And so, you know, if even for a very experienced developer, they'll probably wanna see something quick 1st, right?
Like, just the easy thing to do 1st before they actually try, uh, fine-tuning. Um, what I'm generally interested in, and and this is maybe a little bit tangential to your original question, is, uh, step change in fine-tuning. I think, um, it would be, I I think that would probably force us to rethink our own abstractions. But imagine there's some like new capability that came out through fine-tuning that made people realize, wow, like, I always want to be fine-tuning on top of any data.
Because this is a way to, like, memorize information, ingest new information about the world, and basically get it to to do better. And so far, I just haven't seen that, uh, dip.
Interesting, so, y- I guess you think fine-tuning is probably not worth the the effort generally.
Uh, no, I I think it, it's, um, it's like the funnel, I think it's definitely not worth at the top of the funnel, it's probably worth like middle to to bottom of the of the funnel. I guess the funnel of AI development as you think about, like, productionization, um, and and think about stuff like, like, costs, um, uh, increasing performance in in specific settings. So, for instance, that use case of like, all else fails, and you have a cost constraint and you don't wanna use GPT-4, try fine-tuning a model to to s like hallucinate less.
So you mentioned, um, text extraction, um, which, in my, if like, boy, like, more than 15 years of experience is one of those things that's like so much more popular in real-world applications than in in research. Like, I just feel like there's always this big mismatch. Um, I'm sure, like everyone in the real world who's who's listening to this, it has some kind of text extraction use case they can think of. Um, can you walk me through how that might work, like a real-world example?
Actually, I'm kinda curious, Lucas, do you have a real-world example from your side on text extraction? Ooh, weights and biases Yeah,
good question. We haven't done text extraction yet. I think one one place we might look is pulling out, um, information from transcripts of sales calls. So, for example, like, a question that we might ask is like, what fraction of our customers are using AWS versus, um, versus GCP versus Azure? And and, you know, maybe we don't actually mark that reliably in our sales force. We probably should, even when we, you know, ask customers these questions. So I could imagine a situation where we would wanna pull out, okay, like, what adjacent technologies, what cloud providers are our customers, um, using?
We should actually definitely do this, as I as I start to say this, this would be incredibly useful for us right now. Um, we have to, you know, we're relying on our customers to support people to mark it somewhere. And and I don't think that happens all the time. So how, how would that? Um, how could we? how could we do that?
yeah, it's a good question. I think these days, um, just, uh, very concretely, um, I think the way most people do it is. There is increased support for kind of like structured data formatting for some of these models now. Like OpenAI, um, has supported function calling for a few months, and then they just released JSON mode on their new models. Both of which, at a very high level, um, are designed to help, like, the output be like a formatted kind of, uh, semi-structured, uh, piece of text, as opposed to just like free form, right?
And so this is very relevant for text extraction. Because imagine, you know, um, you're analyzing some piece of content and you want the out for output to specifically be, hey, you know, I want to see if, you know, uh, like, let's say it's whether it conforms to certain guidelines. You want to output like some Pydantic object, like, uh, a schema with like a few boolean fields and some numbers saying, you know, like, here is my rating, like one to five, like here is like the whether it c uh adheres to something true or false.
Um, and so it's possible to do that, especially with these types of models. Um, certain models are better at it than others. Actually, structured data out, uh, extraction is something that a lot of open source models tend to struggle with. And so if you're gonna use like some sort of just general text extraction model, like the better models, still, uh, tend to be better. The main, um, s, yeah, the main systems challenges with structured data extraction is, um, cost and and and latency.
Um, I think with longer context windows, the maybe kind of depends on your use case, but like, you could imagine latency might go down a bit. But if, um, you're running GPT-4 on, say, like a gigabyte of data, just raw text data, that is probably gonna cost you a lot of money. And so, like, um, you ki- kinda have to be a little bit careful and selective about, like, how do you actually do extraction, uh, from like a giant corpus of documents.
One other pattern I've seen, which is very interesting, is you could do some si- something like 1st, especially if it's giant corpus of documents, 1st do retrieval, then do, um, structured data outputs. So it's actually very similar to just like a RAG pipeline, but you just add in like some structured data extraction at the at the end of it. And so, like, you know, given some giant corpus of data that's much bigger than what the context window can handle, retrieve the relevant chunks and then just make sure the output is some sort of like structured field.
And so therefore, whenever you ask a question, you can basically get back some sort of a structured, uh, result from it over a large corpus of data.
cool. Um, can you say more about how you think about agents? I feel like I've seen so many evocative, um, examples of agents, but, you know, they're, I know, they're like, notoriously hard to get to work. And we actually haven't yet, I think, really gotten any agents really working for our, um, use cases. I mean, how? How do you think about that? How do you support it? And and what's the state of the art of agent-based systems right now?
Yeah, I would actually be curious if you were able to get it working, cause that that'd be a pretty interesting, uh, case study. Uh, I would love to chat about that, cause I think I think your sentiment echoes, like, some of the other, uh, enterprises that we've talked to. I think a lot of people are building RAG, I think RAG, uh, it tends to be very popular because it's relatively simple as a concept, but also easier to implement.
Um, agents are very powerful. I do think agents are probably the future. Just it's like, current state is gonna be a little hard to, um, fully realize that vision. Um, I think, maybe taking a step back, people have different definitions of what an agent is. Um, some people think of an agent specifically as like a loop, um, like execution loop with tool use. Uh, if you look at the assistance API that OpenAI released a few weeks ago, um, it's basically this.
Like, you give it some task, it'll go ahead and solve it, uh, plus, like, maybe do some function calls on tools that you provide. Um, similar to, like, a React loop, uh, like, you know, the React paper that came out last year. You do some sort of chain of thought processing, tool use, and interleave that in, uh, prompt-based fashion. So I think that is one definition of it. I think that tends to be pretty popular. That's basically for all intents and purposes, like, um, the way we implement agents in LlamaIndex, and and so it's some sort of execution loop with tool use.
But, you know, honestly, like, the the idea of an agent is really just like reasoning, like anything that's more agent-like is just things that do more reasoning. And so even things like, um, basic routing, where you, uh, get a question and then you have an LLM pick and choose what is the right, like, answer to route this to. That's basically agentic, cause you're using some sort of automated decision making. Um, even, like, structured data extraction is kind of like agentic.
I think the stuff that makes it more agentic is if you start, um, like, basically chaining some of these reasoning calls together so that it does more stuff at once instead of just a single step. Um, I if it actually is able to take actions that affect, like, surfaces, so, uh, can do like writes and not just reads, right? And and then the 3rd is some some vague notion of of memory, um, or just like being able to retain information over time.
But that's kind of like more of a subset. I think I think that probably matters, uh, for some agentic use cases, but but less others. The people that think about that, uh, are more thinking about like personalization, where you can create like customized, like chatbots that you know, like your your personalized AI companion, like the character.ai and the function AIs of the world. Um, but there's a few definitions floating around. Um, I think right now, basically, um, with the current model capabilities, even if you had an infinite budget, I think it's hard to get reliable, uh, agents fully working.
Um, in a general, unconstrained setting, it is very possible to get it working in a constrained setting. So we're seeing, you know, if you have, like, uh, two or three tools at most, and you give that to an agent, that agent can generally figure out what to do with these tools. Actually, if you take a look at a lot of our apps that we put out, um, for RAG, that's basically an agent using, um, on top of some tools that do like vector search and retrieval.
And so we're actually extending a little bit beyond just like the top K RAG concept by adding in agents as well. It's a very powerful concept, but the moment you p- you extend it too much, right? And you saw this with the initial release of, like, AutoGPT, as well as all the other agent projects. Um, you let it do a lot of infinite things, and, yeah, it can do some of it, but it's not gonna be able to do all the very well.
Interesting, so so what else? What other tool would you give, um, a RAG agent besides lookup?
Yeah, it depends how, um, creative you wanna be with it, actually, I think, um, for instance, a basic RAG use case would be give it access as a tool, uh, to a vector database, and also give it access as a tool to a SQL database. Um, uh. Additional capabilities you could augment it is web, so you could, if it doesn't actually find the information, uh, do stuff to look up stuff from the web and then add that information to the database.
Um, some other examples here include, um, being able to use this data to then automate your workflows. And so this definitely moves a little bit beyond the RAG use case. But imagine drafting up an email, sending it, and also, like, um, kind of sending a calendar invite, those types of things.
My co-founder made a, um, like a code-generating agent that seems to work pretty well. Doesn't always work, but, um, you know it, uh, it's really, actually very impressive.
yeah, that's really cool. So code, actually, I haven't really talked about code at all. Code is very interesting, cause it's basically when you write code, it's like the language that, like, writes it's like programs, right? It kind of writes itself. And so it's kind of like the tool that an agent can use to construct, like other things that can execute things. And so that that is very interesting. You start getting into, um, basically, like, um, I think an interesting concept for agents is building things that can build themselves and then do other things.
And you can see this in various flavors. You can see this in terms of, like, an LLM being able to generate code that can then, uh, execute. You also see this with, like, GPTs, uh, that OpenAI launched, right? And like, it's it's the whole idea of, like, you use natural language to program something that can then do, like, a certain task, right? And so that, honestly, like, if you can do that, maybe that is one criteria of of AGI, right?
Like, cause, like, just generate things that self-replicate.
totally, totally. What other, um, surprising use cases are you? Are you seeing out there?
Uh, in terms of the surprising use cases, it's a good question. Um,
Or maybe actually, I, I, what are the use cases that besides this kind of RAG workflow? What else is like working, like, what are companies out there doing, um, with with LlamaIndex, where, where they're finding value?
I think the main use cases is around, um, search and retrieval, um, summarization and structured data extraction. Those have continued to probably be the most popular use cases that we've seen. Um, I'll anything that's very workflow heavy, we haven't seen as much. Um, and and one thing that we have seen, but that we personally have not really done as much is, uh, using, um, LLMs to basically index code, uh, and and so being able to look up stuff that's very technical or code, and a code-like in nature.
So, for instance, helping you, like, do refactors, helping you, like, do like Copilot, like, uh, stuff. Uh, we just haven't invested as much into that. Another thing that we have seen kind of, but but haven't really invested too much in is, um, just like web automation stuff. Like, for instance, like, you're, uh, you know, you, you go on a page and you can, like, automate stuff, like, checkout experiences, those types of things. Like, it, it's just we, we, we haven't really invested that much, honestly.
Um, the the the kind of like the the area of this, like vertical, if you will, of search and retrieval over data is still very complex. And I think, like, no one's actually solved it that much. And so that's why we're s we're still, uh, investing a lot of effort into this.
Can you tell me the story of of LlamaIndex? Like, what were you thinking when you started it? How did it become a company?
Yeah, so I think I've told variants of this story a few times. Um, and maybe I'll just drop in a few details just to see if if there's anything, uh, new that I I didn't share. I mean, I I started this project, um, uh, the 1st commit was on November 1st of 2022. Uh, I posted this on Twitter November 8th of 2022, and I started hacking on LLMs, um, basically late October of 2022. And so this was around last year.
I had started playing around with LLMs, GPT-3. Um, honestly, I got into it pretty late. I wish I got into it earlier. I've been following generative models for most of my working career, cause I I I did research on generative models. Like, I, I, um, was, uh, did AI research for like two and a half years at Uber, um, have worked in ML, uh, for most of my career. Um, but basically, you know, I was hacking around with it and and it was one of the 1st realizations.
And what we s what I said at the beginning was, how do you actually, you know, make this thing useful over your own data? And that got me down this rabbit hole. I created this, um, kind of exploratory design project called like the GPT-like tree index, where I organized data in some sort of tree. Launched on Twitter, got woke up in the morning, got like 300 likes, got really excited cause I was like, wow, like, this is, this is amazing.
And then that kind of motivated me to, um, basically lean into this like developer, uh, kind of, uh, this like open source flywheel, basically, if you will. Like, if when you're building out in the open, there's a lot of feedback constantly coming in, you're seeing what other people are building. And so it just got me really excited about the space. Um, and it was at the time when a lot of people were discovering applications of of of LLMs and GPT.
So it was a pretty opportune, uh, time.
And when did you turn this into a company?
Uh, it was officially incorporated April, um, and then, uh, did the fundraising like end of April, announced it in June. Um, and then, yeah, and and basically had had talked to, uh, my now co-founder, but, uh, he, he was a good friend of mine. And and, uh, starting in January,
Cool, did you wanna talk about your data agents release, like, what, what is that? and and how? How would somebody use it?
yeah. So, um, going back onto the thread about agents, uh, data agents is basically our implementation of agents. And a lot of the way we design these abstractions, as well as how it hooks into the framework, is centered actually around, like, RAG use cases with with additional tool use. So it consists of, uh, a few different parts. Um, what is a data agent? Um, we call it a data agent. It's basically just for all intents and purposes an agent that we, uh, like to think is a bit more focused on data, like data search, retrieval, usage.
Those things cont- and consist of a few different components. One is the execution loop, so you have your choice of, uh, React, like the, you know, general chain of thought reasoning loop, um, open AI function calling, like a while loop over OpenAI. Um, we also wrap their assistance API so you can just directly call it if you want. Um, the 2nd component is tool use, so you can plug in a bunch of tools. We make it very easy to basically define any tool that you def- uh, or any kind of, like, um, RAG system, uh, in LlamaIndex as a tool to plug into an agent.
You can actually plug in agents as tools to other agents too. And so it's very easy to create, like, hierarchical agents that can do again focus on, like, search and retrieval use cases. But that's kind of what we're focused on. Um, and 3rd aspect is some aspect of, like, kind of conversational memory, um, more for the chat use case. Um, and so those are some of the core components. Uh, a lot of the use case that we've highlighted is how adding, like, agentic, like this agent execution loop on top of your RAG pipeline can enable more sophisticated behaviors.
You can ask multi-part questions like, Oh, you know, can you tell me the answer to this? and given this answer, tell me the answer to B? Or can you, um, you know, do summarization of this document for me? And then compare it against, like, this other document, like a resume, for instance. Uh, enables more capabilities on top of, um, your existing RAG pipeline. The other piece is we have, uh, like 40 to 50 plus tools on LlamaHub, um, which is our open source hub for community-driven contributions.
Um, to just which consists of a bunch of tools for users to just use to plug into their agent. So if, like, you wanna do beyond, like, you know, more stuff than just RAG, you want to actually, um, like, like, send an email, uh, send a calendar invite, do Google search, um, generate images. Even you basically plug those in as tools too. So if you're interested in contributing a tool to LlamaHub, basically to to listeners like, please feel free to.
We're always looking to for more contributions.
Is there any like missing functionality that you'd especially like to see?
Uh, there's always new tools that we could add. Like, you know, imagine if you can plug in, yeah, especially nowadays with all the multimodal stuff, like plug in, um, like text to speech, plug in text to image as a tool like Dall-e 3. And then you you hook that into your, uh, other tools that the agent can use, and then you just have some automated system that can, you know, generate images along with, like, reasoning over your information.
Totally. Do you worry at all that, um, you know, folks like OpenAI will replicate your functionality? Like, is that? Uh, do you try to stay away from stuff that they might do?
Um, I mean, to some extent, there is maybe a little bit of overlap with, like, retrieval, um, the retrieval API they launched. Um, I think, yeah, a few people have asked me this. I think my overall response is, I think, yeah, LLM providers are gonna move into this space. Like, uh, OpenAI has a retrieval API now, kind of, um, it's part of their assistance API. Uh, I think Anthropic has some RAG thing going on, um, so does Cohere.
Uh, and then, like, I think, uh, AI21 as well. So I think my general take is, I think that's fine. Um, you know, it is what it is. Uh, there is certain advantages to designing kind of vendor-agnostic retrieval because this enables users just to have way more options in terms of, like, you know, you can pick and choose whatever LLM you want. And then we really just focus on kind of making the retrieval part work well, but it will plug in well into your model.
Um, and and I think, like, the bet is basically, as long as there's kind of like a healthy kind of space of, like, LLMs, then, um, this, uh, like, the. There'll still be kind of need for third-party orchestration services that easily integrate with, like, existing models that you use.
Hm, when you think about, um, LLMs, evaluating LLMs, do you have any best practices there? I think it's often surprising to to people who first hear about this, that that's like a good strategy at all, like, how, how effective do you find it? And are there any best practices that you could share?
Yeah, I think we are actually working on it right now, so I'll probably have better insights in like a month or two. Um, because we're kind of doing some research on what, what's like, yeah, how exactly do you do this? And and there's a lot of open source projects that do it better than we do. And and we, we don't want to, like, really own this space where you. We're just more than happy to, like, kind of, um, just have general abstractions, up, plug into other libraries.
Um, some overall concepts for how do you do LLM eval? Um, if you have the ground truth, uh, like response, you can basically prompt, uh, GPT-4 to output a score between 1 to 5 on how much your predicted response, like the generated response, matches the ground truth response. So given this, um, if you make the number range too wide, GPT-4 gets confused, uh, but if it's small enough, then, like 1 to 5, it typically can can reason over stuff.
Um, you can also do evaluation in the label-free setting. So, for instance, like, if you actually don't have ground truth, that's actually fine. There's some general things you can do to make sure that it's not hallucinating. So, like, given the response and the context, can you, um, make sure that some of the context supports the response, right? So it's not just making up facts. Um, you can determine whether the response is, like, adhering to guidelines, relevant to the question, and those types of things.
So far, I think most of our evals use, uh, some variant of GPT-4. Um, 3.5 tends to be a bit noisier, um, so the the variance, uh, of results tends to be higher. Um, but we anticipate that as models get better, they'll get better at being a judge. I think that the main issues I'm personally running into for for evals is, uh, cost and latency. So I think GPT-4, the way we implement it, is kind of inefficient.
And it's running up like big bills on my OpenAI account. And so if we can somehow even fine-tune, like embeddings or smaller models to to do eval as good as bigger models, that'd be amazing. I think, uh, we just haven't really done that yet.
So there's quite a proliferation of open source projects in your space, but it does seem like LlamaIndex has really emerged as like a a leader. What? Why do you think that happened? Like, is it, do you think it was timing, or do you think it was some architectural choices you made, or the way you managed the community or listened to users? Like, what, what do you credit the, um, the growth to?
Yeah, definitely, a lot of it was timing. I think, um, if we started at, like, November of this year, it absolutely would not have taken off. I think I think it was just we happened to think about these things at the right place at the right time. Honestly, I think that's also why I said, like, I feel like I started a little bit late. Like people were already getting excited about LLMs, like in September, uh, August of 2022.
And I think if I, uh, started doing stuff, then I would have been able to do stuff sooner. Um, I think the other piece is, um, yeah, I mean, so some aspects of like execution and and like, oh, when you're designing a framework, you gotta make sure, um, this is something I I learned over time. Um, honestly, I wish I d did this from the beginning. Like, building a framework is a little bit different than just being a backend engineer at, like, a company and trying to build like a service for internal use.
You gotta make sure basically everything you build is gonna be, uh, used by other people, right? And it's gotta have like good documentation, and it's gotta have good abstractions. So you're not really building a system, you're building a toolkit. W- So you need modularity, composability, just like core engineering principles. And so, honestly, like, I think we added, like, a lot of, uh, stuff to LlamaIndex in the early days. But and we ended up having to kind of, like, refactor, rewrite a bunch of stuff in April.
Um, just because it just wasn't at the point where we could actually help, like, um, basically easily add new abstractions and make it a useful framework. And so, honestly, the refactor, the the like, just rewriting the entire thing that my co-founder Simon led, really helped. Um, like, like, encourage the growth of a future growth of this framework. Cause if we didn't, we'd probably be stuck a little bit in a place where, like, yeah, it was useful for a specific purpose, but it would be hard to actually add new abstractions and things to to advance this.
So part of it is like the engineering.
It must have been a scary moment to do a refactor when you have kind of a popular project, like, how did you weigh that?
Yeah, I think at a certain point, we just realized that it kind of needed to be done. The layers of tech debt was was, like, becoming a little bit hard to actually adapt to new things we wanted to do. Uh, so agents, um, were like different retrieval algorithms, like parsing, ingestion, um, basically kind of decoupling it, making it more modular, making it more useful for users. Uh, that was a conscious decision we made.
Got it? And and what fraction of your code is written by you versus community contributions?
Uh, well, me, uh, my co-founder, Simon, uh, our founding team, uh, and then community contributions. Uh, these days, I haven't written as much. I I don't really write that much code. I mostly just like, maybe put out some tutorials and guides. Um, these days it's really most of the the team working on both the open source as well as kind of like this, uh, enterprise stuff. Um, I think in the early days it was, it was mostly me.
But now is it, is it? It's mostly your team that contributes, or are do you get, like, useful 3rd party, um, pull requests?
Oh, I see, yeah, yeah, we've gotten a pretty decent rate of community contributions. Um, I think we're almost at, like, 500 contributions on on open source, a and then, uh, like almost 200 on on the hub. Um, but it's it, it always could be more. I think, oh, that's the other thing, which is why, um, if you want to encourage, uh, open source contributions. And and this is basically advice for anyone running an open source project.
You need to have a good, like, good set of like tools and abstractions. Cause if you don't set that up, then people like, won't they? they won't like, even if people are excited, they're not gonna know what to contribute to. And so if you have, like, the right base classes set up, basically, uh, a lot of people will, like, kind of add the integrations.
And how do you see interacting with big text corpuses evolving over the coming years? Like, do you think, as context length gets bigger and bigger, the way we do it will change? I mean, you could imagine even putting the entire corpus in into a context, like, is that possible?
Yeah, so it's it's an interesting question. I think I'm not discounting the possibility, like, I mean, I, I, I can't make a negative statement against that. I can only say say what the current state is today. As of today, like Anthropic, uh, 1 was a 2.1 just came out, and and Anthropic has a 200K context window. So, like, you know, the GPT-4 Turbo has 128K, uh, Claude 2 has 100K. Um, and I was just reading some analysis on, like, the attention, like, the ability of the LLM to attend to different part like areas of its context when you stuff the entire context window.
There's still a lot of red in the middle, meaning the LLM can't actually find things, uh, well when it's in the middle of the context. Um, and so I think there's still real risks to not only cost and and latency, but actually performance when you try to stuff, like, an entire LLM full of information right now. Right? And that will get better over time, but I think it's just, like, for a lot of users, um, it probably still makes sense to do some sort of retrieval from, like, your large knowledge corpus instead of just, like, trying to always fill in as much information into the LLM as possible.
I think we're constantly keeping an eye on this, though, like, uh, context windows will get bigger, um, the, like, cost will come down, uh, we'll probably need to adapt some of the paradigms. I do think retrieval will probably be necessary for basically anybody that has datasets greater than, like, a gigabyte just because, like, you know, even if you think about network transfer costs, like, you if you need to send, like, a gigabyte of data to the cloud for every inference call, it's gonna take, like, a long time to for for you to actually do that, right?
And so having more efficient ways to, uh, structure the interactions between the LLMs and data would really make sense.
Okay, we, we, we always end with 2 questions, and I'd love to get your take on these. So, um, what is what is an underrated aspect of machine learning LLMs that you think people should pay more attention to?
Yeah, I mean, I don't know if this is the answer you're you're looking for, but I rode in a Waymo for the first time 2 days ago, and it was awesome. I I I used to work in self-driving. Um, I think, like, and I had I took about a, like, two to three-year hiatus. Rode in a Waymo. It's it's fantastic. I think I think if everybody rode in a Waymo, they'd get excited about, uh, robotaxis again.
And what do you think is the biggest, uh, challenge of getting LLMs to work in the real world as of today?
Yeah, the biggest challenge is probably, um, a good chunk of the conversation we had, uh, like, at the beginning of this, uh, which is stuff around evaluation and making sure you have the right strategies for actually tuning your entire system, right, and not just the prompt. Um, I think, basically, uh, there's a big education and knowledge challenge in terms of how do you apply LLMs for different use cases, and we're help hoping to help figure that out for a lot of users.
But, basically, like, that includes instilling the best practices as well as, like, understanding the best techniques to really maximize the value that you can get. This includes RAG, but also, like, agents, like, pretty much most use cases that people are building today.
Awesome. Well, thank you so much. Um, we'll put some links to the stuff you mentioned in the, um, in the show notes. But, um, a lot of fun to talk, and congrats on such a successful project.
Awesome. Thank you, Lucas.
Cool. Take care. Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.