Unlocking the Power of Language Models in Enterprise: A Deep Dive with Chris Van Pelt
Gradient Dissent · 16 Nov 2023
Source: https://wandb.ai/site/resources/podcast
Transcript: https://podcasts.opennash.com/shows/gradient-dissent/unlocking-the-power-of-language-models-in-enterprise-a-deep-dive-with-chris-van-05a3a8/ (via OpenNash Podcasts, machine-generated)

Hello, my name is Lavanya Shukla and this is my co host Caryn Marooney, and you're listening to Gradient Dissent Business. Welcome to the show, welcome everyone. Our guest today is Chris Van Pelt. Chris is the co founder of Weights and Biases, which a lot of you might know already. Before Weights and Biases, Chris founded Figure Eight, and Chris has been doing machine learning for over 15 years. So he was doing ML before ML was cool, before it was called deep learning.

Um, I've personally worked with Chris for more than four years and I've learned a lot from him and I'm super excited to talk to him. Chris, welcome to the show.

Thanks for having me guys, excited to be here.

Thank you. So, um, Chris. In this show, we're taking a little bit of a different approach from Lucas, and we're trying to understand, like Karen said, the people behind and the stories behind the products. So I have a lot of f. We all have a lot of friends who are in that state where they are showing their product to their 1st few users. And it's easy to get lost in, like, chase down r wrong rabbit holes, or like, not know what the right 1st user is.

So, like, how do you identify them and like, make sure you don't like? Customize the product too much for them? And then how do you keep going with the current idea when you're hearing no? When do you know that you actually should?

Yeah, I mean, I think early for us. One of the things we did right was to partner with, you know, people that were actually doing machine learning, like we knew when we started the company, we wanna help people do machine learning. That was that was like, very clear. Um, there's a number of different personas within that in that space. And we were fortunate enough through some relationships that we had built at at CrowdFlower Figure eight to be able to, like, talk to a team like Toyota Research Institute.

And in in those discussions, they were very open ended, right? We barely had a product, uh, so we would be going in and and kind of talking to them about the kinds of challenges is that that they were having. Um, you know, it was like Luke, Sean and I would drive down to, um, Silicon Valley, have one of these meetings. And then on our drive back up, we'd we'd be processing it and we'd be like, Hey, you know, that person said they really wanted that.

Should we maybe, you know, look into building something there? What? How can? How can this all tie in to to make something useful? Um, so, I mean, I'm already forgetting your your original question, Lavanya. But I I think the advice would be, you know, find, find folks that you enjoy being around. Like we. We really enjoyed going down and and talking about their issues and what they were working on and thought what they were doing with.

Like autonomous vehicles and and making machines able to, like, navigate the real world, and they had like sick robots like in tier, it was just a joy to go to that place. Um, and we got along. And because of that, they were more open and sharing more, and that allowed us to kind of process and and and decide what was important and what direction we would take the the product in early on. Without those partners, I think we we likely would have made, um, a lot of the wrong things, for sure.

Great? So that's really good advice of find customers that excite you, and I think that's good advice in general. Uh, when you're picking co founders, the people that like, uh, you're working with at the beginning, like, if they inspire you and excite you, that's probably, uh, good things are gonna happen there. Yeah, I mean,

I, I've heard you speak before, and you get this real love for ML practitioners like it comes through and I, it's a joy to like, watch and see. How did you find this love of tools? So you were like, I wanna build tools like, why? What drew that?

well, like, as someone, you know, I mentioned, I studied art and computer science in in college, and like, as an artist, tools are like, fundamental, right? I like, need the paper, I need the, the paint, the the pencils and and whatever other interesting tool I might be using in my in my art media. Uh, I mean, literally, I did a lot of sculpture in college and would use, like, you know, welders and saws. And what we would think of as like things in a tool shop.

Um, as a developer, uh, tools make me feel like pretty intensely, and they they can make me feel either very elated or incredibly frustrated and angry. Um, we care about our tools a lot as a as a developer, and we spend a lot of time with them. Um, so, I think, you know, between art and the development I've done over the years, and the discovery of like, really good tools and the suffering through, like, really bad tools, it it felt obvious to to work on, uh, tools.

yeah, I, I get that. I just. It's interesting, cause a lot of people just don't aren't drawn to the thing that's most useful to others. I wanna go back to something. You are a technical founder, you know you? And yet you get on the airplane and you talk a lot to customers and you sell. And I know a lot of technical founders are like, I would like to outsource the selling part. What, what do you say and how do you think about that?

Yeah, I mean, I I think, I mean, not everyone's cut out for selling for for sure. And I, I don't remember a time where I was like, Oh, I really need to hone my selling skills. It it came, you know, somewhat naturally to me, and by no means do I, you know, believe that. I'm, uh,

I would agree with that. just this like I've seen you do ace up your sleeve, it. uh, seller. But

It might just be your curiosity or your interest in the people. That's sort of what's coming across versus I love what like

they're doing, you know, the people, and like, what problem are you solving? How can I help you? That's like how Chris approaches, like these selling conversations.

us, I guess, you know, I don't know. Like, sometimes I feel pretty introverted, and sometimes I feel pretty extroverted, right? There's not. I wouldn't say I'm necessarily, you know, really one or the other, but I do enjoy, uh, group settings and like people laughing and and connection and the the feeling of energy you get. Um, when you, you know, you feel seen, heard and understood like that, that's a that's a thing that I go after. And I I think you definitely need to be wired that way to to go and and sell.

it. It sounds like you can do both, because when you're coding, that's in your own, and then you go and you get the energy from other people, so it sounds like you play in both. Lavanya, are you an introvert or an extrovert? I'm gonna guess, but I don't know what do you? What would you say?

I feel like people think I'm an extrovert, but I am an introvert with a lot of energy. I think, what about you?

Uh, probably the same. When I was at Facebook, I was gonna write an introvert's guide to an offsite because I would find myself hiding in the bathroom. Um, but I really do enjoy talking to people, so I didn't know how to square that. And Chris, that really resonated with me when you said that. Because I like to work with people and I get energy from it. But then I need a place to go and think and retreat.

So I don't know what we call ourselves, then it sounds like we're all a little similar. yeah, I

think, ooh. One thing we all have in common is like, if you find your people, then you can be extroverted all day long, right? But like, you gotta find that energy that Chris was talking about earlier. This, uh, makes me think about, uh, Chris. You've talked about this thing called the T group, so you are watching you as a leader is so amazing, cause you are so empathetic. Like, if there's anyone in the company who's going through stuff, you really feel for them and you really lead when people, people are going through a tough time from your heart.

How much of that comes from the T group? And can you tell us about how that's shaped your leadership style in general?

uh, yeah, well, maybe we should like level set and explain to listeners like, what? or attempt to explain what T group is.

Yeah, I don't know what it is, it's new to me.

well, I'm actually I'm going to a T group tomorrow. Ooh, I'm really excited. Uh, this, this will be my 3rd T group. Um, T group's, it's. It's a difficult thing to describe, it's it's much better like experienced. But essentially, like, ten random startup leaders are gonna go on a little field trip, uh, Thursday and Friday this week. Um, our kind of guide and and moderator, uh, a woman named Anna Marie is is gonna be hosting the the T group itself.

There's, you know, a number of kind of, it's, I guess it's a little bit like it's not like Fight club at all. There there are rules about T group, though. Uh,

And are you allowed to talk about it? Isn't the rule with Fight club, you can't talk about it?

no, that's not one of the rules of T group. Uh, but uh, yeah, we try to stay in in, like the moment, in the here and now. So it's not like a group help session. Where we're talking about all the trials and tribulations that happened in the past or the things we're worried about in the future. It's really about, like, what's happening in this group? And we're SEN, we're like in a circle, so we're all like looking at each other.

And we spend like the entire day

how many people?

just kind of in an open discussion about 1010 folks. Um,

Why do you do it?

so we learn a lot about how we're perceived by others in a in a leadership context, in a group context. We're also encouraged to kind of come into it with some goals. So maybe I wanna participate more, or I want to be able to receive feedback better, or I wanna be able to give feedback better. Um, and it can get like, intense, so, you know, people will be very frank about how they're feeling. I mean, we're.

We're taught to really sit there and think about what feelings are coming up, and often those feelings aren't good. And we're gonna tell the other people in the group that they made us feel that way. Um, and it, I don't know. I've come out of it just with this like afterglow of like, Wow, it's. It's almost like a meditation retreat or something, cause you're sitting there in a a pretty vulnerable place. And and, uh, I've grown a lot because how of, doing has that shaped this.

you as a leader?

Well, I mean, I think the a big one was really prior. I mean, I've always, yeah, I don't know well prior to T group. I think in my mind, feelings were not important, especially in a work context, feelings got in the way. They were, uh, a nuisance, and they would prevent me from being the like ice cold, logical executor that I needed to be. And going to T group made me realize, like, oh, wait, like. These feelings are like, critical to to how I relate to others in the company, to how I can lead, and to how I'm able to know.

Like, where my boundaries are, what? Um, what's like good for me? In any given, you know, often very stressful and, uh, it would put it in the icky feeling bucket. Um, that, you know, we. I don't let myself feel that, and then, you know, that's a recipe for stuckness.

avoidance? And yeah, When you start, you're pretty much writing code 100 of your time. And over time, it like the percentage of time of what you spend it on. Between like talking to customers or internally, or do you have a rough breakdown of today? I know it's gonna change, but today percentage of your time spent in different areas.

uh, well, it, it really fluctuates, but I guess my my like, ideal um time partitioning would be like 50 of my time. I'm I'm writing code in some capacity, whether that is is code for like a new product feature or to squash a bug, or it's it's code to actually use the product. It's like doing some modeling or playing with, you know, some of the new LLM stuff. I've heard people are pretty excited about that. Um,

I hear it's, yeah.

yeah, hm, hm, uh. And then 25 of my time will be, uh, like, customer related things like, I just. I had a call with a customer this morning. Um, if I can help, whether it be something like more technical and how things work in the infrastructure space, or just, as you know, by bringing like a founder's perspective to a, um, a strategic conversation. I, I love that and try to do that as much as possible. Uh, and you know, talks like, Um, there's a talk at Ray Summit next week, and then, um, gonna be going to DockerCon doing more of those.

It's always fun. And then the, you know, the last 25 would just be, you know, the day to day, like work that needs to be done, right? It's like more meetings. The thing I miss about the very like in the very beginning, there's just no meetings, there's zero, right? That's the thing I miss. The most is same, that, like, no meetings. Yeah, uh, but now, you know, at least like 25 of my time is just meetings.

Um, you know, y, and I'm, you you're lucky, know, trying to bring my best self to those meetings. But

you're lucky. It's 25, some of yeah, I know us. Yeah,

some are very different than

So here we're gonna change the pace, I'm gonna do a lightning round. Lavanya's gonna jump into. So this is the 1st thing that comes to your head. Don't overthink it and just go 1st. One. Three adjectives that others would describe you as, and you can't say curious, cause I've heard curious, so three others besides curious adjectives that people would describe you as. Go

oh boy, uh, can't use curious. Nope, dang nope, it's I'm. I'm trying to find a synonym now for curiosity. can't use okay, you can use tech Jesus Nope, curious. either. yeah, no, uh, You can use okay, uh, nerdy, um, weird, yeah, actually, uh, Lavanya said that, nice, good, I'm. I love weird. weird is oh, in man, a value a good way, of mine. in a good Yeah, yeah, way. good, weird, good weird. yeah, good, weird, good, not bad, weird.

weird, Yeah,

Chris's introduction to the company is called Weird Stuff with CVp. That's his 101. And I asked

other people this, and so far, curious and weird both came up, so I'm I'm curious if you have a 3rd

uh, maybe silly.

silly. I heard humble a bunch from other people. We we like, asked Oh, you the other people, can't, you can't, I know you you can't say can't humble, brag. you're humble. That's I know, like, totally, I know that is a I'm so total humble. okay. So inside voice when you're maybe even now, what's in your inside voice that you like? Tell yourself not to say out loud, don't say it out loud. What is that?

uh. Well, honestly, it's like it's usually very little. I tend to be like an open book. I've got I've gotten in trouble for saying, uh, more than I, more than I should. Um,

That's the weird thing that people call you,

yeah, maybe that's a part of the weird it's cause thing.

your inside voice is your outside voice.

I just wanna. I mean, I I just,

think I am like, a little, Oh, go ahead, Lavanya.

I was just gonna say, Chris is very genuine. It's not like in a bad way. Chris is like, so himself at all times that whatever he thinks comes out, go ahead. Uh, thanks, Lavanya. Um, Big time.

you know, I will say that, um, I am fairly conflict averse. Like, I'll, if there's something that's like, annoying me, I'll just kind of like, grit my teeth and push through it. I'd rather just like, deal with it than confront it, um, but T group's teaching me to actually confront in, you know, the appropriate situations. So,

Uh, you're doing a great job at it. I've seen you improve over the years. Uh, next question What's the 1st thing you ever built?

uh, Ron the bus nut com, That's right, uh. Specifically, this was in an era where flash introductions were a big deal. Like when you'd I remember go to the website and you'd need, like, that yeah, all of the moving stuff and the sound. I made a sick flash animation with like, a bus, and I rem. I used like the Prodigy was my background music, which my dad didn't really like the Prodigy. But too bad, you know, it's my website intro it makes for Ron the Bus Nut sense.

dot com. Yeah, I think you're

next in your next meeting at Weights and Biases. You guys need to, you know, we need a revival of this.

I've looked for it. It's the Wayback Machine has some some early Ron the Bus Nut dot com, but I I haven't found the original flash. I need to do some more digging.

If you could hack with someone dead or alive, who would you hack with and what would you hack on?

Ooh, hack, um, uh, probably. Yukihiro Matsumoto, the like author of Ruby. I

What would?

haven't done, I haven't done like Ruby in a long time. But I love, uh, Japan and Japanese culture, and Ruby's what got me into this. So, like, the thought of sitting down, like writing code with the UM kind of author of such a, uh, impactful tool is exciting. I don't know what we'd, what we'd hack on something sick, we'd figure What it out.

was I gonna say? That doesn't matter, That is Chris? Chris is known for building insane things in a day. You just give him a day and he just goes off. And like, so that's Yeah,

the okay. What's the biggest mistake work wise? Don't need to dive into your personal life too much that you've made in the last two years,

oh, two years.

three years.

Well, okay, I'll, just. This isn't quite three years. This is probably like a little older, but um, the team is like, still griping about it. Uh, when you start a company, you need to, like, choose all the technologies and you know that you're gonna be like, stuck with these mostly. Um, and like, one of the decisions I felt I was forced into. We. We used MySQL as the like data store for for everything because at the time we were on Google Cloud.

And all they had was a managed MySQL offering, they didn't have managed Postgres. Um, so yeah, the the team is still they. They would much rather be, uh, on team Postgres. But alas,

oh, how sweet. Oh, it's good, I also, but you still get crap for that

oh yeah, for sure. Um, and uh. I chose a UI framework, it's called like Semantic UI, I probably this, uh, this is a failure mode of mine. I should've just gone like Material UI or something like a little more mainstream. Semantic UI isn't really being updated anymore. Now. It's like all over the code base, so the team's like creating a new UI framework. But it's this like long, painful process to move things over. Like, once that stuff's in there, it's really hard to rip out.

Um, so yeah,

thanks. Uh, what do? What do you think people get wrong about weights and biases?

uh, well, I think a lot of the market sees us as experiment tracking, like all we do is just track experiments. And, uh, you know, we've grown now. We're a company of 200 people. We have lots of engineers working on, um, lots of different aspects of the the ML Ops pipeline, uh, and especially now with with the advent of of Gen AI and and um, the expansion of our targetable persona base. Like it used to be like, Okay, ML researcher builds model, understands, uh, backprop, and and all of the, you know, underpinnings of deep learning.

Now we have just regular software developers that are able to leverage, um, AI. And still need to care about the same things, like evaluating how well the model is doing and monitoring that in in production settings. Um, yeah, we're, we're working on lots of, uh, really powerful tooling in all of those different aspects. And and uh, I think a lot of times the the market doesn't realize that we can be used beyond just, uh, tracking of experiments.

question. Something that we talked about How do you know if you're in a cave or a tunnel? And a tunnel is a hard time, and a cave is a dead end and a real change needs to be made. I think of a tunnel as like, you need to be patient, it's a hard time. A cave is like, gonna have to figure out something else. It's a mental model that I have for myself, like, when I'm in a tough time, I'm like, caver tunnel, caver tunnel

Yeah, totally. Um, I mean, I immediately like, think visually, and I'm like, Okay, I'm looking for like light, I'm looking for like a pinhole somewhere in the

in the cave.

distance. Even even if it's like, if it's not clear, that's actually a way out. Maybe it's just like a little crack in the cave, and it's it's, you know, um, if I have something I'm clearly marching towards, then I feel like, you know, I'm. I'm more in this, uh, tunnel, situation. But so many times as a programmer, you have this where you're. You think you have this great idea and you go down this like rabbit hole.

And then you get a little sign that's like, this seems like, maybe a cave, you should maybe turn around, but you're like, Oh, but I came this far, I wanna keep going. Uh,

so you're just such a cave, that's right. and yeah, I put all this you learn, time into it.

I've learned like it can actually feel really good to throw away, like days of of cave walking. Um, if you will, yeah,

it's, that's, that's a unique skill, I mean to, to be like, all right, you know, let's go, let's go try another cave, because it's it's hard to know, and it's really. It reminds me of, like, this fallacy of one thing. And when you start a company, and I've heard you talk about this. Like, if you can just do one thing, if you can get your 1st customer, if you can get, uh, one more feature, if you can fix this bug.

Like, as a founder, you're constantly thinking like, this one thing will change everything. I mean, you guys did have OpenAI as your 1st customer, so you could argue that one thing was pretty meaningful, but did you have this one thing idea frame? And how do you think about it now?

uh, yeah. I mean, it was always like, uh, how do we get people to use what we're, what we're making right? So, uh, if there was one thing, it was like, get something into the hands of of someone and and try to help them be successful. And you know, we were fortunate to have a partner in OpenAI that that did use us early on, but that also wasn't easier. Claire like I rem, we went to OpenAI and presented to, um, a lot of the researchers.

I rem. I was doing the demo, which, you know, I give great demos, right? Well, it didn't feel like it like they were all on their computers, just looking down. It felt like so So this is deflating.

a visual that really every founder needs to see. Just because when you're like, we were, giving a speech or a demo, all you're really gonna see is like, yeah, it's rough, you gotta make

it's real rough.

your own energy. So that happened You, you, you come to me.

out of that and you, you come out of those meetings and you just think, like, whew, man, that's rough. And you feel like we missed an opportunity. Like a part of that must've been my fault, like I wasn't engaging enough to to, like, pull them away from their, from their laptops. Um, but you know, it's a lot of of small, like baby steps. And everything we do can potentially help some, you know, future big milestone. And, you know, ultimately, because we had been talking to them and they were aware of us.

When they had a problem, they thought, Hey, maybe Weights and Biases could help us with this problem, and we jumped at the at the opportunity. Um, but yeah,

you know, um, this phase that you're talking about. Sometimes it can last only six months, sometimes it can last 14 years, like with crafts are like, how do you keep going? and like, when do you know it's time to? Like? Karen said, like, is it a cave or like a tunnel? but like, also, even if you know it's a tunnel, but it's a long tunnel, how do you keep going? And I think you're also really good at taking the rest of us and like lifting our spirits and like making sure we're going along and optimistic with you.

well, I mean, you, you find the things that that are working, cause oftentimes, you know, a lot of things aren't working. But, uh, as long as you've got some things that are working, just remind yourself of those. Uh, it's important. Um, you know, e. Even this week, like I was, I was talking to the the sales folks and they were telling me about some conversations they've been having. And it's like, great news, it's like really exciting stuff that's happening.

And sometimes when you're not in a lot of those sales meetings, or you're grinding away on some. Some feature that's now like late and you feel like it has to be out yesterday. And if we don't get it out, it's gonna be really, really bad. You remind, Okay, well, um, these things, these things are working, it's gonna be okay, let's keep pushing the other thing. Uh, one of our early, like seed investors, this is such a cliche, but it's it's true.

It was like, you know, work smarter, not harder. Like, at some point, there's so much stuff that you have to get done, but you're not gonna be able to do it all. So step back and and think, like, well, what really matters? or what's the thing that could matter the most? With the, you know, most reasonable mo amount of effort? Uh, in the in the near term, um, and just, you know, take a take a step back.

And anytime you're starting to feel like really overwhelmed and like down in the dumps, uh, you can take a step back and march towards a a near term goal. Crowdflower, right? That we would have periods of growth and then periods of like, flatlining, or even, um, you know, like losing, uh, revenue during periods in the in the company. And it's at the end of the day, it's like, or we're trying to get this like revenue number to go up.

Um, so, you know, at a long enough timescale, it might look a little sad. But right now I have like, this month, can I make it go up this month? can I make it go up like next month? You know, let's let's just like, stay in the here and now and keep pushing forward.

I know it's cliche, but it's hard to be like, all right, am I? How do you not mistake motion for progress? Cause you can be really busy, but the progress isn't either happening, or you're you're doing this march and it actually isn't the thing that matters anymore. We're all in ML. It's changing so bleeping fast that getting the outside input with, like, the relentless march. How do you check yourself that you're, you know, not just motion, but progress?

And that, you know, how do you know when you need to change?

Yeah, well, I mean, I think Lucas, Uh, has really put good systems in place at at Weights and Biases around, uh, asking those questions. So, um, you know, every two weeks we sit down and say, Okay, what are what are like top two to three goals for these two weeks? And those goals are then, you know, shared with the the company. So, um, you're, you're kind of, you know, claiming, all right, this is what I think is is most important.

Um, and maybe that changes after like a week, and that's that's fine, but at least we're. We're having, like, a you're not. Hopefully you're not getting stuck in this, like, all right now, you're just you're working on, like all the wrong things, or it doesn't matter, or you're not correcting, right? Because we have an actual like process built in here, we roll that up into, you know, quarterly goals. Um, so that hopefully these things can kind of fit into the, um, the bigger picture.

But I, I also think the the other important thing is that your peers right. Hopefully we've made a culture where we can actually, like, share disagreement in a in a way that people can can hear. Like, if if I think some team's working on the wrong thing, we'll. We'll share that and have a discussion and decide if if that that is the case or not. And having, like, a a good leadership team that feels safe enough to to call each other on, you know what?

What the various initiatives are ensures everyone's kind of held accountable. And that, hopefully, we're marching towards, um, goals that are actually gonna make a difference.

you are known as tech Jesus. Will you tell us what where that nickname came

you're known as? from? Tech What? No, okay, I'm

Jesus? not, he's, I'm not Yes, look known, I'm not at it. known

I didn't know that.

as tech Jesus.

Who calls him that? does he? Oh, a lot Did you even of people, know somebody? Really? Okay, I Yeah, I'm not how didn't know that, do you get but I I've heard the brand

people to?

of it. It's the hair Oh my god, and the tech, it's the hair and the tech. yeah, well, it's alright, I don't even

yeah, hair and tech, that's definitely wait, there's a a a part whole story, of it.

tell us the story.

there. Oh, there's a story.

It's not, I mean, okay. At our previous company, we, uh, were working with the Department of Defense, so we had a meeting in the Pentagon. Uh, which is a fascinating place to go to and a little intimidating. But How did you know, we had you feel to get rid of all of

walking in?

our phones. And, uh, I thought, I mean, it was cool, I was Did they do like, I felt a background honored to be check on you there.

before you got the?

Yeah, thank they didn't find anything, thank I'm just God.

saying, like, I think they probably had to, okay, okay, uh, but so you had we had to get rid to, you of your know, phones, you probably had to go through a bunch of screens.

we had, you know, meeting. I'm like, meeting with the generals, and I'm, you know, I look like this. I didn't like really clean up that much, you know? So I think that was a bit jarring. But I guess my charisma, um, and my savior, like, uh, just like aura, you know, had them calling me, uh, tech Jesus also Christopher, my name, it means Christ bearer. Okay, so So I wait, the people maybe I, at the Pentagon yeah, call you Christ Jesus, yeah, the, I mean, I think maybe some of the lackeys, I'm not, I'm not sure the generals did, but yeah,

were they in uniform? Do people in the I've never been in the Pentagon? when you walk into the Pentagon, are people in uniform? or are yeah, it's they a yeah, definitely, really

what?

it's a little, it's intense. Yeah, they're all wearing their hats and everything too. that's Yeah,

I just you don't re. You don't really run into people wearing hats that much unless you're at a baseball game. That otherwise it just doesn't happen.

I know.

It's the tech bros.

Or like a a horse race, like in in Southern California, you have Del

the Derby.

Mar, the big hats, the fancy hats, the Derby. Yeah,

do you do you go to horse races in Southern California? Not as much,

I I mean, I've been but yeah, I to one, I've, I've been to one before. Yeah, it was neat. we gotta go, I'm not, I'm we, we have not a regular. to, like, we gotta Yeah, go. I wore a really nice hat. hm, what's the weirdest hm,

place you've ever done? A customer meeting oh, that's so besides the Pentagon? funny. It's not like ooh, it's sort of like, Where's the weirdest place you've ever done? A customer meeting? I know Like, that would be none of the questions you were expecting.

uh, okay. This wasn't really a customer meeting, but I like the story. We were driving back from, uh, it was Lucas and I driving back from like, a meeting down in Menlo or something. And Luke. He didn't get his driver's license until he was like, 2829, so this maybe is like I said, that, shortly after he got his driver's license. And he, he can make me like a little nervous. But I You like had to take the call.

handwriting, Yeah, he was driving, cause I had to take a call. I was taking a call in the car and and like, talking to this person and li, I like, had to, you know, push my feet down. I was. It was very, very nerve wracking, but uh, the call, I think, went well.

what's the weirdest thing about Lucas that people don't know?

Uh, he is really into, uh, like the evolutionary tree, oh, and prisms, and he's like, really into prisms. magnets and yeah, and magnets

really into magnets. and weird, weird, uh, Is it weird Is he a mad scientist? metals. psychology? No, I'm not into no, he's in church, science, he's into science, it's

the church of science. Yeah, yeah, he's in, like yeah,

the edges of science, you know, okay. So the left if I field.

were to ask Lucas the same question about you, what would he say? We already said, weird is one of your adjectives for yeah. He would yeah, oh, wait,

just say he's weird, like he couldn't describe how he was was weird, weird, but

a weird thing.

weirdness, weirdness. Like, I definitely made Luke feel uncomfortable, um, It's a number of times, and I like, I like that, that makes that's like affirmation You get, that I'm

you get a pleasure. being weird enough, you Okay, know? Yeah,

uh. What relationships in your life, um, friends, co founders, people at WLBe have, like, made you a better human, a better leader, like, you know what has shaped you?

uh, well, I mean, certainly the the biggest relationship that shaped me is my, my life partner, my wife, Snejana. Um,

hm, hm, how has she s uh shaped you?

well, we came from like, very different worldviews. So, you know, I grew up in Iowa, um, and my wife grew up in in Serbia, uh, in eastern Europe, in a, you know, a war torn environment. So it was, you know, a very different, um, experience of things and and very different cultures. Like, just the the culture of of Serbia is like, pretty different than the culture of like Iowa or San Francisco or California. Um, so, you know, through curiosity and and uh, what has been like real heartfelt connection with with a lot of people in that culture?

I've, I've, like, fallen in love with, um, that part of the world. And, uh, yeah, my wife pushes me, calls me on my on my shit. She is not conflict averse like myself. She is like, ready to to, um, confront anything that is that she's feeling, which is definitely, um, something I can learn from.

uh, how have, uh, Lucas and Sean? uh, for those of you who don't know that, those are our co founders. How have they shaped you as a human, and how do they press your buttons?

Uh, yeah, I mean, this is like a relationship. Um, I actually, I mean, Luke and I started our previous company about a year before my wife and I got married. So I've actually been with Lucas longer than, um, m, my wife. Uh, you know, early on we would, we would all go on offsites together, so we'd literally we'd have our own little like field trip and go to a cabin in the mountains and and hack and and um, work together for a week.

And you, you, you're basically your roommates. You start to like, really, um, understand all of the the intricacies of each other and and what you know where buttons are pushed. And um, you also, uh, grow close. Like, I, you know, I love them both. I, uh, we've been through a lot. It's like, you know, I don't like the analogy of like soldiers or something, but it, it. It feels like we're, we're brothers in this. And and uh, we can uniquely understand the hard parts and and hopefully celebrate the the good parts.

And and come out of this, uh, in the in the long term, still close friends.

cause there's when you have a founder, it's like a work marriage, I mean, that's how I think about it. Is there a sense of, um, and you have to also work at the relationship like a work marriage. In my opinion, that it's, you know, it doesn't just happen, um, it takes work and trust. When you think about somebody picking a co founder. Or, you know, I see this now where people are like, Oh, I wanna be an AI and I have to find a co founder.

And it's just like, No, this is, this is chemistry, this is relationship. You don't just like, I any advice to people about that? Because I'm seeing this on repeat and it kind of scares the bejeebies out of me. Cause I just like, it's like, like, what is it, the 90 day fiance or whatever it is? Love at 1st? I don't, I don't haven't watched any of them, but it reminds me of Love is blind, that and I just don't.

Love is blind. Do those work? I don't know.

I mean, every once in a while they do, but I would imagine most of the time they don't. Um, you know, Luke and I, we. We met because we both worked at the same startup, um company called PowerSet. When I 1st moved to San Francisco, uh, and we both lived in the mission district, and um, we would hang out, uh, as as friends, and then also, you know, work together on stuff. And uh, you know, we did that for like, seven, eight months before we decided, Hey, let's go found a a company together.

Like that wasn't, we weren't from the start, you know, thinking about leaving and starting a company. We we became friends, had worked on a problem and, uh, saw an opportunity to to make a company out of that problem and and was ready to take the risk. But I think going the other way around, saying like, Okay, I have this problem now, I wanna like, find someone out of thin air. It's just it's a tall order, like, at the very least, you should be, uh, I mean, it's.

It's very different than just like hiring an engineer, right? When you're when you're hiring an engineer. We've talked about this, we've done this in the past, where it's like, we're not quite sure. All right, let's let's contract with them for, uh, a month and like, see how it, how it works out. This is more like, let's, yeah, let's go rent, um, an Airbnb somewhere and spend a week together trying to solve problems and see, uh, how we feel coming out of that.

And and even that, you know, isn't gonna be as good a signal as as having worked with someone and and been a friend with someone for um, months.

Yeah, so, uh, different question. I've been a longtime listener of Grady and Descent, and I often hear Lucas use Lavanya's name, like Lavanya says. Lavanya doesn't want us to talk about this. Lavanya's yelling at me, Um, so what are things about Lavanya that most people don't know? Oh man, Don't be shy,

uh, well, I think Lavanya doesn't get burnout, as far as I can tell. I've been worried that Lavanya's gonna be like, burnt out for years now. And I tell her to, like, take more breaks and and not work as hard. But apparent she's got some gene or something. We need to go to 23andMe figure out what it is.

Exactly. There's an energy thing. She gets energy from these problems. We'll be talking. She's like, Oh, we're gonna do our 1st event. And the amount of energy she got from, like a stupid deadline and a big idea, was just like, I'm I feel like, if glad you're you, The more impossible excited.

or outrageous the task is, the more like, stoked. Lavanya's gonna be to try to, Cause she um, get it done.

told me and I went, When are you doing this? Like, it's already happened? It was great, it was amazing. But for those of you that went, the deadline was super short. It was comical from an outsider, but not from I think you.

it goes back to like the people you know, like, I feel, uh, like Chris and Lucas and Sean. They're such great leaders and they take the people who are in their team and like, they really like, unblock them. And they're like, you can achieve anything you want and here's the resources, just go nuts, you know, like, and like. That is such a rare opportunity, you don't get that often. Uh, to have someone who just believes in you, so like, I think, you know, that's why I keep going, cause like, they keep believing, and then it, it keeps thinking, yep, we can do it.

Uh, I think what I have learned from Chris is Chris's ability to take on like the gnarliest problems. Like he always picks the hardest problem in the company and he's always working on it. And somehow he just doesn't give up and like, he doesn't get bitter, he doesn't, you know, he just figures it out every single time. And I try to take yes, it's like, insane.

yeah, no, I'm gonna try to take that from you. Cause you said you are conflict averse, but you're not conflict work averse, like big issues seem to be like, yay, so maybe just

Oh, it's the best feeling when you can, like, go up against something that's gnarly and come out like having defeated it. I I mean, that's like a dragon like, uh, it's slayer. super satisfying. Um, I feel like you I remember that do, that like flash animation, I couldn't get that flash animation to work when I was a teenager. And I went to bed going like, how do I get this to work? And I like, woke up with a solution and got it to work.

And I felt like a king, you know? Um,

I'm dying to see it. I wanna know what prodigy song it was, though. Okay, I'm gonna switch gears and ask about LLMS and AI, so AI, what is what industry wide? Are people getting wrong? Because you guys started this company way before this was a thing? And so I'm curious, as what I consider is an. When I look at you, you're sort of pushing a boulder up a hill, saying machine learning's very important, we're gonna give you tools.

And people were sort of like, Uh, huh, um, and now it's a very different mindset about this. But at this very moment in time, what do you think the AI industry is getting wrong?

uh, well, I think everything changed. December last year when Chatgpt was released. Now you you gotta like, think, what, what really happened there? It wasn't like, uh, we didn't have these, these models. It's like the interface and the RLHF tuning to to make it like a real joy to to interact with. Suddenly unlocked people's ability to see, like a lot of people's ability to see what was possible. Um, and now we've got a lot of just application developers, uh, a lot of entrepreneurs.

Like 70 of the YC class is doing some gen AI thing, uh, they're coming and using these things, seeing it just as another API they can hit and use. But it's under the hood, it's this probabilistic modeling problem. Something that, uh, you know, we're very familiar with here at Weights Biases and have been helping teams. Um, to create systematic ways in which they can validate this thing. That is is not gonna be like red and green in CI.

It's gonna be, you know, some, uh, some gradient in between, like you're gonna have some percentage that is okay. And you need a way to actually measure that and then systematically, um, see how that changes over time. And then, as you're changing your prompts or trying one of these new models that came out, uh, you have baselines and you can actually, um, compare it. So I think, uh, probably. One of the more common mistakes is like not putting a standard way to actually measure this stuff in a way that can be shared with folks.

Which is, I mean, that happens to be really good for for weights and biases, because we we build a lot of tools to help do that. Um, but it, I think we're gonna find it being more and more critical as the space gets more, um, complimented or complicated, and there's more iteration.

What do you think the AI tech stack looks like for LLMs, and how is that different from the machine learning tech stack?

Uh, well, I mean, I think now, certainly, at least in prototyping. Like, you're gonna probably use openai, um, to kind of measure, you know, GPT four. Very expensive. Probably not gonna be something you can put into production, but certainly something you can use to kind of evaluate how good whatever other approach you're, you're using. So suddenly now the AI stack involves, like hitting third party APIs, which, um, I think historically didn't happen as much. It starts to look more like traditional web development now.

If you deploy as a server, you need to, like, deal with the same things we've been dealing with as web developers for years. In terms of, um, seeing if there's errors, rate limits, retrying, um, lost connections and whatnot. Uh, I think today still, most of what I'm seeing is is Python, but, uh, JavaScript, TypeScript is, uh, the most commonly understood language. Um, I'd imagine if you look at the the YC group that came out, the majority of them are are using, um, JavaScript or typescript, uh, to to interface with these things.

So that's kind of that's a big leap. Like before, if you were doing data science or ML, it's just Python, you don't, you don't need JavaScript. Um, but now I think we'll have, um, a fair number of folks that are more comfortable in TypeScript or or Javascript land, and there needs to be better tools, um, for them. And then the last area is like, I think there's really cool stuff happening in the embedded, like running LLMS anywhere, whether that's on my laptop or some Raspberry Pi, or, um, on my phone.

With projects like, uh, Llama.cpp and Whisper.cpp, um, you can also, you can, like, run it in the browser using, uh, WebGL. I think we'll start seeing more of this and and more novel ways to make the performance better. Uh, because there's a lot of of really cool um things you can do when when the data is like, not leaving a given environment, uh, especially as the chief information security officer. They're I think that's pretty sweet,

all about the security.

cause then it literally couldn't touch any sensitive data. It's just all like, right there on the edge.

Do you see any problems with taking a model, ha, uh, having it output something and then using it to evaluate its own output?

Well, uh, I mean this, it. It sounds like, you know, when we train a model, we don't test it on anything that we've we've shown it. So I would say the same should probably apply for a model that you're measuring. Like, you can certainly use an a model to measure, you know, you can use an LLM to measure an LLM. But I would, I would think you'd wanna use a different LLM than the one like outputting stuff.

So if I'm using GPT 3.5, I could evaluate it with with GPT four. Um, and I would probably, even if I did that, I would want, like, a human labeling pass, just to see if, um, I'm getting more signal than noise from from whatever problem I'm working on.

When you are advising big enterprises, you know, the type of companies that have, um, traditionally move slow. And and I think that there's a I'll pick a bank, you know, or something something big with all kinds of compliance issues. How would you advise they start? Because I think there's just everybody's doing something and there's a ton of interest. But then I hear from Y, you know, they, they are actually also scared. It's this FOMo of being left behind, fear of moving forward because of data issues.

Uh, yeah, baby steps, like 1st thing, just like, get away for your developers to actually try stuff on. Um, the data that you're, you're concerned about, right? So that means either, uh, they can only run inference in the infrastructure where the sensitive data is. Using, you know, things like Llama or other, um, open source models. Or you just get that enterprise contract with with Microsoft to have, like, a private, um uh, instance of of GPT. And then let the team try.

And then, while they're trying stuff, you can start to navigate how you com, how you stay compliant if this is released to actual users. And and goes, um, beyond that. But what you need is like, strong, um, cases for for how you could actually deploy these things within your organization. And, uh, any blocker to the developer is just, you know, it's deflating. We just wanna make stuff. Um, so when we had Let to jump through a your developer bunch of hoops.

Yeah, free, let them at

least prototype. You, you said, like, let your developer try.

and you said, like, um, f figure out how to use these LMS in your org. Can you help us think big? Like, I still don't think people understand the impact these LLMs can have. Like, you can have a natural interface for anything that your product does, and there's other examples like that. So what, in your mind, is the craziest thing these LLMs can do?

uh, well, I mean, for me, I think like, as a developer, we immediately got the magic of these things because they're like, remarkable at writing code. Um, an example recently, one of our designers, I needed to make a loading icon on one of these pages, and one of our designers, you know, gave me in English, what that loading icon should do. Should rotate to the left ten, zoom out one, change the opacity to point nine, rotate right 30.

I'm reading this and I'm kind of thinking, like, Oh man, I haven't like written CSS in a while. I'm gonna have to, like, look up how to do all this. And then I said, Wait, I could just ask ChatGpt and it perfectly translated that English into CSS3 transforms. It's like, so cool, that's like, it's magical. Um,

I just got goosebumps and I know it can do this. But for some reason, hearing that with like, Oh, what a nightmare and such a gar, you know,

it's so cool, it's so

it's like magic.

cool, and it's such a It's magic yeah, the that like interface of just like, use your words and, uh, it gets It's like, for those you.

a game. of us with kids,

use your words.

uh, huh, um, I think the other, you know, the use cases that I think are especially interesting for enterprise, uh, in terms of, like, internal business intelligence and and automation. Like, you know, really cool stuff around just in English, asking for some, uh, business metrics. And if you've set things up right and and given the LLM access to the schema and where all the data is, it can generate and then run, you know, SQL and actually give you, give you charts or or information about things.

What do you guys think makes this time different than all of the other cycles we've had?

I mean, for me, it was like, Well, why didn't I get it? When I played with GPT two? Like, I thought it was cool, but I also thought, like, all right, well, what can you do with this? Like it's? It felt a bit toyish, um, but to see that, oh, whoa, language turns out great interface, and we can have it write code and we can specify the format of of things. It becomes, you know, a really powerful interface, I think, much like the transformer architecture became a really powerful way to parallelize, uh, bigger and bigger compute.

So I think things are just like all aligning, and um, there's probably a fair bit of of innovation still to come, and yeah, it's exciting times,

Thank you, Chris.

And so do I thank you.

you know, this was a lot of fun, guys, thanks so much.

Thank you for listening to that episode of Grade in the Scent. If you like what you saw, don't forget to subscribe to us and we'll see you in the next episode.