11 Sep 2025 · 46m · Daniel Whitenack, Chris Benson · 17 quoted facts
The hosts argue that the widely reported MIT finding that 95% of generative AI pilots fail says little about whether the models work. It shows that companies lack the skill to design full systems around them. Having access to a model, even a strong one, does not solve a business problem without custom data integration, several models and sound software architecture.
The central tension is between executive enthusiasm and the expertise needed to deliver. Companies hold back on hiring junior developers and bet on vibe coding, while the pilots fail. Models are becoming interchangeable, so value moves to integration and services. The hosts read OpenAI's open-weight models and new consulting arm as a sign of that shift.
Daniel says the MIT report found the models could do the tasks. What was missing was knowing how to design tools and workflows that use them well.
Daniel's SharePoint example needs a document-processing model, a language model, embedding models, a re-ranker and safeguard models. Choosing a single model for the proof of concept sets the project up to fail.
A colleague can start a Llama model with tools like vLLM or Ollama, but that gives only one model endpoint. Daniel says this skews how open models are judged and is a reason not to build your own platform.
Chris notes that junior developers have historically become the senior developers who supply architectural judgment. Companies relying on vibe coding and a few seniors are betting that models can supply that judgment, a risk Chris says executives often do not see.
Daniel says GPT, Claude, Llama, DeepSeek or Qwen can all serve a given use case. He argues that data, domain knowledge and integration make the difference, which is why Accenture, Deloitte and McKinsey are earning money while OpenAI loses it.
The hosts cite an MIT report saying most generative AI pilot projects at companies fail before reaching production. Daniel says the report blamed a gap in knowing how to design and integrate AI into business processes, not weak models.
Daniel says general chat tools are generic and don't know a company's data, jargon or processes. Real solutions need custom data integration and automation built around the models.
Daniel says that is the wrong first question, because models change quickly and are largely interchangeable. A real use case needs a set of models working together, so he recommends thinking about the whole system architecture.
Chris believes OpenAI wants a foothold in the premium end of the services market and wants to point to its own open models as support. Daniel sees it as consistent with models becoming commodities, with money shifting to customization and services.
Chris argues it is risky, because juniors grow into the senior engineers who know how to architect and guide AI-generated code. Daniel adds that at scale you still need software and infrastructure expertise to guide vibe coding.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
An MIT report on the state of AI in business was widely reported as finding that 95% of generative AI pilots fail.
“one of the things talked about in the MIT report was that ninety five of AI pilots fail, and, uh, that that I think has generally spooked a lot of business leaders”
According to Daniel Whitenack's reading of articles on the report, the failures were attributed to a learning gap in designing AI tools and workflows, not to model incapability.
“It's not that the AI models were incapable of doing the things that people wanted to prove out in the AI pilots.”
Daniel Whitenack is CEO of Prediction Guard and Chris Benson is a principal AI research engineer at Lockheed Martin.
“I'm Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host Chris Benson, who is a principal AI research engineer at Lockheed Martin.”
Daniel Whitenack says a document-processing workflow may need a document structure model such as Docling, a language model, embedding models, a re-rank model and safeguard models.
“You need a re rank model because you've got to re rank your results after doing retrieval.”
Daniel Whitenack says open models such as Llama can be run internally using tools like vLLM or Ollama, even on a laptop.
“So you can use VLM, or you can use Ollama, or or whatever it is, right? I can spin up one on my laptop”
OpenAI released GPT-5, which Daniel Whitenack says was received poorly by the wider public.
“OpenAI released GPT five, uh, which, um, we haven't talked about a a ton on the show yet, but you know, they released GPT five.”
OpenAI released open-weight reasoning language models, which Daniel Whitenack says was its first open-sourcing of models in about five years.
“OpenAI open sourced some models again for the first time in a very long time, Five years, open sourcing, uh, a a couple of, uh, reasoning models”
Daniel Whitenack says OpenAI opened a consulting arm whose minimum engagement price was around $10 million.
“I think the minimum price for for a consulting services, uh, arrangement with OpenAI was like 10 million or something like that.”
Sam Altman said in January that OpenAI may have been on the wrong side of history regarding open sourcing technologies.
“He noted in, uh, in January that maybe OpenAI had been, and I quote, on the wrong side of history, close quote when it comes to open sourcing technologies.”
Chris Benson says some users preferred the earlier GPT-4o model to GPT-5 and had personified it.
“People were preferring the four o uh model, you know, there was quite quite a bit of um personification of that model”
Chris Benson believes OpenAI released open-weight models so it can compete with rivals that offer open models and support a premium services business.
“I think that's, uh, making sure that with their competitors all having open source models that they can play in the space as well.”
Daniel Whitenack says that in enterprise AI the firms making money have been services organizations such as Accenture, Deloitte and McKinsey, while OpenAI has been losing a lot of money.
“The ones making the money is Accenture, Deloitte, McKenzie, uh, et cetera, the the services organizations, right?”
Daniel Whitenack names Llama, DeepSeek and Qwen as examples of open models, and GPT and Anthropic's models as examples of closed models.
“There's closed models, there's open models, like closed models being the GPTs or Anthropic or et cetera, the open models being the you know, Llama or Deepseek or Quinn or whatever.”
Daniel Whitenack says companies can use general orchestration tools such as Rancher to deploy and manage AI services.
“Everything from the company still managing a lot of the the what they want to deploy and using, you know, orchestration tools, whether that be like a Rancher or, you know, something like that.”
The Midwest AI Summit is scheduled for November 13 in Indianapolis, with Daniel Whitenack helping to organize it and an AI engineering lounge planned.
“we're having this, uh, Midwest AI summit in, uh, November November thirteenth in, uh, Indianapolis, and I'm helping organize that.”
Chris Benson says his mother, in her mid-eighties, was once a computer science professor at Georgia Tech and later worked for Lockheed Martin.
“My mother once upon a time, she's in her mid eighties, and once upon a time was a computer science professor at Georgia Tech.”
Chris Benson says companies are holding back on hiring junior developers out of school while attempting vibe coding with limited success.
“companies are holding back on hiring, uh, junior devs out of school.”
The hosts introduce the MIT finding that 95% of AI pilots fail and set it against companies cutting junior developer hiring.
“one of the things talked about in the MIT report was that ninety five of AI pilots fail”
Daniel explains that business processes need custom tools, data integration and automation, not just a chat prompt.
“It's not that the AI models were incapable of doing the things that people wanted to prove out”
Real use cases need a set of models, and models are commoditized, so the architecture matters more than the model.
“what you don't need is a model, what you need is a set of models”
Running one open model internally is not an AI platform, and teams should apply ordinary software and infrastructure practice.
“it's not so much that Frank did a poor job and the deployment is bad”
Chris argues that dropping junior hires risks losing the future senior engineers who supply architectural judgment.
“it seems, uh, to me, a lot less risky to go ahead and continue to bring in some junior level developers”
The hosts read OpenAI's recent moves as a sign that value is moving from models to services.
“they opened kind of a consulting arm of their business and are entering into these services consulting type of engagements”
Chris compares models to ingredients that still need a recipe and a skilled cook, and the hosts note that services firms are making the money.
“The ones making the money is Accenture, Deloitte, McKenzie, uh, et cetera, the the services organizations”
The hosts promote the Midwest AI Summit's engineering lounge and Chris shares his mother's return to AI study.
“we're gonna have like an AI engineering lounge, where you can actually sit down at a table with an AI engineer”
Welcome to the Practical AI podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn X or Bluesky to stay up to date with episode drops, behind the scenes content and AI insights.
You can learn more at practicalai dot fm. Now onto the show.
Welcome to another episode of the Practical AI podcast. In this fully connected episode, it's just Chris and I, no guest. And we'll spend some time digging into some of the things that have been released and talked about in AI news and and trends, and hopefully spend some time helping you level up your machine learning and AI game. I'm Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host Chris Benson, who is a principal AI research engineer at Lockheed Martin.
How you doing, Chris?
Doing great today, Daniel, so much happening in the world of AI. Holy cow,
Yes, yes, lots, lots to catch up on. Um, there have been a number of interesting developments in the news that maybe people have heard about. And yeah, might might be good to just kind of distill down and synthesize a little bit in terms of what they, you know, what they mean, what they signal, how people can think about how certain things are advancing. So, yeah, looking forward to looking forward to this, this discussion. Any things that have been particularly standing out to you that that you've been hearing about, Chris?
sure, I I think the thing that has really been noticeable in recent weeks has been so many people, uh, that are both in the AI world and outside of it. But impacted, like everybody is, are talking about jobs. And you know, we've talked about the impact of AI on jobs many, many times on the show, uh, over time. But people are really, really feeling it at this point. The job market is pretty tight. Uh, I've talked to lots of people, uh, out there looking, uh, you know, whether they're currently employed or whether they're out of school.
And particularly, there's a lot of people in technology coming out of university that are really struggling right now. And, uh, I believe there was a report recently from MIT that highlighted that
Yeah, it's interesting that we spent a good number of years on this podcast, of course, talking about, you know, uh, occasionally talking about some of the wider impacts of this technology, you know, within a certain company or industry. Now this is kind of like a a global across all, all people thing, you know, all sectors. You see things being hit hard, uh, especially in, you know, maybe it's sales and marketing, or kind of junior developer type of cases.
If I, if I remember, right, Chris, the MIT report that you're talking about, which we can, we can link to some of the news articles about the MIT report. I don't think I've actually seen the actual MIT report yet, so so I guess our listeners can can keep that in mind. But one of the things that I've actually heard on multiple calls, and I was at a last week at a AI corporate type event where there were a bunch of corporate leaders and they were certainly talking about this.
That that one of the things talked about in the MIT report was that ninety five of AI pilots fail, and, uh, that that I think has generally spooked a lot of business leaders, you know, investors, uh, lots of different people across industry. Just the level at which these kind of ninety five of AI pilots fail, what? What's your thought on on that?
I I think I think it's a weird juxtaposition right now that we're in. And that that's accurate, that you're having, uh, a tremendous number of gen ai in particular, uh, efforts fail. But at the same time, companies are holding back on hiring, uh, junior devs out of school. And so you have this weird mesh of, uh, people going hardcore on trying vibe coding and things like that, but with very limited success, struggling to get it adopted.
And at the same time, they're they're making bets on the future by not going ahead and bringing in the the junior level developers that that they always have. Which kind of leads to an interesting kind of what if situation in the months ahead. If you're, you know, junior developers eventually historically have turned into senior developers. And right now, companies are betting on those senior developers with these new AI capabilities over the last couple of years to make up for that deficit, hoping to save money.
But if you're failing ninety five of the time, it puts things into an interesting place.
Yeah, one of the things that I I'm just reading one of these, these articles about the the AI pilots. And one of the things that it highlights is that this isn't from the report's perspective. It's not that the AI models were incapable of doing the things that people wanted to prove out in the AI pilots. But that there was a kind of major learning gap in terms of people understanding actually how an AI tool or workflow or process could be designed to operate well.
So I find this very, very prevalent across all of the, you know, conversations that I have, especially in workshops and that sort of thing. Is there? There's kind of this disconnect, and people are used to using these kind of general purpose models, maybe personally. And there's this concept that the way I implement my business process with an AI system is similar to the way I prompt a an AI model or a, you know, chatgpt type interface to summarize an email for me.
And that is always due to create some pain. Number one, because these AI models only sometimes follow instructions, um, but number number two, your business processes are complicated, right? They're complicated. They rely on data that is only in your company and probably has never been seen by an AI model unless you accidentally leaked it. Jargon, all of those sorts of things and number two, you know, these these business processes that people are trying to automate or create a tool around.
Often, the best thing for that is not a general chat interface, right? It's not like you want to create a chat interface for everything you want to do in your business. No, actually, in this, you know, in a one particular case, right? It may be that you wanna drop a document in this SharePoint folder, and when it's dropped in the SharePoint folder. It triggers this process that takes that document and extracts this information and compares it to information in your database.
And then creates an email and sends out an email to the proper people. Or, you know, add something or the. These sorts of processes are not general chat interfaces, so people are coming at it like, Oh, I know how to kind of prompt these models to do some things, and so they try to kind of kind of build or prompt these models in a certain way. Without the kind of I. I don't know why there's such a disconnect, but without the the understanding that really what they need is a a cus- maybe a custom tool or automation, they need data integration, uh, data augmentation to these models.
They don't just need a model plus a prompt. And I think that that's a that's a pitfall that I see, unfortunately, very, very often. So it's not super surprising for me to see this kind of highlighted in the MIT report.
Yeah, I agree, it's, I I think, you know, to your point, these chat interfaces, you know? It's kind of becoming the universal hammer in most people's heads, and everything is starting to look like a nail. Uh, for that hammer to hit. And they are neglecting a toolbox full of tools that give them the right software components, uh, for for getting their workflows put together the way they want. And so, yeah, I I think, I think it's, uh, I think it's a a certain amount of of over expectation.
That is then, um, exacerbated, abated by choosing the wrong software approach, uh, or an incomplete software approach. To try to get the job done, that workflow, that business, uh, workflow done. So, um, that's certainly my sense of it. Uh, I think a lot of these are driven top down, you know, by excited executives that haven't taken the time to really understand, uh, how to optimally use these tools. So,
Yeah, yeah, and another thing that was was kind of interesting in the study is this kind of just prompting of the models generally kind of failed. There's kind of a knowledge gap on how to integrate data and and how to build custom solutions in a way that could succeed in kind of a POC sort of thing. But I, I actually don't kind of agree with the premise of the you know, that this should spook investors, you know, from kind of AI, maybe some AI companies, but AI companies that especially are verticalized in general.
And um, you know, I'm part of an AI I I uh, lead an AI startup, so I may be biased, but our AI startup isn't one of these kind of verticalized application layer things. So I feel like I can maybe speak objectively with respect to those it. It's really these kind of AI companies that are in whatever it is, healthcare, or, you know, the public sector, or education, or finance, or that sort of thing. They are putting in the work, I think at least many of them are, to understand business processes and build robust AI workflows and kind of tools that are fitting certain business use cases and and that sort of thing.
And I think one of the stats in the report were that a lot of the kind of trials of these sorts of tools did actually succeed kind of a majority of the time. And so there's this kind of major gap between on one, I I guess, in summary, what I'm trying to say is there's this major gap between. On one side, you have, uh, this idea that all you need is access to a model, and then on the other side, these kind of pre-built, purpose-built AI systems for particular verticals that understand the business processes.
There's a whole kind of gap in the middle because many companies, especially in the enterprise setting, will have to customize something. I don't think in the end they will be able to use always use a tool off the shelf that works completely for them. That's if you look at any enterprise software, it's always customized, right at at some level, whether it's manufacturing software or ERP software or CRM, whatever, um, like, it's always customized. I think that's where maybe this is highlighting that gap of companies not understanding, you know, the gap between having a model and a verticalized solution for their company.
And that actually does require, you know, significant, you know, understanding of how to architect and build these systems. Which, unfortunately, there's a skills gap and a learning gap in terms of people that actually have that knowledge, right?
yeah, I I think I think you're drawing out a great point there. And I know we've talked a little bit about this in in, uh, previous shows. Where, you know, the model constitutes a component in a larger software architecture, uh, and we know, as you just pointed out, that that the the expertise of those business, uh, workflows being integrated into vertical software stacks. You know where, where it is, it is designed to solve the problems, and not just a a chat box is really important to getting to a good solution that works for your business.
And, uh, and I think that there is, uh, I think this is where one of those challenges that we're seeing in in a lot of folks out there in the business world is kind of forgetting that that core tenet. And leaping straight for the model will run my business from this point forward. Without that, uh, without that supporting infrastructure. So maybe there are some hard lessons to be learned in the days ahead for some companies, but hopefully, hopefully, that will, that will happen.
Well, friends, when you're building and shipping AI products at scale, there's one constant complexity. Yes, you're wrangling models, data pipelines, deployment infrastructure. And then someone says, Eh, let's turn this into a business. Cue the chaos. That's where Shopify steps in, whether you're spinning up a storefront for your AI powered app or launching a brand around the tools you build. Shopify is the commerce platform trusted by millions of businesses and ten of all US. e commerce. From names like Mattel, Gymshark to founders just like you, with literally hundreds of ready to use templates, powerful built in marketing tools.
And AI that writes product descriptions for you headlines, even polishes your product photography. Shopify doesn't just get you selling, it makes you look good doing it, and we love it. We use it here at Changelog. Check us out Merch dot changelog dot com. That's our storefront, and it handles the heavy lifting, too. Payments, inventory, returns, shipping, even global logistics. It's like having an ops team built into your stack to help you sell. So if you're ready to sell, you are ready for Shopify.
Sign up now for your one per month trial and start selling today at Shopify dot com slash practical ai. Again, that is Shopify dot com slash practical ai.
Well, Chris, there's a couple things that, um, I've been following with respect to the the model builders. But it does actually connect, maybe to this MIT report as well, because one of the other the the other kind of common things that I see that companies, or or a way of thinking that companies have when they approach kind of AI transformation is. They come at the problem of AI adoption, kind of with the question of what model are we going to use?
Which I think is the completely wrong question to to be asking for a number of reasons. Um, first of all, y, you know, if you're coming to, you know, AI for the first time with your company. And you want to transform your company with AI and build, you know, knowledge assistants and automations and adopt other tools and build verticalized solutions. The model actually will shift over time a lot, so that's I think number one. There's no one kind of on the market that, at least right now, is.
There's certainly many providers of models, there's a lot of good models. No one knows who will have kind of the edge on models in the future. And I think what we're actually seeing is that the model side is fairly com commoditized. You can get a model from anywhere. The second reason I think that that's the wrong question to be asking is that if you're trying to build an AI solution within your company again, think about that SharePoint thing that I talked about.
I'm gonna process this document from SharePoint and extract these things and send it to an email. You actually don't need a model, you need a set of models, and potentially, you know, other things in the periphery around those models. So you likely need a document, you know, structure processing model, like a Docling, you need a language model to process pieces of that. You maybe need embedding models to embed some of that text or do retrieval. You need a re rank model because you've got to re rank your results after doing retrieval.
You need safeguard models because you want to be responsible and check your inputs and your outputs of your model. So once you start adding these things up, even for that simple use case of processing this document, you know, through SharePoint and out the other end. If you're coming to one of these proof of concept, you know, projects like we've been talking about. And you're thinking, what is the model I'm going to use for this? And you decide, Okay, the model I'm gonna use for this is a GPT model or a Llama model or whatever.
Well, you're already setting yourself up for failure, cause what you don't need is a model, what you need is a set of models. You sort of need an AI platform, you need an AI system that gives you access to multiple kind of different types of functionality, right? And so I I think that that kind of perspective plays into this kind of POC's failing thing. And I I have more thoughts about that. Re related to the model builders, but do do you think I'm off in that?
Oh, how would you correct me?
no, I, I have no correction. We've talked about this as well a bunch of times and I I keep waiting for, you know, I? I I think we've had a lot of really in depth conversations with people on the show in the past about the need for multiple models to tackle different, uh, concerns within your larger business, uh, focus and the software architecture that supports that. And I think as you look forward, and you know, we're seeing agentic AI, we're seeing, uh, physical AI developing, uh, more and more.
And all of those require a number of different models to do that. You know, uh, uh, kind of obviously, you know, very different, distinct things that that you can see in those spaces. And so there seems to be this hang up, uh, in the in the public about the model, you know, and w- which model? And then I pick the model. And as you pointed out, that's completely the wrong thing. It's it's what does your model architecture look like as related to your business workflows?
Um, and and the, you know, the job that you need doing, and how do you do that? And as you were describing, uh, some of those potential models that one might have in a workflow earlier. As I was listening to your examples, I was thinking, Wow, sounds a lot like software architecture, you know, just it's just each. Each component is invested with with one or more models now in that component, um, but there are still many components that make up a full business workflow.
And so, um, I guess maybe because we talk about it fairly regularly, uh, on the show here. It seems quite obvious to me that that's the case, but clearly it's not if you look at the business decisions that are being made out there. Um, some of which is is there is clearly, uh, a need at this point. You may have a a senior, uh, developer type, by whatever title you're applying, uh, working and kind of sort of, uh, knowing software dev at some level and sort of vibe coding and putting their knowledge of architecture together for a solution.
But if you're not gonna bring in junior devs, then you're making a gamble that you're not gonna need that at some point. And yet what we're seeing is a is, you know, per that report, a ninety five failure rate on using these tools. Uh, in at the current, at the current point in time we're at now on that, so it's there's some it, you know, we've we. We had a recent episode on risk management and from a risk management, uh, perspective.
I think that there are a lot of very risky decisions being made by executives, largely in ignorance. I think of kind of understanding how models and software architecture fit together. Uh, to your point, so no, I'm, I'm in violent agreement with what you were saying a moment ago.
Yeah, this idea also that that you kind of bring in an a model also produces a little bit of problematic behavior around a around adoption of models, uh, particularly in private kind of secure environments. And I I know this one from experience, where you kind of think, well, which model am I going to use? And then you think, well, there's a couple categories of models, right? There's closed models, there's open models, like closed models being the GPTs or Anthropic or et cetera, the open models being the you know, Llama or Deepseek or Quinn or whatever.
And you know, you have smart people in your company and who, whoever you know. Frank over in infrastructure is like, Yeah, I can spin up a Llama model in our infrastructure, right? And there's in innumerable ways to do that at this point. So you can use VLM, or you can use Ollama, or or whatever it is, right? I can spin up one on my laptop, and so you spin up the model and then you're like, all right.
Well, let's build our POC against, you know, Frank's model that he's deployed internally. Cause that's, you know, we now know how to do that. But again, it's not so much that Frank did a poor job and the deployment is bad or the tools are bad, like VLM or something is very powerful, but it's just a. It's not a proper comparison because what you've deployed is a single model, not a set of AI functionalities to build, you know, rich AI applications.
You now have a private model, which again only does what a private model does, that one particular model, it doesn't give you that rich set of AI functionalities. And so it's not really a knock against open models. What it is an indication of is that you maybe shouldn't roll your own AI platform, right? So there's a lot of things that, uh, that that go into that, and you know, uh, there's various ways to approach it. But I think that misunderstanding of what model do I need also impacts the perception of these open source models.
Because most of the time when you deploy that open source model, you're only getting a single model endpoint versus kind of a productized AI platform.
Yeah, I I guess, ha, you know, I know that in your own business, that, you know, you do help bridge some of that gap there and some of the things that you guys do. But in general, if you're talking in the broader market, you know how? How do people they can? They have service providers, uh, out there that can give them some of those services, um, but as they are going and deploying, and, you know, in a sense, we've always encouraged, uh, open source and open weight models out there.
Uh, we've talked about that a lot, um, and we like to see that. And yet there is this skill gap or understanding gap that you've just d defined, uh, in the business community of. Yes, you have these capabilities, but you've got to connect all of your, uh, resources, you know, broad. Using the term in a very generic way, all of your resources together to give you the capabilities you need for your business to operate the way you envision.
And, you know, that's definite falling down in understanding, uh, you know, within that gap space, you know, what are some of the different options, how people can get through that gap?
Well, I think one of the things that can be done is to approach this sort of problem from a a software and infrastructure and, uh, architecture standpoint to what you were saying before. A lot of what we're talking about really kind of falls in that architecting side of things. And so I think from the beginning, the question is not again, if you come to it with the question not being what model are we going to use and who's going to deploy it internally, right?
But you come to it from the standpoint of we will be using models, there will be many models, they will be connecting to many different software applications. Okay, well, that changes the game a little bit in terms of managing that and making it robust over time. And there's very many capable engineering organizations that know how to scale, you know, bunches of services and keep them up and set up uptime monitoring around them, and, you know, alerting and centralized logging and all of those sorts of things.
But you never get to have those conversations. If if you kind of cut it out before you get there by just saying we will have a model, you know, living, living somewhere. And so you really need to approach it from this kind of distributed system standpoint. And once you start doing that, you start talking to, you know, the experts that are on your team. And there are very many tools, depending on the the standpoint that the company wants to approach this from.
Everything from the company still managing a lot of the the what they want to deploy and using, you know, orchestration tools, whether that be like a Rancher or, you know, something like that. That's generic and not AI related, right? But they're used to using it, maybe they're already using it in their organization, and they can orchestrate deployments of various AI things, and then there's AI specific, you know, approaches to to this as well. So I think it is really a perspective thing.
And as soon as you kind of get into that zone of, well, we do need this software architecture, we need this kind of, uh, sRE and DevOps kind of approach to things. Then you really have to ask some of the hard questions, like, can we vibe code our way through this? What, what kind of? What kind of software knowledge do we need to actually support this at scale? And I think what people will find is you, you do, at the minimum, I think, still need that software engineering and infrastructure expertise to do it at scale.
Well, or at least to kind of guide some of the vibe coding type of things that happen, right? So there needs to be an, uh, an informed pilot to help guide some of these things and make sure the ship is going in the in the right direction.
I I think that's, I think that's very, very well put. A And I think, you know, kind of going back to the fact, you know, kind of. Combining this with the hiring decisions that we're seeing out there in the job market and kind of the collapse of the bottom end of the software dev industry. There is a lot of developed expertise over the course of a career. And while all of the discrete points of knowledge may be, you know, captured in various models out there, there's still the necessity of extracting the what you need from those models in the right context and the right order.
Uh, and at least for the time being, in in our, in this kind of gen ai, gen ai dominated world. You have to have somebody who can provide that kind of architectural view, know how to provide the context to get the things you need from your vibe coding. So, and I, you know, I think people are finding small successes under that. Where they say, I want an app that does this thing, and they describe the app in great detail.
And the and the the models, uh, that they're using will turn out kind of an app, uh, it may or may not be, uh, architected the way for sustainability. And you know, there's a whole bunch of issues. It might make a very good prototype. But if you're not going to bring in junior level coders that in the future will be your senior level coders that have this knowledge, then you're kind of betting on today's talent, producing something.
And you're hoping that your model gets the nuance of all of those components and is able to generate its own context without your expertise, which may happen, but it's a big gamble. If you're a company right now, it seems, uh, to me, a lot less risky to go ahead and continue to bring in some junior level developers for the purpose of growing them over time and being able to have that. And, you know, maybe at some point that does change in the future, but I think I think the companies that are doing that today are are taking some some fairly significant business risks that are largely invisible to their executives.
Well, friends, you don't have to be an AI expert to build something great with it. The reality is, AI is here, and for a lot of teams, that brings uncertainty. And our friends at Miro recently surveyed over eight zero knowledge workers. And while seventy six believe AI can improve their role, most, more than half still aren't sure when to use it. That is the exact gap that Miro is filling, and I've been using Miro from mapping out episode ideas to building out an entire new thesis.
It's become one of the things I use to build out a creative engine, and now, with Miro AI built in, it's even faster. We've turned brainstorms into structured plans, screenshots into wireframes, and sticky notes chaos into clarity, all on the same canvas. Now you don't have to master prompts or add one more AI tool to your stack. The work you're already doing is the prompt. It can help your teams get great done with Miro. Check out miro dot com and find out how that is miro dot com m i r o dot com.
Well, Chris, I, I do think that there's some consistent news stories from other sources outside of the MIT report that kind of reinforce some of what we've been talking about. And I also, I think, are just generally interesting as individual data points. And I've seen a number of those as related to OpenAI specifically in terms of if we just look at what kind of has happened with OpenAI in the previous number of weeks. Some interesting things have happened, and I think that they signal some things that are, like, I say, very consistent with what we've been talking about as prompted by this, this MIT report.
Just to highlight a few of those, and then we can dig into to individual ones of them. But one of the things that happened was OpenAI released GPT five, uh, which, um, we haven't talked about a a ton on the show yet, but you know, they released GPT five. Generally, the reception in the wider public has been that people don't like it, um, and sort of it's it's fallen flat a bit. I, I, I guess, would be a way to put it.
So that's kind of thing one at at the same time. OpenAI open sourced some models again for the first time in a very long time, Five years, open sourcing, uh, a a couple of, uh, reasoning models, LLMs that that do this type of reasoning, and they they open sourced those. And also near the same time, I forget the exact dates. Someone, you know, listening can, can maybe provide those in a comment or something. But but the the other thing that happened was that they opened kind of a consulting arm of their business and are entering into these services consulting type of engagements, which are not cheap.
I think the minimum price for for a consulting services, uh, arrangement with OpenAI was like 10 million or something like that. So you've kind of got this thing that's happening, which is a model that's kind of in this area of what has been their moat, kind of these closed models kind of falling flat, them giving giving out some of the the model side, you know, publicly, openly, and then opening the services business side of things. Now I've drawn my own conclusions in terms of what some of those things signal and mean.
But any, any initial reactions or thoughts that you've had as as things have have come out like that, Chris?
I think, you know, Sam Altman is the CEO of OpenAI, and, um, yeah, it's. He's a curious, uh, individual with, uh, you know. He noted in, uh, in January that maybe OpenAI had been, and I quote, on the wrong side of history, close quote when it comes to open sourcing technologies. Uh, but it was Sam who made those decisions. And I think what he's seeing now is the market is evolving, it's maturing. As as you would expect, you know, kind of the the early phase of focusing on these founda, you know, these kind of frontier foundation models that were driving the hype for the last few years and might be, uh, producing diminishing returns.
And even though the GPT five model is more capable, and that's what I, that's what I use for most of my stuff. Um, maybe some of the uh nuances, for instance, the the the interface itself, the way it works, the way the model's working. You know, people were preferring the four o uh model, you know, there was quite quite a bit of um personification of that model, I think, going on with the public. And that, I think, I think OpenAI realized that, you know, there are these concerns.
In addition to the fact that they had kind of left the services market to others. And so, you know, I, my, my belief is that the uh, is that they are starting to open, uh, source some of these models with their, you know, open weights. For the purpose of supporting a solid footstep and at the, you know, kind of the premier end, the expensive end of the services market. And I think, I mean, I think that's the motivator right there is is is doing that.
I think that's, uh, making sure that with their competitors all having open source models that they can play in the space as well. And they can go in with their services organization and make money on services and and point to their own open source models to be able to, uh, support that services uh, business model that they're doing. So, uh, maybe a little bit of a jaded uh answer from me, potentially, uh, just having, uh, like you watched them over a number of years, month by month, uh.
But yeah, I, I would. I would definitely say they're. They're trying to lean in, recognizing that the the business of AI is both expanding and, uh, and maturing into that area.
Yeah, I I think if we combine this with the knowledge from the MIT report around kind of these use cases and enterprises failing. What we know at this point and what I would kind of distill down as very uh, or or, or trends and uh, and insights that are backed now by various data points. Is that generic AI? M So the the generic AI models? Just getting access to a model does not solve any kind of actual business use cases and problems, and, you know, verticalized uh solutions that are needed.
That's what we kind of learned from the MIT report. And this is. This would be true whether your company has access to ChatGPT or Copilot or whatever, the whatever models you have access to. These are generic tools, these are generic models. It's very hard to translate that into customized business solutions, right? And that is why, you know, the insight one from my perspective is just having access to a generic model or generic tools is not going to solve your business solution.
And that's why, partially why a lot of these POCs are are failing now. OpenAI offers those generic models and tools, right, which is really great on the consumer side. But enterprise wise, the ones making the money, at least so far, have not. It hasn't been open AI, they've been losing a ton of money. The ones making the money is Accenture, Deloitte, McKenzie, uh, et cetera, the the services organizations, right? Because really, how you kind of transform a company with AI and bring these models in and do something is by creating custom data integrations, creating these custom business solutions.
And that's is still really a services related thing, or it's at least a kind of customization related thing. There's data integrations there. So this is totally consistent for me with open sourcing models at the same time that they're creating the services side of the business. Because essentially, from the business or enterprise side, there really is not a moat on the on the model builders front. Um, it doesn't matter, uh, from my perspective at least, and of course, I'm biased.
It doesn't matter if you're using GPT, it doesn't matter if you're using Claude, it doesn't matter if you're using Llama or Deepseek or Quinn, really doesn't matter. Any of those models can do perfectly great for your business use case solution. And I, I, I think that's true. I've seen it time and time again. What makes the difference is your combination of data, your combination of domain knowledge, integration with those models and creating that customized solution. And either you're going to do that internally, or you're going to hire a services organization to do that right.
On the one front, you need software architects and developers, and you know, even if they are using vibe coding tools, you'll you will need that expertise. On the other side, you know, you can pay millions of dollars to one of these, uh, consultants, or to open AI and their services business, et cetera. And again, it's a hard thing because those resources are scarce, right? Which I think is why it is a good time if you're kind of providing that level of services around the AI engineering stuff.
Yeah, I, I, I think, I think you think you've hit the nail on the head, uh, and I'll offer sort of a way of restating it with an analogy. You know, when you you're gonna have friends over and you wanna have a magnificent dinner, uh, at your dinner party. And so you walk into the kitchen and you may have a lot of great things to make stuff with. And some of those might be big, expensive things that are raw materials, things like, you know, I, in our analogy, those things represent models and and other software components.
But there's some skill in putting that meal together and going into the refrigerator and picking the right things out, uh, and going into the pantry and picking the right things out, and putting them together. According to a recipe that is your business objective, and understanding how to, uh, produce that final dinner. Which is maybe a little bit different from the way your neighbor would do it, and maybe a little bit different from the way another friend would do it, to produce that fine meal.
That that you're able to enjoy at the end of the day, and that that meal is a bit unique, because, uh, in our analogy, your business is a bit unique. But it takes the skill. And, you know, we do expect technology to develop and those refrigerators to be smart refrigerators and and other things to help in the kitchen. Um, and that might be represented in our vibe coding, uh, thought, but we might not be all the way there yet.
So if you're kind of buying your ingredients and thinking, well, I don't really need to have great skill in the kitchen because I'm sure that some of this technology that's coming into play will take care of that for me. Maybe eventually, but I don't think we're quite there yet is what we're seeing. And I think that report that we've been talking about has kind of, uh uh, provided some evidence of that fact. And so, yeah, there's still, there's still the need for nuance and complexity to be addressed, uh, and the recognition that these commoditized models, whether they be closed source or open source, either way.
Um, uh, it's gonna take more than one, and they're gonna be, uh, they're. You're gonna need to have the recipe to make it all come together the way you're envisioning. So, uh, lot of good lessons, um, for hopefully, f uh, some of the uh managers and executives in these companies making some of these decisions, uh, to do that might help them out going forward.
Yeah, yeah, I love, I love that analogy, and you know, it, it fits so well. Because you can develop that, you know, cooking expertise internally, you can, uh, or you can hire in, uh, a professional chef, um, into your house. Um, it's gonna be expensive, right? but but you can do that, but it, it is a necessary component. So I love that, I love that analogy. I do wanna highlight, um, we always try to highlight some some learning opportunities for folks as they're coming out of this.
Maybe you're motivated to not let your AI POC fail, and you want to kind of understand what it takes to to build these solutions. There's a a couple of things I wanna highlight. One is I'm really excited about, uh, we're having this, uh, Midwest AI summit in, uh, November November thirteenth in, uh, Indianapolis, and I'm helping organize that. Um, it's gonna be a really great time. One of the unique features about this event, different from other conferences, is we're gonna have like an AI engineering lounge, where you can actually sit down at a table with an AI engineer.
Maybe you don't have that expertise in house, but you don't want your POC to fail. You can actually sit down with an AI engineer and talk through that and maybe get some some guidance. So I haven't seen that at at another event. I'm pretty excited that that we're doing that. And you can always, as I mentioned in previous episodes, go to practicalai dot fm slash webinars. There's some webinars there as well that might be good, good learning opportunities for you.
That's, uh, awesome, and, uh, on on the tail end, as we close out of learning opportunities, I just wanted to share one two second thing here. My mother once upon a time, she's in her mid eighties, and once upon a time was a computer science professor at Georgia Tech. She, uh, also happened to work for the same company I work for, Lockheed Martin, years ago. But she had retired and kind of moved out of the technology space.
And, uh, uh, but she is and and very aware of what I do in the space and our podcast and stuff. But she at her in her mid eighties reached out to me this weekend and said, I'm thinking about going back to school for AI and maybe even into a PhD program or something like that, I don't know. And we talked about it for a while, and starting small, uh, she's into some Coursera courses now, and, uh, I just as we're thinking about learning and ramping up, I just wanna.
You know, we've talked about learning recently on the show, uh, you know, we had a couple of episodes where we talked about kind of it's never too late. We've had some. We had a congressman who was not a spring chicken not too long back. Uh, diving in incredibly inspirational. And I wanna say, if my mom in her mid eighties and decades out of the computer science space is willing to dive in and do technical work, uh, on Coursera courses.
I would encourage all of you to reconsider. You're never too old. And I just wanted to leave that as we talked about learning items to say, uh, go get it. The world's changing fast, and, uh, my mom in her mid eighties doesn't wanna get left behind and wants to be on top of it. And I think, uh, I think it's a good thing for all of us to take some inspiration from and go do.
That's awesome, appreciate that, that perspective, Chris. It's been a been a fun conversation. Thanks for hopping on.
All right, that's our show for this week. If you haven't checked out our website, head to practicalai dot fm and be sure to connect with us on LinkedIn X or Bluesky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner prediction guard for providing operational support for the show. Check them out at predictionguard dot com. Also, thanks to breakmaster cylinder for the beats and to you for listening.
That's all for now, but you'll hear from us again next week.