Topic
7 facts from 1 episode, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2023
Josh Albrecht says Imbue's perplexity-based evaluation idea came from a paper asking whether emergent properties of language models are a mirage.
“are emergent properties of language models a mirage? or something like that was the title of the paper.”
2023
Josh Albrecht explains that the apparent sudden emergence of abilities in language models disappears on a log scale, indicating the metrics were poor.
“if you look at their performance on a log scale, it's linear. So what was really happening is just our metric was not very good”
2024
According to Josh Albrecht, CARBS runs hundreds of experiments at different scales, starting small and then scaling up to learn how hyperparameters change.
“it's tried hundreds of experiments at different scales, so it tries at a really small scale and it sees, like, okay, for a really small scale like.”
2024
Josh Albrecht says that if the theoretically right parameterization is used, the learning rate no longer needs retuning as models are scaled up.
“the practical implication is like, you no longer need to tune the learning rate as you scale them up.”
2024
Josh Albrecht says Imbue's evaluation data show most system failures come from under-specified user requests rather than model mistakes.
“most of the time when systems fail, it's actually from like under specification, and not from, oh, the model, like messing it up.”
2024
Josh Albrecht argues that an agent able to write software can perform tasks beyond the actions pre-programmed for it, linking agents, coding and reasoning.
“if it can write software now, it can do some huge set of things, and even things that you never intended or programmed in the first place.”
2024
Josh Albrecht predicts that robust AI reasoning will cause more labour displacement and disruption in the next year or two than people expect.
“Now we're talking about a lot more like labor displacement and disruption than we were before.”
Episodes