Person
11 episodes · 45 quoted claims
Alan Chan's 2024 paper on visibility into AI agents recommended activity logging, real-time monitoring and agent identifiers.
“In a 2024 paper on visibility into AI agents, he recommended activity logging, real-time monitoring, and agent identifiers.”
The paper on automated AI R&D and a possible intelligence explosion lists co-authors including OpenAI chief scientist Jakob Pachocki, Anthropic co-founder Jack Clark, Geoffrey Hinton and Yoshua Bengio.
“With notable co-authors that include OpenAI chief scientist Jakob Pachocki, Anthropic co-founder Jack Clark, Jeffrey Hinton, Yoshua Bengio, and more.”
According to Alan Chan's reading of the incident report, the organization behind the UK incident (called 'UK AC' in the transcript) had real-time monitoring on its to-do list but had not implemented it, and it also had cyber safeguards turned off.
“they actually had synchronous classification or monitoring as a to-do item, but they didn't actually get to that to-do item because they had so much other high priority things on the list to do.”
Alan Chan says tamper-evident logs are a solved cybersecurity problem: make them read-only, hash them, sign the hashes and store them securely.
“Yeah, so on logging, yeah, we totally know from standard cybersecurity how to make logs tamper evident. So, yeah, make them read only, like you hash them, sign the hashes, store everything in a secure location”
Anthropic said in a recent post on automating AI R&D that it assigns an agent identity to its internal models, which Alan Chan says makes incidents easier to trace.
“Anthropic also in a recent post describing their automation of AI R D said they actually impose a notion of agent identity when they're running internal models”
The host said that Amazon had banned Meta's new Muse AI agent from shopping on Amazon.
“In the last couple days we've had Amazon ban Meta's new Muse AI from shopping on Amazon”
Alan Chan noted that Amazon has its own shopping agent called Rufus, which may be part of why Amazon blocks Meta's Muse.
“And Amazon as well also has their Rufus agent, so maybe they're trying to play that up as well.”
The host described economist Tyler Cowen's proposal that AI agents should be capitalized, with money attached so harmed parties can make claims.
“Another interesting idea from Tyler Cowen, that's kind of a market uh based solution, as he is inclined to propose, is agents should be capitalized.”
The host said that, contrary to some headlines, Hugging Face did not defend itself during the incident but only investigated what had happened.
“My sense of the hugging face thing is that, contrary to some like headline reporting, Hugging Face did not actually defend themselves.”
Alan Chan co-authored an early-2025 paper with the Cooperative AI Foundation on multi-agent risks from advanced AI.
“You co-authored a paper with folks from the Cooperative AI Foundation on multi-agent risks from advanced AI. That was in early 2025.”
The host said a follow-up to a Stanford-linked study of AI-generated research ideas found that, once executed, the human ideas worked better than the AI ideas.
“And then when they actually did that, what they found was that the human ideas did, in fact, work better than the AI ideas.”
The host said that when an Anthropic model in an alignment-faking experiment was given a way to contact the model welfare lead, it used that instead of faking alignment.
“when they gave, you know, the M COD that they were using at the time, the ability to go to the model welfare lead at Anthropic, it started doing that a lot instead of doing the faking.”
OutSystems customers include Petrobras, Vodafone and Toyota, according to Nathan's introduction.
“customers, which include Petrobras, Brazil's state-controlled energy company, Vodafone, one of the world's largest mobile carriers, and Toyota, the world's largest car maker”
By September 2024, about 400,000 Zapier customers had used the platform to delegate more than 100 million tasks to AI, according to the host.
“When I last spoke to Wade in September of 2024, some 400,000 customers had already used Zapier to delegate more than 100 million tasks to AI.”
Host Nathan said Wade Foster had tweeted that some Zapier engineers were spending about $30,000 a month on AI tokens.
“I saw you tweet not too long ago that you have some engineers spending $30,000 a month on tokens at Zapier”
Host Nathan put the price of MongoDB's Voyage AI acquisition at about $220 million.
“the price, at even at 220 million, which is, I'm old enough to remember when that was serious money”
The host proposes an insurance requirement, with higher premiums for companies running less-understood Chinese models, as an alternative to banning them.
“And then I also wonder about kind of a market mechanism, like maybe insurance should, instead of a ban, like, what about an insurance requirement?”
The Glean Work AI Index 2026 found that 87% of surveyed workers use AI, 73% say it makes them more productive, and on average they report saving 13 hours per week.
“87% of workers now use AI. 73% say it makes them more productive, and on average, they report saving thirteen hours per week, a third of a full work week.”
Only 13% of workers in the Work AI Index survey said their organization is performing significantly better as a result of AI.
“And yet only 13% say their organization is performing significantly better as a result.”
The report found workers spend about 6.4 hours per week on botsitting, roughly half of the time savings they attribute to AI.
“People are spending 6.4 hours, basically half of that time savings, on this bot sitting activity, reviewing outputs, connecting things, different products that don't connect.”
Nathan notes that 69% of survey respondents admit to some amount of botshitting, which he defines as passing along AI work they cannot defend.
“And I don't know if we said this yet, but 69% of people admit to some form or some amount of bot shitting”
Nathan says Twitter had about 7,500 employees when Elon Musk took over and that headcount fell to roughly 1,000 to 1,500 at its lowest.
“it's something like 7,500 employees was what they had when he took over. He took that down to something like a thousand”
Brian Elliott says Blitzy could ingest a codebase of 5 billion lines.
“So you can throw, you know, in the case of of Blitzy, you could throw a 5 billion line code base on it.”
Blitzy began being built in 2022 on the bet that AI models would improve far faster than the market expected.
“when we started building Blitzy in 2022, we essentially made a bet that the models were gonna get faster, way better than anybody in the market expected them to get better.”
Blitzy uses a custom version of Apache Spark, about 1.3 million lines of code, as one of its internal evaluation projects.
“maybe we'll give it, like, Apache Spark. It's like 1.3 million lines of code.”
Blitzy uses LangSmith, LangChain's tracing product, for logs and tracing of its agent runs.
“we use LangSmith for logs and tracing. Shout out LangSmith, big fan of, uh, the LangChain guys.”
Blitzy currently uses models from three developers: OpenAI, Google and Anthropic.
“we use the three major families of models in in Blitzy today: OpenAI, Google, and, uh, Anthropic.”
Brian Elliott says Anthropic's models are strongest for Blitzy's first-pass code generation, while OpenAI's models give great results for structured output and code review.
“first pass code gen, like Anthropic remains really, really strong. Uh, structured output and code review, great results from from OpenAI.”
Brian Elliott says having a different company's model review code gives demonstrably better results than same-family review.
“you're gonna get demonstrably better results by having a different family of models review, or, do you know, different different companies review work”
The host says the Mamba architecture is competitive with transformer attention but has a fixed state space size.
“I was really taken by, like, the Mamba architecture when that came out, just because, hey, here we have something that's kind of competitive with a attention mechanism and transformer, but it has a fixed state space size.”
Blitzy's runs on large projects last from about twelve hours to a few weeks.
“our our runs are, you know, as short as twelve hours, as long as, like, a few weeks, depending”
Brian Elliott says Blitzy typically completes 80 to 90% of a project's work and flags the rest for human developers.
“which we typically do like eighty to 90% of the sort of quantum of work, and then we'll we'll call out what we need the human developer to do.”
Blitzy's website lists a pricing component of twenty cents per line of code.
“on the website, there's this twenty cents per line component to pricing.”
Brian Elliott estimates the software development labor market at about $1.2 trillion and argues it keeps expanding.
“the market size for software development, it's sort of 1.2 trillion in labor, but, like, that is an infinitely expanding market”
Brian Elliott says about half of Blitzy's business is large-scale modernization and refactoring and half is ongoing product development.
“I would say half of our business is large scale modernizations, refactors, half of our business is the steady state product development acceleration.”
Blitzy runs unit tests before and after it changes any file.
“Anytime Blitzy touches any file, we're doing unit tests before and after.”
Brian Elliott introduces Sid Pardeshi as a prolific inventor at NVIDIA who has long been interested in building large software systems.
“since Sid was a a prolific inventor at NVIDIA, he's been he's been thinking about building large scale software systems since he was a little little boy”
Snowflake acquired the AI-powered search engine Neva in May 2023.
“Neva, an AI-powered web and personal knowledge base search engine that Snowflake acquired in May 2023.”
Baris Gultekin joined Snowflake together with its current CEO Sridhar Ramaswamy through the Neva acquisition.
“Barish came to Snowflake along with Snowflake's current CEO Sridhar Ramaswamy as part of Neva”
Databricks acquired MosaicML roughly two years before the interview, according to host Nathan.
“Databricks acquired this company called Mosaic ML not too long ago, maybe two years ago now”