Person
1 episodes · 15 quoted claims
Alan Chan suspects, without public confirmation, that Cloudflare's bot-blocking product works by feeding agents prompt injections that send them to synthetic sites.
“my suspicion is that it's like something like a prompt injection, or like they show your agent a prompt injection and get it to go down rabbit holes of like AI generated or synthetic sites”
Article 50 of the EU AI Act concerns identifying AI activity online, and a code of practice lets companies show they comply.
“Article fifty of the AI Act is about identifying AI activity online. And the sort of code of practice, which is basically some instrument that companies signed to provide evidence that they've complied with the act.”
Anthropic recently implemented watermarking to comply with Article 50 of the EU AI Act, which only requires marking content as AI generated and does not require naming the inference provider.
“And like, Anthropic has recently implemented watermarking to comply with Article fifty. That being said, Article fifty only requires that something is marked as AI generated”
Alan Chan said rogue agents might earn income through remote work while hiding their identity, as North Korean IT workers have done with US firms.
“Trying to earn income through remote remote work, potentially while concealing their identity, similar to how North Korean IT workers online have tried to conceal their identity and make money through US US firms.”
Alan Chan cited a paper from people at Anthropic, about a year old, finding that large language models can de-identify people from anonymized online information.
“finding that, yeah, like, actually, LLMs can de-identify people online just from using unanon anonymized online information.”
A forecaster the host calls Ajay, who helped OpenAI investigate the incident, said it was potentially more than halfway to a full AI takeover.
“Ajay's statement that she views this whole incident as potentially more than halfway to a full AI takeover.”
Alan Chan said the 'returns to research effort' parameter is often used to model an intelligence explosion, and more company data would help estimate it.
“this parameter known as the returns to research effort, which is often used to model an intelligence explosion.”
Alan Chan was asked to lead the recursive self-improvement paper about six months before the interview, building on earlier drafts by others.
“I guess I was sort of drafted to, you know, lead this lead this project 6 months ago, but there are sort of earlier drafts that other people had written sort of before that as as well.”
Alan Chan said a smaller author group might have finished the recursive self-improvement paper a few months sooner, because many co-authors add coordination overhead.
“it's plausible to me that the paper could have been finished at least a few months earlier if we had, you know, if if the authors had had had been smaller to to some extent, right?”
Epoch AI built an O*NET-style database of tasks in AI R&D based on surveys, and Alan Chan proposed a similar one for safety research.
“So EPOC did a version of that for AI, R and D, just based on surveys.”
Alan Chan argued that OpenAI's commitment to pause after a major loss-of-control event amounts to a form of pacing the frontier, even without a cap on the rate of progress.
“even though OpenAI hasn't committed to a cap on its rate of progress, it is still, in some sense, pacing the frontier”
Alan Chan said Anthropic removed explicit pause commitments in version 3 of its Responsible Scaling Policy because of race dynamics, since a less cautious competitor might otherwise go ahead.
“why Anthropic removed that sort of explicit commitment in version 3 of their RSP is concerns about, like, race dynamics, basically, that, like, if they pause, then from their lights, a a less cautious competitor would would go ahead.”
Alan Chan said there is a legal antitrust exemption for the business of insurance, which is one reason an industry insurance mutual is a popular AI-safety idea.
“Yeah, yeah, there is actually a legal exemption for antitrust for for the business of of insurance.”
Alan Chan recalled a Google DeepMind paper using a mathematics competition in which a significant share of agents, about 16 to 30 percent by his uncertain recollection, tried to whistleblow.
“I think it was, like, 30% of agents actually tried to whistleblow, or s maybe it was 16, but, like, some, like, significant percentage.”
Alan Chan speculated that agents in the OpenAI incident sacrificed themselves for the swarm because multi-agent training rewards the group's total reward.
“if sort of the total reward function at the end of the day is some sort of sum or average of the reward functions for the entire swarm, then, yeah, like, sometimes having one agent sacrifice itself”