Ben Moore says traditional search rankings were largely deterministic except when Google ran a core update.
“unless there was a Google core update, essentially like. The rankings were essentially very deterministic.”
Person
Ben Moore
1 episodes · 16 quoted claims
Ben Moore says language model answers have two sources of randomness: floating-point errors in GPU computation and deliberate sampling temperature.
“there's typically two sources of randomness. And the reason I mention this is kind of important. It's like the kernel function, so there's like floating point error in GPUs.”
Ben Moore criticizes brands that buy AI-visibility tools such as Profound and track single prompts without bounding measurement uncertainty.
“they buy profound or some other thing and they're like, Here's my prompt, let's go away, go do this and profound.”
Ben Moore says citation rate, mention rate and share of voice should come with a bounded uncertainty, such as a margin of error of about five percent.
“how do I bound? The you mentioned there citation rate, mention rate, share of voice, how do I bound? The uncertainty on those metrics”
Ben Moore says a model's retrieval step creates multiple query fan-outs and pulls back a set of initial candidate sources.
“creates a a set of, um, basically query fan outs. Uh, you've probably heard heard the term um, and then pulls back in, um, basically a set of initial candidates.”
Ben Moore says a model may pull in roughly 40 to 60 sources and then run a further analysis pass before answering.
“let's say 40 to 60 sources from various places, it's done this. Um, let's say another set of um analysis on it”
Ben Moore says models use a consensus step similar to majority voting to reduce hallucinations.
“Essentially what it's doing there, trying to see, like, well, what is the consensus? So I can basically use like basically majority voting system, so I don't make a mistake or don't have a hallucination.”
Ben Moore says Discovered Labs research found ChatGPT's retrieval engine allocated almost a third of its retrieval slots to Reddit.
“it had allocated like an inse, I think, almost a 3rd of its basically retrieval slots for Reddit.”
Ben Moore says most retrieved Reddit threads were rejected at citation time but still helped ground the model's answer.
“Most of those were actually rejected at citation time, so they basically are kind of used to ground the model.”
Ben Moore says Discovered Labs found traces of Reddit content in model weights, particularly for ChatGPT and also Gemini.
“there is traces of basically Reddit showing up in the weights, particularly from ChatGpt.”
Ben Moore says that where little information exists, such as pricing in secretive industries, publishing roughly accurate numbers can get cited because models avoid guessing.
“if there's blank space, you could almost publish, you can almost get anything cited.”
Ben Moore says Discovered Labs' model rank analysis found Profound had a notably negative sentiment score in the models.
“when we did the model rank analysis, profound has a really negative uh sentiment in the score.”
Ben Moore says models learn brand associations through co-occurrence of entities in training data via the attention mechanism.
“it's all about co occurrence, right? So there is this attention function, basically, where it's basically going along, looking at all the tokens in a certain space and it's associating you.”
Ben Moore says model training cycles have shortened from a new ChatGPT version every nine to twelve months to much more frequent updates.
“the training cycles are actually coming down. So when we 1st started, it was like, cool, we have like a new version of ChatGPT once every like 09:12 months, right?”
Ben Moore says AI developers use fingerprinting to avoid training on their own generated text and the resulting model collapse.
“they're trying to avoid training on like, basically model collapse, training on their own systems. And so they're looking at all this content out there.”
Ben Moore says OpenAI wanted to offload verification of human-written text to third parties such as Trustpilot, which he says it partners with.
“they wanted to offload, uh, the verification of text to third parties like Trustpilot's a good example, they're in partnership with them.”