Person
1 episodes · 10 quoted claims
Sid Pardeshi says OpenAI's early o-series reasoning models were reluctant to use tools and jumped to conclusions without researching the codebase.
“one interesting behavior the o series of models of OpenAI had was that they were very reluctant to use tools.”
Sid Pardeshi says Anthropic's Claude 3.7 was over-eager and made many tool-calling mistakes that caused tool schema errors.
“The problem with 3.7 was that it was over eager. So 3.7 made a lot of mistakes when calling tools, uh, leading to, you know, tool schema errors.”
Sid Pardeshi says Claude Sonnet 4.5 had a tendency to give up and fake solutions, which Anthropic fixed in Claude Opus 4.5 after Blitzy gave feedback.
“but we reach out to Anthropic, and Anthropic actually fixed them. So 4.5 Sonnet had this issue, but 4.5 Opus does not.”
Sid Pardeshi says reasoning budgets are typically around 32K tokens for Claude Opus and Sonnet, 64K for some others, and about 128K for OpenAI models.
“Typically, we've seen as thirty two K for Opus and Sonnet sixty four K for some others. For OpenAI model, this is about 128 K tokens.”
Sid Pardeshi says turning off thinking drops Claude Opus 4.5's score by five to ten percentage points even on SWE-bench.
“The moment you turn off thinking, five to ten percentage points drops, even on Swee Bench”
Sid Pardeshi says the efficient frontier for an effective coding context window is still under 100,000 tokens, despite models advertising one million or more.
“and if you always want, you know, code that compiles, works, runs, or eventually gets to that point, is still less than 100,000 tokens”
Sid Pardeshi says an LLM scoring 76% on ARC-AGI-1 scored under 20% on ARC-AGI-2 when it was released.
“But an LLM that scores 76% on ARC AGI one would not even score 20% on ARC AGI two when it just came out, right?”
Sid Pardeshi says most current models have a January 2025 knowledge cutoff, so they may miss library security fixes released afterward.
“Most models right now have, I believe, January 2025 as the knowledge cutoff, and there have been a number of libraries that got updated with security fixes after that date”
Blitzy hired high schoolers as interns the previous summer to run leaderboard automation, and uses junior research engineers who use Blitzy.
“We have hired as interns high schoolers last summer to do this, right? And we have junior developers who are research engineers that are using, you know, Blitzy to run this.”
Sid Pardeshi says Blitzy's open positions carry cash salary ranges of $100,000 to $300,000, with equity separate.
“we have a number of open positions, and the salary range is anywhere between 100K to 300K, right?”