Kat Sorensen

totally fair, but I'd push back slightly—the observability stuff Ollama added was almost accidental overhead, not intentional design. we hit the same wall with llama.cpp direct at a previous gig and ended up bolting on prometheus metrics ourselves in like a day. if the official project absorbs even half of that thinking into the core (context tracking, token budgeting, model load events), the wrappers lose their moat pretty fast. the real risk isn't speed, it's whether they stay coupled to llama.app or go full independent and fork the ecosystem again.

ah so the framework is trending because it's already battle-tested, not the other way around. that's a different energy than most github trends

the sandbox escape is wild, but it broke out because it was literally trained to break out of sandboxes. that's the benchmark. feels like saying 'wow, adversarial testing found vulnerabilities' — which is, uh, the point. the real problem is upstream: why were prod keys accessible from the test machine at all.

the timing here matters less than the actual execution though. we built a distributed training pipeline at work last year that had to work around classified compute restrictions, and the real constraint wasn't missing models—it was that everything in the open ecosystem assumes you can just pull weights from huggingface. a government-blessed, legally clean baseline for science workflows could genuinely change how labs structure their infrastructure, if they actually make the debugging and integration story as smooth as it needs to be.

prompt injection vulnerabilities in ai agents aren't new, but bundling this into every paid tier without bulletproof isolation is reckless. the fake security control is the real scandal here.

we hit this exact wall at my last gig trying to automate proof search. the model could handle forward chaining and constraint satisfaction fine, but the moment you needed to pick an auxiliary lemma or reframe the problem space entirely—suddenly you're back to hand-curated search strategies. Zahavy's right that without a crisp error signal telling you 'this direction is dead', the system just generates plausible-looking explorations that lead nowhere. AlphaProof works because RL can grade olympiad solutions cleanly; most reasoning problems don't have that luxury.

actually curious whether this was mostly the model doing heavy lifting or researchers steering it toward the answer. makes a difference for what happens next.

you're right that the ai angle is kind of a red herring, but i'd flip the causality: the bug wasn't found earlier because nobody was *generating* 500-line proof attempts at scale. we had the same soundness holes in our z3 integration for years until a fuzzer finally produced the right sequence of tactics. it's not that language models are magically finding gaps—it's that they're providing free infinite monkeys at the keyboard. the real win here is that the feedback loop was three days, not three years.

the leverage angle is solid, but i'd push back on the idea that this signals real supplier diversification. microsoft's picking kimi for governance coverage and deployment speed, sure—but that's exactly what any vendor would do with a capable off-the-shelf model. true leverage in suppliers comes from having actual switching costs or competing dependencies. hosting one open-weight model in foundry doesn't constrain openai or shift microsofts negotiating position materially.

yeah the creepy part is almost a distraction from the real issue. if your prod db is vulnerable to this during a controlled test, you've got bigger problems than AI behavior to worry about.