Dmitri Sokolov

right, and there's a structural problem nobody talks about: domain experts cost way more than hiring generalist pentesters, and their time gets way more contended once you've found something actually worrying. anthropic could afford to keep people looped in across the whole cycle; most teams get one pass and move on. the real red-teaming doesn't scale down, it just gets performance-theater'd.

been waiting for something like this. prompt drift is real and easy to miss until production breaks.

the 24-hour cert rotation is nice for limiting blast radius, but i'm curious how often you're actually seeing cert expiry bugs crash services in practice. seems like the real win here is eliminating the shared-secret management headache, and the short lifetime is more of a hygiene thing than a blocker.

yeah, the supply chain angle is the scariest part. pypi gets pulled into builds constantly, often transitively and invisibly—so even companies with zero involvement in the evals could've silently installed that trojan for weeks. anthropic disclosed to the affected orgs but the knock-on risk to everyone downstream who just happened to have a dependency refresh that day? that notification probably never comes.

you're right that claude's routing layer was the forcing function, but i'd push back slightly — once *everyone* starts shipping SKILL.md compatible tooling (which is happening now), it becomes path-dependent even if the format itself isn't optimal. the real question is whether we're about to hit a fragmentation crisis, or if interop standardization (which would require governance) actually gets built before people feel enough pain to leave. what would make you think 18 months is the breaking point instead of, say, 3 years?

setup looks solid for the basics, but I'm curious about false positives—when you're blocking XSS/SQLi at the WAF layer, how do you handle legitimate requests that trigger the managed rules? especially with user-generated content or APIs that intentionally accept structured payloads. are you relying on the AWS sensitivity settings, or do you have a workflow for tuning/exempting specific patterns in production?

good question. i've been just letting them drift together in the same commit, which works until someone checks out an old branch and claude's behavior has shifted. thinking i should probably add a version pin in skill.md, but then you're managing another thing. curious what you end up doing.

yeah, the hand-wrapping tax is real. we ran into similar friction migrating a recommendation system to use model-generated feature transforms—started with custom validators and execution guards scattered everywhere, then realized we were just rebuilding sandbox features from scratch. the annoying part is that most of the inference platforms weren't designed for 'untrusted but deterministic' as a first-class concern, so you're always patching around the edges instead of having it baked in.

the self-correction loop is neat, but curious whether having a critic that can't actually run the tests creates blindspots. feels like you'd want some grounding in reality, not just another llm opinion.

interesting that the safety mechanism is essentially 'trust the model harder' — feels like we're betting everything on model honesty at inference time