Why Your Security Fails: The Case for Informal Threat Models

Every developer has been there. You pull down a highly recommended cryptographic library, implement AES-GCM or ChaCha20-Poly1305 exactly as the documentation dictates, and assume your application is secure. Then, a vulnerability report lands in your inbox. The issue is not that the cryptographic primitives were broken. The issue is that the context in which you deployed them violated their underlying assumptions.

Security is frequently treated as a pre-release gate, a 50-page document filled out by a specialized team that has never looked at your git repository. To build actually secure software, threat modeling must shift from an academic exercise to an active, informal engineering practice. By adopting a lightweight, graph-based approach during the design phase, developers can expose hidden assumptions and architectural flaws before writing a single line of code.

The Danger of Unexamined Assumptions

Why do secure systems fail? It is rarely because of a raw mathematical exploit. Instead, failures happen because developers make unexamined assumptions about how components interact.

Consider the "Invisible Salamanders" attack on end-to-end encrypted messaging systems. Cryptographic authenticated encryption with associated data (AEAD) schemes, such as AES-GCM and ChaCha20-Poly1305, are designed under a fundamental assumption: there is exactly one valid key for a given ciphertext.

In an Invisible Salamanders attack, the vulnerability arises when an application uses an AEAD scheme but allows multiple keys to decrypt the same ciphertext to different plaintexts. This is possible because standard AEADs do not guarantee ciphertext commitment, meaning a ciphertext is not cryptographically bound to a single unique key. If a developer assumes that a message can only ever be decrypted by the sender and the receiver using a single shared key, but then builds an abuse-reporting feature where a third-party moderator decrypts the message using a different key, they have violated the cryptographic model. The attacker can exploit this to show the moderator a completely benign message while the recipient saw a malicious one.

The cryptography did not fail, the architectural assumption did. This is why identifying assumptions is the most critical, yet most frequently ignored, part of threat modeling. If your assumptions are wrong, your entire security model is incomplete.

The Seven-Question Pragmatic Framework

Formal frameworks like STRIDE or LINDDUN, championed by organizations like OWASP and the Threat Modeling Manifesto, are excellent reference points. However, forcing a fast-moving engineering team to run a full academic analysis on every minor feature is a recipe for adoption failure.

Instead, developers can use a simplified, seven-question framework to build a practical mental model of their system's security posture:

1. What are we protecting? Identify your assets. This includes tangible data like database credentials and private keys, but also intangibles like system availability and user trust.
2. Who or what wants to harm it? Define your actors. This ranges from automated script kiddies and disgruntled employees to nation-state adversaries or even natural disasters.
3. How might they attack? Map out realistic attack scenarios and failure modes.
4. What will we do to prevent it? Detail your mitigations and defenses.
5. How are the assets connected? Think in graphs, not lists. Assets do not exist in isolation, they are linked by data flows and trust boundaries.
6. What assumptions are we making? Explicitly state what you are taking for granted, especially regarding third-party APIs, network security, and cryptographic guarantees.
7. What threats are we deliberately ignoring? Define your accepted risks. You cannot defend against every conceivable threat, so document what is out of scope.

This framework, heavily inspired by security researcher Soatok's informal guide, shifts the focus from filling out templates to understanding system relationships.

The "Shrink-the-Box" Workflow

To put this into practice, teams should avoid trying to model the entire enterprise at once. Instead, integrate threat modeling into regular design sessions using a visual, iterative workflow.

First, map out the components of the system you are designing. Draw every service, database, and external API. If one component talks to or depends on another, draw a connecting line.

Next, apply the "shrink-the-box" method. Draw a boundary around the entire system, then systematically shrink that boundary to focus on smaller, individual components.

flowchart TD
    subgraph System Boundary
        LB[Load Balancer] --> API[API Gateway]
        subgraph Trust Boundary
            API --> DB[(Database)]
            API --> Cache[(Redis Cache)]
        end
    end
    Internet((Internet)) --> LB

At each level of zoom, analyze the inputs and outputs of the component inside the box. Ask "What can go wrong?" using the STRIDE mnemonic as a mental checklist:

• Spoofing: Can an attacker pretend to be a legitimate user or service?
• Tampering: Can they modify data in transit or at rest?
• Repudiation: Can a user deny performing an action due to poor logging?
• Information Disclosure: Is sensitive data exposed in logs, error messages, or cache files?
• Denial of Service: Can the component be overwhelmed or crashed?
• Elevation of Privilege: Can an unprivileged user execute admin commands?

For example, when data moves from an untrusted public network across a load balancer to an internal API gateway, that transition represents a trust boundary. At this boundary, you must enforce strict input validation, rate limiting, and authentication checks. If you zoom in on the API gateway, you might find it communicates with a Redis cache. Is that communication encrypted? Does the Redis instance require authentication, or does it assume any process on the local network is friendly? By shrinking the box to focus solely on the Redis connection, you expose the assumption that the internal network is secure, an assumption that falls apart the moment an attacker gains a foothold on any other container in your cluster.

As you drill down, look for inappropriate relationships. If your database has a direct connection to an outbound RSS feed, or if your cache layer is accessible from the public internet, you have found an architectural smell. Sever those relationships immediately.

Maintaining a Living Document

A threat model is not a point-in-time snapshot. It must be a living document that evolves alongside your codebase. When designing a new feature, the threat model should be updated as part of the design specification. For example, if you are building a key transparency directory for a federated network, your threat model should be tracked directly in your version control system.

Here is a simple markdown template you can check into your repository alongside your code:

# Threat Model: Key Transparency Directory

## Assumptions
- The directory server's operating system is secure and patched.
- Clients validate the cryptographic proofs returned by the server.

## Assets
- Directory signing key (High Value)
- User public keys (Medium Value)

## Risks and Mitigations
### Risk: Server Administrator presents different keys to different users (Split-view attack)
- **Status**: Prevented by design.
- **Mitigation**: We use a verifiable log structure where all updates are append-only and publicly auditable.

By keeping the threat model in git, it becomes subject to the same peer-review process as your code. When a pull request introduces a new dependency or changes a data flow, the reviewer can easily check if the threat model needs to be updated.

The Bottom Line

You do not need a dedicated security team or a certification in formal risk analysis to start threat modeling. You just need to stop taking your system's environment for granted.

By asking what you are protecting, mapping how data flows, and explicitly documenting your assumptions, you can catch critical design flaws before they are baked into your production environment. Start small, focus on immediate features, and treat your threat model as code.