Stop the Linux OOM Killer From Wrecking Your PostgreSQL Database

The Linux Out-Of-Memory (OOM) killer is a necessary safety valve for a general-purpose operating system. If a runaway process leaks memory, the kernel steps in, selects a target, and terminates it with a SIGKILL to keep the OS alive. For a stateless web server or a background worker, this is a minor speed bump. The process restarts, reconnects, and continues. For PostgreSQL, however, the OOM killer is a catastrophic event.

To understand why, you have to look at how PostgreSQL manages its processes. Unlike single-process, multi-threaded databases, PostgreSQL uses a multi-process architecture. The main supervisor process, known as the postmaster, forks a separate backend process for every client connection. These backend processes do not operate in isolation. They share large segments of memory containing shared buffers, Write-Ahead Log (WAL) buffers, lock tables, and other critical global state.

The Linux kernel has no awareness of this shared architecture. When the system runs out of physical memory, the OOM killer selects a process to sacrifice based on its internal heuristics, typically targeting the process consuming the most memory. Often, this is an active PostgreSQL backend. If the kernel kills that backend process while it is actively modifying a shared memory segment, the shared memory is left in an inconsistent, potentially corrupted state. Because there are no transactional guarantees at the operating system level for shared memory, a half-written page in the shared buffers could corrupt your data on disk.

The postmaster is designed to protect against this. The moment it detects that one of its child processes has been terminated abruptly, it assumes the worst: shared memory corruption. To prevent corrupt data from being written to disk, the postmaster immediately terminates all other active backend processes. Every client connection is dropped instantly. Every in-flight transaction is aborted. The database then restarts and enters crash recovery, replaying WAL files to restore consistency. If your database was handling a high volume of writes, replaying those logs can take a long time, turning a brief memory spike into a prolonged outage.

The Half-Measure of OOM Score Adjustment

A common, yet incomplete, defense is tweaking the kernel's Out-Of-Memory score adjustment. The kernel calculates an oom_score for each process, combining the percentage of system memory the process uses with an adjustment value, oom_score_adj, which ranges from -1000 to 1000.

Many administrators configure the postmaster with an oom_score_adj of -1000, which tells the kernel never to kill the supervisor process. They then set the backend processes to a neutral score of 0, or let them inherit a higher score. You can configure this in systemd service files using the OOMScoreAdjust directive:

[Service]
OOMScoreAdjust=-1000

While this prevents the postmaster itself from being assassinated, it does nothing to protect the backend processes. If the OOM killer terminates a backend, you still suffer the exact same consequence: the postmaster drops all connections and triggers crash recovery. Protecting the postmaster keeps the database service from needing a manual systemd restart, but it does not prevent the database-wide outage.

The Production Standard: Strict Overcommit

The only reliable way to protect PostgreSQL is to prevent the OOM killer from ever waking up. This requires changing how the Linux kernel handles memory allocation requests. By default, Linux uses an overcommit policy that allows processes to allocate more virtual memory than the system physically possesses. The kernel assumes that processes rarely use all the memory they allocate. When that assumption fails, the OOM killer is invoked.

Linux controls this behavior via the vm.overcommit_memory kernel parameter, which supports three modes:

• Mode 0 (Heuristic): The default. The kernel allows overcommit but uses a basic heuristic to block obviously absurd allocation requests.
• Mode 1 (Always Overcommit): The kernel always approves memory requests, regardless of physical limits, maximizing the risk of OOM kills.
• Mode 2 (Strict Overcommit): The kernel checks if physical memory and swap are actually available before granting a virtual memory allocation. If the request exceeds the available space, the allocation fails immediately.

When you enable strict overcommit (vm.overcommit_memory = 2), the kernel stops guessing. If a PostgreSQL backend requests more memory than the system can provide (for example, during a massive sort or join operation), the malloc() call fails and returns NULL.

PostgreSQL handles this failure gracefully. It aborts the specific query that requested the memory, rolls back that specific transaction, and returns an out-of-memory error to the client. The backend process does not die, the connection remains open, and every other database connection continues running without interruption.

Implementing and Tuning Strict Overcommit

Configuring strict overcommit requires careful calculation. If you simply turn it on without adjusting the limits, you can starve your system of memory and cause valid allocations to fail prematurely. The kernel calculates the maximum allocatable memory using a strict formula:

CommitLimit = Swap + (RAM * vm.overcommit_ratio / 100)

Alternatively, you can specify an absolute value using vm.overcommit_kbytes. If both are set, vm.overcommit_kbytes takes precedence.

To implement this, you must configure two sysctl parameters. Open /etc/sysctl.conf and add the following lines:

vm.overcommit_memory = 2
vm.overcommit_ratio = 80

Apply the changes immediately with:

sudo sysctl -p

Why set vm.overcommit_ratio to 80? If you set it to 100, you leave no headroom for the operating system's page cache, filesystem metadata, or kernel structures, which can lead to system instability. A ratio of 80% is a standard starting point for dedicated database servers, ensuring that 20% of physical RAM remains available for the kernel and caching. If your server has 64 GB of RAM and 8 GB of swap, an 80% ratio limits total virtual memory allocations to 59.2 GB (8 GB + 51.2 GB).

You can monitor your current memory commitment status by reading /proc/meminfo:

grep -E 'CommitLimit|Committed_AS' /proc/meminfo

CommitLimit is the maximum virtual memory that can be allocated, while Committed_AS is the amount of virtual memory currently allocated by all processes. If Committed_AS approaches the CommitLimit, subsequent memory allocation requests will fail.

There is a major caveat: you must have swap space enabled when using strict overcommit. Without swap, your commit limit is restricted entirely by your physical RAM and the overcommit ratio. If you run a database server with 128 GB of RAM and zero swap, an 80% ratio limits your allocations to 102.4 GB, leaving over 25 GB of expensive physical RAM completely unusable for active allocations. Adding even a small swap partition (such as 2 GB to 8 GB) acts as a safety valve, allowing the kernel to safely allocate virtual memory pages that are rarely touched without wasting physical RAM.

Also, remember that PostgreSQL's startup requires a predictable amount of resources. You can inspect the number of semaphores your instance will request before starting it by running:

postgres -D /var/lib/postgresql/data -C num_os_semaphores

Relying on the OOM killer to manage memory pressure is a gamble that eventually ends in database downtime. While adjusting oom_score_adj is a useful secondary defense to protect the postmaster process from manual restarts, it does not address the core vulnerability of backend process termination. By configuring vm.overcommit_memory = 2 and calculating a sensible vm.overcommit_ratio backed by swap, you shift the failure mode from catastrophic, database-wide crashes to predictable, isolated query errors. For any production PostgreSQL deployment, that is the only acceptable trade-off.