Out of Memory Errors: Diagnose and Fix OOM Killer

Out of Memory (OOM) errors occur when your VPS exhausts all available RAM. The Linux kernel's OOM killer terminates processes to free memory, often crashing applications. Root cause is usually a memory leak, misconfigured process, or insufficient RAM. Detect, identify, and fix OOM events to maintain uptime.

OOM Overview

When OOM happens:

• All RAM exhausted (physical + swap)
• OOM killer selects victim process (usually largest)
• Process killed abruptly (SIGKILL, no cleanup)
• Application crashes, requests fail
• Event logged in syslog/dmesg

Detect OOM Killer Events

# Check kernel message buffer (last 50 lines)
dmesg | tail -50

# Look for "Out of memory" or "oom-kill"
dmesg | grep "oom-kill"

# Check syslog for OOM events
grep "Out of memory" /var/log/syslog

# Watch real-time for OOM
tail -f /var/log/syslog | grep -i memory

OOM killer output example:

[1234567.890123] Out of memory: Kill process nginx (123) score 456 or sacrifice child
[1234567.890456] Killed process 123 (nginx) total-vm:4567890kB

Find Memory-Hungry Process

# Show top memory consumers (RSS column)
ps aux --sort=-%mem | head -20

# Show memory usage in MB
ps aux --sort=-%mem | awk '{printf "%-10s %8.1f%% %s\n", $1, $4, $11}'

# Show processes over 500MB
ps aux | awk '$6 > 500000 {print $2, $6/1024 " MB", $11}'

# Check specific process memory
cat /proc/[PID]/status | grep VmRSS  # Resident memory
cat /proc/[PID]/status | grep VmSize # Virtual memory

Diagnose Memory Leaks

Memory leak pattern (always growing):

# Watch process memory over time
watch -n 5 'ps aux | grep nginx | grep -v grep'

# If memory keeps increasing, it's a leak

Tools to find leaks:**

• valgrind: Detect C/C++ leaks (dev environment)
• Node.js heapdump: Capture heap snapshots
• Python tracemalloc: Track memory allocations

Immediate Fixes

Quick solutions when OOM happens:

# 1. Add swap space (temporary fix)
# Check current swap
free -h

# Create swap file (if needed)
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

# 2. Kill problematic process
kill -9 [PID]

# 3. Restart service
systemctl restart nginx

# 4. Free cache (safe)
echo 3 > /proc/sys/vm/drop_caches

Long-Term Solutions

Kernel Tuning

# In /etc/sysctl.conf, configure OOM behavior
vm.overcommit_memory = 1        # Allow memory overcommit
vm.panic_on_oom = 0             # Don't panic, let killer work
vm.oom_dump_tasks = 1           # Log killed task details

# Apply immediately
sysctl -p

Monitor Memory

# Check memory usage (human readable)
free -h

# Monitor continuously
watch -n 2 free -h

# Get detailed breakdown by process
smem -s rss -r  # Requires smem tool: apt install smemory

Related: Memory management | Performance optimization | Linux monitoring | earlyoom daemon

Related articles