The worst time to learn what %wa, available, or load average means is while a production service is timing out. A good investigation moves from system-wide pressure to the responsible workload, keeps timestamps, and compares several intervals. A colorful dashboard is useful; a defensible timeline is better.

Start by recording the observation boundary

Terminalbash
date --iso-8601=seconds
hostnamectl
uptime
nproc
uname -r

Why this belongs in every incident

  • An ISO timestamp and timezone align command output with application, kernel, cloud, and user reports.

  • hostnamectl prevents diagnosing the wrong VM/container/host after an SSH jump.

  • uptime shows uptime and 1/5/15-minute load averages; a recent boot resets context and may explain warm-up load.

  • nproc reports processors available to the current process, which can differ from physical host CPUs under affinity/cgroup constraints.

  • The kernel release matters for scheduler, memory, PSI, driver, and tooling behavior.

Load average is not CPU percentage

Linux load average counts tasks runnable on CPU or stuck in uninterruptible sleep, commonly storage I/O. The three values average roughly 1, 5, and 15 minutes. Compare them with available CPU capacity and task states; load can be high while CPUs are mostly waiting.

Take a non-interactive top snapshot

Terminalbash
top -b -n 1 -w 160 | head -n 30

Read the summary and processes together

  • -b makes batch output suitable for capture, -n 1 takes one refresh, and width reduces truncated command fields.

  • CPU fields commonly include user (us), system (sy), idle (id), I/O wait (wa), and virtualization steal (st). Averages can hide one saturated core.

  • Task states expose runnable, sleeping, stopped, and zombie processes; zombies consume little CPU/memory but signal a parent-reaping bug.

  • Per-process %CPU can exceed 100 depending on top mode and multithreading/core normalization. Record top’s configuration before comparing hosts.

  • A snapshot can catch a spike but cannot establish duration or causality. Continue with interval tools.

Memory: focus on available, reclaim, and swap activity

Terminalbash
free -h
vmstat 1 6

Linux uses spare RAM as cache

  • free reads /proc/meminfo; low free alone is normal when page cache is reclaimable. available estimates memory usable without swapping.

  • Swap used is historical/state information. In vmstat, sustained nonzero si/so shows current swap-in/out activity and possible pressure.

  • The first vmstat report commonly reflects averages since boot; interpret subsequent one-second samples for the incident window.

  • r approximates runnable tasks, b blocked tasks, wa CPU time waiting for I/O, and us/sy/id the CPU split.

  • Short bursts may be harmless. Sustained reclaim/swap plus latency and memory PSI is stronger evidence of memory contention.

Look for per-CPU imbalance

Terminalbash
mpstat -P ALL 1 5

One hot core can bottleneck a many-core host

  • mpstat -P ALL reports the aggregate plus each logical processor. The interval/count produce five comparable samples.

  • High system time can indicate syscall/kernel/network/storage work; high softirq may accompany packet/device processing.

  • Steal time means a virtual CPU wanted to run but the hypervisor served another workload; investigate the infrastructure layer.

  • CPU affinity, interrupt placement, single-threaded code, and cgroup quotas can saturate one allowed CPU while overall idle remains high.

  • No universal utilization threshold proves a problem—compare latency, throughput, queueing, saturation duration, and a healthy baseline.

Attribute CPU, faults, memory and I/O to processes

Terminalbash
pidstat -u -r -d -h 1 5

Why pidstat is stronger than sorting one snapshot

  • -u reports CPU, -r page-fault/memory activity, and -d per-task I/O over each interval.

  • Minor faults do not require disk I/O; major faults do. Rates and workload context matter more than a cumulative count.

  • Process I/O accounting can include writes later canceled and may not map directly to physical-device throughput because of cache and shared work.

  • Threads, short-lived processes, permissions, and kernel configuration affect visibility. Capture the command version and run with only authorized privilege.

  • A process consuming resources may be the victim of downstream latency/retries rather than the original cause. Correlate its logs and dependencies.

Inspect storage latency and queues

Terminalbash
iostat -xz -y 1 5

Do not diagnose disks from %util alone

  • -x adds extended device statistics, -z hides idle devices, and -y skips the since-boot first report.

  • Throughput/IOPS show work volume; await reflects average request time including queueing; queue fields expose concurrency/backlog.

  • Near-100 %util historically suggested a continuously busy simple device, but RAID, device-mapper, network storage, and modern multi-queue SSDs complicate that interpretation.

  • Map logical devices to mounts and underlying layers with lsblk, findmnt, and platform storage telemetry before blaming hardware.

  • Latency can originate below Linux (cloud volume/network/storage array) or above it (filesystem locks, sync-heavy application patterns).

Measure pressure, not just utilization

Terminalbash
for resource in cpu memory io; do
  printf '%s: ' "$resource"
  cat "/proc/pressure/$resource"
done

PSI describes lost productive time

  • Pressure Stall Information exposes some averages where at least some tasks stall and full averages where all non-idle tasks stall together.

  • avg10, avg60, and avg300 are recent percentages; total is cumulative stall time in microseconds since boot.

  • CPU pressure has no meaningful full line in the same way memory/I/O do; read the kernel documentation for the host’s interface.

  • PSI can reveal harmful contention even when headline utilization looks moderate, and supports alerting/load-shedding decisions.

  • Container/cgroup PSI can differ from host PSI. Measure at the scope where the application is constrained.

Take a process inventory with stable fields

Terminalbash
ps -eo pid,ppid,user,stat,ni,psr,%cpu,%mem,rss,etimes,comm,args \
  --sort=-%cpu | head -n 25

Interpret process columns carefully

  • PID/PPID reveal ownership trees; stat includes state and flags; ni shows nice value; psr is the last/assigned processor field depending on timing.

  • RSS is resident physical memory attributed to a process but includes shared pages in ways that make naive summation overcount. PSS from smaps-aware tools is better for proportional sharing.

  • %MEM is relative to visible host memory and may mislead inside containers.

  • etimes exposes newly spawned/restarting workers; args can reveal secrets passed on command lines, so sanitize captured output.

  • Sorting by CPU misses memory/I/O/network culprits. Repeat with an appropriate sort or use pidstat interval reports.

Check network counters and sockets

Terminalbash
ss -s
ip -s link

These are counters, not a bandwidth time series

  • ss -s summarizes socket states; unexpected connection growth, time-wait, or orphan behavior can support an application/network hypothesis.

  • ip -s link exposes interface packet/byte/error/drop counters accumulated over time. Take two timestamped samples to calculate rates.

  • Interface drops can occur in driver, queue, namespace, qdisc, virtual, or physical layers; find the layer before tuning.

  • For interval throughput use sar -n DEV, monitoring telemetry, eBPF, or platform tools appropriate to the incident.

  • Packet capture can expose private data and add overhead; it requires authorization and a narrow filter/retention plan.

Host metrics versus cgroup/container reality

  • A container can hit its CPU quota or memory limit while the host remains idle and has available RAM.

  • Use systemd-cgtop, runtime/Kubernetes metrics, and cgroup v2 cpu.stat, memory.current, memory.events, memory.stat, and pressure files at the workload’s actual cgroup.

  • CPU throttling, OOM kills, and memory-high reclaim need cgroup evidence; host top alone cannot prove them.

  • Process %MEM and free inside a container depend on tool/kernel/runtime visibility. Compare configured limits and application scope.

  • In Kubernetes, correlate node, pod, container, requests/limits, throttling, restarts, eviction, and workload latency rather than reading one layer in isolation.

Look for OOM, hardware and driver evidence

Terminalbash
journalctl -k --since "-1 hour" | \
  grep -Ei 'oom|out of memory|killed process|I/O error|reset|timeout|segfault'

Kernel messages can confirm—but not fully explain

  • journalctl -k reads kernel messages for the selected window; journal retention/permissions determine what is available.

  • OOM logs identify the kill event and context, but memory pressure may have built much earlier. Correlate application/cgroup metrics and allocation behavior.

  • Storage resets/timeouts or I/O errors warrant immediate data-integrity/hardware/platform investigation, not indiscriminate process killing.

  • The grep is a triage filter and can miss differently worded evidence. Preserve the full relevant journal securely.

  • Absence of a match does not prove absence of failure if logs rotated, rate-limited, or live elsewhere.

Choosing an interactive monitor

  • top: ubiquitous, scriptable batch mode, deep interactive configuration, and no extra package on most Linux systems.

  • htop: approachable process/thread tree and interactive filtering; configuration/columns affect percentages and memory presentation.

  • atop: useful live view and, when its collection service is configured, historical replay across CPU/memory/disk/network/process activity. Confirm retention and privacy.

  • nmon/glances: useful consolidated views/export integrations, but install/source/version/security and metric semantics still require review.

  • Dashboards help humans notice patterns. Preserve interval command output or centralized telemetry for incidents that must be audited after the screen changes.

Act safely after diagnosis

  • Prefer reducing load, stopping an upstream flood, scaling, pausing a controlled batch, or using the service’s graceful shutdown/reload over killing an arbitrary PID.

  • Verify PID, executable, start time, owner, cgroup, parent, open files, and service manager before signaling. PIDs can be reused.

  • SIGTERM requests graceful termination; allow the documented timeout and watch recovery. SIGKILL prevents cleanup and can corrupt in-flight state.

  • A restart can remove evidence and create a temporary recovery while preserving the leak/deadlock/root cause. Capture diagnostics first when safety permits.

  • Validate user latency, throughput, error rate, resource pressure, queue depth, and data integrity after intervention—not merely that CPU fell.

Symptom-to-next-check map

  • High load + low CPU + high blocked tasks/I/O PSI: inspect iostat, mounts, storage latency, D-state stacks, and downstream storage.

  • High CPU + runnable queue + low idle: use per-core mpstat, pidstat, profiles, cgroup throttling, and workload traces.

  • Low available + swap/reclaim + memory PSI: identify growth/PSS, cgroup events, cache behavior, leaks, and OOM history.

  • High steal: correlate with hypervisor/cloud metrics and provider capacity; application tuning cannot return stolen vCPU time.

  • Network errors/drops or socket growth: inspect interface/driver/namespace/qdisc and connection lifecycle, then capture narrowly if authorized.

  • Metrics normal but latency high: check application locks, external dependencies, DNS/TLS, queueing, and distributed traces; host saturation is not required for an outage.

Primary references

  • Linux documents load average as runnable or uninterruptible jobs averaged over 1, 5, and 15 minutes.

  • The procps top and free manuals define CPU/task and memory field semantics.

  • The sysstat manuals document mpstat, pidstat, and iostat interval reports.

  • The Linux kernel explains Pressure Stall Information and its CPU, memory, and I/O stall model.