nvidia-smi: monitor GPU usage, memory and processes live

Read the nvidia-smi summary

nvidia-smi ships with the NVIDIA driver. Run with no options, it prints a snapshot of every NVIDIA GPU on the machine: the top table shows each device with its temperature, power, memory use and Volatile GPU-Util, and the bottom table lists the processes using the GPUs.

nvidia-smi

Some devices and environments don't support every metric. When a value isn't supported on your hardware or container setup, the table shows N/A in its place.

Refresh it every few seconds

A single snapshot won't tell you whether training stalls between batches, so poll the card. -l (or --loop=SEC) repeats the report at an interval in seconds until you press Ctrl+C; a bare -l uses 5 seconds. For sub-second checks, -lms takes milliseconds.

nvidia-smi -l 1
nvidia-smi -lms 500
watch -n 1 nvidia-smi

-l and -lms print a whole new report on each tick, which fills your scrollback fast. To keep the table pinned in one spot, wrap the command in watch, which refreshes every 2 seconds by default (see the Linux watch command guide).

A busy GPU doesn't tell you which epoch you're on, how many steps are left, or whether the loop has hung while a stray kernel keeps utilization at 100%. Your training loop's progress bar does, and with the run under Runpeek, a stalled alert can ping you when the bar and logs stop moving.

Log GPU metrics to a CSV file

The table is hard to parse from a script, so --query-gpu= pulls specific fields by name. Pass a comma-separated list of properties with --format=csv and each interval prints one line per GPU, ready for pandas or a spreadsheet.

nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu \
  --format=csv,noheader,nounits -l 5 >> gpu.csv

noheader drops the column title row and nounits drops the unit suffixes, leaving plain numbers and commas. Run nvidia-smi --help-query-gpu to see every property your driver supports.

Stream compact metrics with dmon and pmon

nvidia-smi dmon prints one compact line per GPU per cycle, for up to 16 devices, without the borders of the summary table:

• -s picks metric groups: p power and temperature, u utilization, c clocks, m frame buffer and BAR1 memory, e ECC and PCIe replay errors, t PCIe throughput.
• -d sets the interval in seconds, and -c stops after a number of samples.
• -o D or -o T prepends the date or time to each line, and --format csv prints CSV.

When several jobs share a machine, nvidia-smi pmon shows which process does the work, in the same one-line layout. Its -d interval must be between 1 and 10 seconds.

nvidia-smi dmon -s u -d 5
nvidia-smi dmon -s pucm -c 10 --format csv
nvidia-smi pmon -d 5

What Volatile GPU-Util actually measures

NVIDIA defines GPU utilization as the percent of time over the past sample period during which one or more kernels was executing on the GPU. Depending on the product, the sample period is between 1 second and 1/6 second.

Because it only checks whether some kernel is running, it says nothing about how many cores are busy: a tiny kernel running back to back reports 100%. If the number stays low or bounces during training, the GPU sits idle for part of each period, often waiting on CPU data loading.

For a richer live view, three open-source tools build on the same data. nvtop 3.3.2 (February 2026) also covers AMD, Intel, Apple and other GPUs, gpustat 1.1.1 (August 2023) prints a compact status, and nvitop 1.7.2 (October 2026) is an interactive NVIDIA process viewer.