Runpeek
Get a live link

Runpeek Slurm workload manager

How to check Slurm job status with squeue, sacct and scontrol

Run squeue --me to list your pending (PD), running (R) and completing (CG) jobs; inside the job, Runpeek's tqdml run streams its progress and output to your phone. Finished jobs drop out of squeue, so read them with sacct -j 12345; without -j, sacct only shows jobs since 00:00 today. scontrol show job 12345 prints the full record, and #SBATCH --mail-type=END,FAIL emails you instead of polling.

5 min readLogs and monitoring

At a glance

  • squeue --me lists your pending, running and completing jobs.
  • tqdml run -- python train.py in the batch script streams the job's tqdm bars and output to a Runpeek link.
  • sacct -j 12345 --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode shows a finished job's state, memory and exit code.
  • scontrol show job 12345 and sstat -j 12345.batch inspect a running job.
  • #SBATCH --mail-type=END,FAIL emails you, so you don't need to poll squeue in a loop.

Check pending and running jobs with squeue

squeue on its own prints every active job on the cluster. Pass --me (the same as --user=<your username>) or -u $USER to see only yours. Unless you pick states with -t, squeue shows pending, running and completing jobs.

Pass -j with a job ID to check one job, and add --start to see when Slurm expects your pending jobs to start.

The default columns cut job names at 8 characters and leave out the time limit. Use -l for the built-in long format, or build your own with -o and format codes:

  • %i job ID, %P partition and %j job name (%.20j keeps 20 characters).
  • %T the full state name, %M the time used and %l the time limit.
  • %R the reason a pending job waits (in parentheses), or the nodes of a running job.
Bash
squeue --me
squeue -j 12345
squeue --me --start
squeue --me -o "%.18i %.9P %.20j %.8T %.10M %.9l %R"
Terminal
$ squeue --me -o "%.18i %.9P %.20j %.8T %.10M %.9l %R"
             JOBID PARTITION                 NAME    STATE       TIME TIME_LIMI NODELIST(REASON)
             12346     debug               second  PENDING       0:00 UNLIMITED (Resources)
             12345     debug         train-resnet  RUNNING       0:03     10:00 node01

squeue tells you a job is RUNNING, not how far along it is. The job's own progress bar and output do, and Runpeek streams both to a link you can open from your phone instead of the login node.

Slurm job state codes

squeue shows short codes in the ST column and full names with %T. These are the ones you will see most:

CodeStateMeaning
PDPENDINGWaiting for resources or priority
RRUNNINGHas an allocation and is running
CGCOMPLETINGFinishing; some processes may still be active
CDCOMPLETEDAll processes on all nodes exited with code 0
FFAILEDNon-zero exit code or another failure
TOTIMEOUTReached its time limit
OOMOUT_OF_MEMORYRan out of memory
CACANCELLEDCancelled by you or an administrator
NFNODE_FAILOne or more allocated nodes failed
PRPREEMPTEDStopped by preemption
SSUSPENDEDExecution suspended, CPUs released

For a PD job, the reason in parentheses at the end of the squeue line tells you what holds it back, for example (Resources) or (Priority).

Check finished jobs with sacct and seff

Once a job finishes or fails, it disappears from squeue, so read its history from the accounting records with sacct. Without -j, sacct starts at 00:00:00 of the current day, so a job that failed yesterday won't show up unless you pass --starttime or the job ID.

sacct prints a row for the job plus rows for its steps, such as 12345.batch. -X shows only the job row, but then usage numbers like MaxRSS read zero, so leave it off when you need memory.

ExitCode has two numbers separated by a colon: the exit code of your job script, then the signal that ended the process, if any. A script that ends with exit 3 shows FAILED and 3:0.

If your cluster has it, seff 12345 summarizes a finished job's CPU and memory efficiency. It lives in the Slurm source under contribs/seff, so it isn't installed everywhere.

Bash
sacct -j 12345 --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode
sacct -X --starttime 2026-10-01 --format=JobID,JobName,State,Elapsed,ExitCode
seff 12345

Inspect a running job with scontrol and sstat

scontrol show job 12345 prints the full record the controller holds, including JobState, the reason, the exit code and time limits. Shortly after a job ends, the controller drops the record, and you switch to sacct.

sstat shows live CPU and memory use for the steps of a running job. A plain sbatch script without srun steps only has the batch step, so query 12345.batch; sstat -j 12345 alone printed no rows in a test.

Bash
scontrol show job 12345 | head -6
sstat -j 12345.batch --format=JobID,AveCPU,MaxRSS
Terminal
JobId=12345 JobName=train-resnet
   UserId=user(1000) GroupId=user(1000) MCS_label=N/A
   Priority=1 Nice=0 Account=(null) QOS=(null)
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:33 TimeLimit=00:10:00 TimeMin=N/A

Get notified and follow the output

To get an email when a job changes state, add --mail-type and --mail-user to your batch script. Besides BEGIN, END, FAIL and ALL, --mail-type accepts TIME_LIMIT_50, TIME_LIMIT_80 and TIME_LIMIT_90, which warn you when a job has used that share of its time limit.

Slurm writes the script's output to slurm-%j.out by default, where %j is the job ID, so you can follow it from the login node with tail -F. Don't poll squeue from a loop: every call is a request to the slurmctld controller, and the Slurm docs warn that running client commands in loops can degrade the controller and even cause a denial of service.

Bash
#!/bin/bash
#SBATCH --job-name=train-resnet
#SBATCH --time=04:00:00
#SBATCH --mail-type=END,FAIL,TIME_LIMIT_80
#SBATCH --mail-user=you@example.com

python -u train.py
Bash
tail -F slurm-12345.out

FAQ

Why is my Slurm job stuck in PD?

PD means the job waits for Slurm to allocate resources. Run squeue --me and read the reason in parentheses at the end of the line, such as Resources or Priority, or run squeue --me --start to see the expected start time.

Why does squeue no longer show my job?

By default squeue only lists pending, running and completing jobs. Once a job finishes, fails or is cancelled, it drops out, and sacct -j <jobid> shows its final state. If sacct says accounting storage is disabled, your cluster doesn't keep job records, so rely on the output file.

What does ExitCode 0:9 mean in sacct?

The number before the colon is the exit code of your job script and the number after it is the signal that ended the process. 0:9 means the process was killed by signal 9 (SIGKILL) instead of exiting on its own; the State column tells you more.

How do I see how much memory my job used?

For a finished job, run sacct -j <jobid> --format=JobID,JobName,MaxRSS,State without -X and read MaxRSS on the step rows, or run seff <jobid> if your cluster has it. While a batch job runs, use sstat -j <jobid>.batch --format=JobID,MaxRSS.

How do I cancel a Slurm job?

Run scancel with the job ID, for example scancel 12345, for a pending or running job. It also cancels job arrays and single job steps, and the job then shows the CA (CANCELLED) state.

Does Runpeek work if compute nodes have no internet access?

No. Runpeek needs outbound HTTPS from the compute node to runpeek.app, and many clusters block it. Test it once from an interactive shell on a node, with srun --pty bash and then curl -sI https://runpeek.app. If the request fails, follow slurm-<jobid>.out with tail -F from the login node instead.

Check on long runs from your phone

Change one import and every tqdm bar prints a link you can open on any device. Get a Discord or Slack message when the run finishes, fails or stalls.

Get a live linkpip install tqdm-live

Sources