Find out why a Python script is stuck, and get alerted next time

Stuck or slow? Check the CPU first

Before you kill a silent job, ask the OS what it is doing. ps and top show the process state and CPU use without touching Python.

ps -o pid,stat,%cpu,etime,cmd -p 12345
top -p 12345

Read the state and the CPU together:

• S (interruptible sleep) near 0% CPU for minutes: waiting for an event, such as a network reply, a lock or a child process.
• D (uninterruptible sleep): waiting on I/O, usually disk or network storage.
• R (running) near 100% CPU: computing, which is either slow work making progress or an endless loop.

State and CPU tell you whether the script is waiting or working, but not which line it is on. The stack tells you that.

See the stack with py-spy

py-spy (0.4.2, released 2026-04-24) reads a running Python process from outside, so you can inspect a live job without restarting it or changing its code.

py-spy dump --pid prints the current call stack of every thread and exits. Add --locals to show each frame's local variables, or run py-spy top for a live view of where the time goes.

pip install py-spy
sudo py-spy dump --pid 12345
sudo py-spy dump --pid 12345 --locals
sudo py-spy top --pid 12345

Take two dumps a minute apart. If the top frame is the same both times and CPU sits at 0%, the script is stuck waiting on that call.

Reading another process needs permission. Without it, py-spy stops with "Permission Denied: Try running again with elevated permissions".

• Linux: attaching to a process that isn't your own child usually needs sudo.
• macOS: always needs root.
• Docker: start the container with --cap-add SYS_PTRACE.

Make Python dump its own traceback

Where py-spy can't attach, the standard library's faulthandler dumps every thread's traceback from inside the process. You have to set it up before the hang.

faulthandler.dump_traceback_later(timeout) starts a watchdog thread that dumps all threads after timeout seconds. Calling it again resets the timer, so one call per loop iteration only fires when a single iteration takes too long. On Unix, faulthandler.register(signal.SIGUSR1) waits until you send kill -USR1; it is not available on Windows.

import faulthandler
import signal

trace_log = open("/tmp/job.tracebacks.log", "w")
faulthandler.register(signal.SIGUSR1, file=trace_log, all_threads=True)

for item in items:
    faulthandler.dump_traceback_later(600, file=trace_log)
    process(item)
faulthandler.cancel_dump_traceback_later()

kill -USR1 12345
cat /tmp/job.tracebacks.log

Keep the log file open for the whole run, because faulthandler writes to it directly when the timer or the signal fires.

Put a ceiling on every wait

Most hangs are waits without an upper limit. requests has no default timeout, and its docs warn that leaving one out "can cause your program to hang indefinitely".

Pass a timeout to every requests call, subprocess.run and queue get(), so a stalled dependency raises an exception instead of blocking forever. For requests, (5, 30) means 5 seconds to connect and 30 seconds to read.

import subprocess

import requests

response = requests.get(url, timeout=(5, 30))
subprocess.run(["rsync", "-a", "src/", "dst/"], timeout=600, check=True)
item = work_queue.get(timeout=60)

For a hard ceiling on the whole script, run it under GNU timeout. It stops the job at the deadline and exits with status 124, unless you pass --preserve-status. -k 30s sends SIGKILL if the job is still running 30 seconds after the first signal.

By default timeout sends SIGTERM, which ends Python without running finally blocks. -s INT raises KeyboardInterrupt instead, so your cleanup and notification code still run.

timeout -s INT -k 30s 2h python job.py
echo $?

Get alerted when a script goes quiet

For unattended jobs, you want to hear about a stall when it starts, not the next morning. The simple version is a heartbeat file: the script touches it on every iteration, and cron sends an alert when the file gets too old.

from pathlib import Path

heartbeat = Path("/tmp/job.heartbeat")
heartbeat.touch()
try:
    for item in items:
        process(item)
        heartbeat.touch()
finally:
    heartbeat.unlink(missing_ok=True)

*/5 * * * * [ -n "$(find /tmp/job.heartbeat -mmin +10 2>/dev/null)" ] && curl -s -d "job.py quiet for 10 minutes" ntfy.sh/your-topic

Cron checks every 5 minutes and sends an alert on each check while the file is more than 10 minutes old. Deleting the file at the end keeps a finished job quiet. The ntfy guide covers the phone side.

Tools compared