How to check the progress of a Python script that is already running

Find the process and its output

Start by finding the process ID (PID) of your job. pgrep -af python lists all running Python processes with their full command lines.

If you started the script inside tmux or screen, reattach to see its terminal. If you redirected stdout to a file or used nohup, follow the file with tail -F nohup.out (see how to follow a log with tail).

Sometimes the log sits at zero bytes even though your script calls print(). Python block-buffers stdout when it is not a terminal, so output stays in memory until the buffer fills or the script ends, and you can't change that on a running process. Check whether output files are still growing instead.

pgrep -af python
tail -F nohup.out
ls -lht /data/output | head

See where it is with py-spy dump

When there is no log to read, py-spy looks inside the Python process from the outside. It reads the target's memory from a separate process and does not run code inside your job, though by default it pauses the process briefly while sampling (--nonblocking avoids the pause, at the cost of occasional partial stacks).

py-spy dump --pid <pid> prints the current call stack of every Python thread without stopping the job. Run it twice a few seconds apart: if the line numbers change, the script is moving. --locals also prints the local variables of each frame, which often includes the loop counter.

On Linux, attaching to a process that py-spy did not start usually needs sudo, and macOS always needs root.

sudo py-spy dump --pid 12345 --locals

Process 12345: python process_rows.py
Python v3.13.14 (/usr/local/bin/python3.13)

Thread 12345 (idle): "MainThread"
    handle (process_rows.py:5)
        Arguments:
            row: 295
    main (process_rows.py:10)
        Locals:
            rows: <range at 0x7ffa3fed03f0>
            i: 295
            row: 295
    <module> (process_rows.py:12)

The i: 295 line under main is the iteration the loop is on right now. If you know how many items the loop has, you know how far along the job is. The py-spy guide covers top, record and permissions in depth.

Measure progress through the input file

If your script reads a big file from start to end, the Linux kernel already tracks how many bytes it has read:

1. ls -l /proc/<pid>/fd shows which descriptor number points to your input file.
2. /proc/<pid>/fdinfo/<fd> shows the current byte offset in the pos: field.
3. Divide pos by the file size to get the share that is done.

pv -d <pid> does the math for you: it draws a bar with rate and ETA for every regular file the process has open, and pv -d <pid>:<fd> watches a single descriptor. lsof -o -p <pid> shows the same offsets with a leading 0t.

$ ls -l /proc/12345/fd
lr-x------ 1 user user 64 Oct 30 00:27 3 -> /tmp/big.bin
$ grep pos /proc/12345/fdinfo/3
pos:    41000960
$ lsof -o -p 12345 | grep big.bin
python3 12345 user   3r   REG 0,2055 0t83001344  31233773 /tmp/big.bin
$ pv -d 12345
   3:/tmp/big.bin:  185MiB 0:00:01 [ 184MiB/s] [========>     ]  64% ETA 0:00:00

Here the 300,000,000-byte file was 13.7% read at the first check (41,000,960 bytes). Each method answers a different question:

Check whether it is stuck

A quiet script might be working through a slow step, or it might be hung. Check its elapsed time, CPU use and state with ps.

If CPU use is near 0%, no new output appears, and two py-spy dump runs a few seconds apart show the same stack, the job is usually waiting on something external such as a network response, a lock or input. strace -p <pid> shows the system calls it makes in real time, and -e trace=read,write narrows that down to reads and writes so you can see whether data is still moving.

ps -o pid,etime,%cpu,stat -p 12345
sudo strace -p 12345 -e trace=read,write

Next time, start it with a progress link

You can't attach a progress bar to a process that is already running; a real bar takes a restart. Before the next long run, wrap the main loop in tqdm, and start Python with python -u so output isn't stuck in a buffer when it goes to a file.