nohup: keep a command running after logout and log its output

What nohup does

nohup runs a command with the hangup signal (SIGHUP) ignored, so closing your terminal or losing your SSH connection does not kill it. It does not put the command in the background on its own, so you add & at the end of the line for that.

Run on a terminal without any redirects, GNU nohup (tested on coreutils 9.7) makes the command ignore SIGHUP and moves its input and output off your terminal. It redirects standard input from an unreadable file, appends standard output to nohup.out in the current directory, and sends standard error there too. If the current directory is not writable, it uses $HOME/nohup.out instead, and it creates the file with mode 600 so only you can read it.

nohup itself exits with status 125 if it fails, 126 if the command is found but cannot be run and 127 if the command is not found. Otherwise it returns the command's own exit status.

Send the output to a file you choose

To keep output out of nohup.out, redirect standard output and standard error yourself, before the trailing &. > overwrites the log each time and >> appends to it. Right after you start the job, bash stores its process ID in $!.

nohup python -u train.py > train.log 2>&1 &
echo $!

What nohup does with your streams, and where its status message lands, depends on which of them are still attached to the terminal:

Check on a nohup job later

While you are still in the shell that started it, jobs -l lists the job with its PID. After you log out and come back in a new SSH session, jobs shows nothing, because it only lists jobs of the current shell. Use pgrep -af or ps -fp PID to check the process is alive, and tail -f to follow the log:

pgrep -af train.py
ps -fp 8628
tail -f train.log

That means an SSH session, tail and ps every time you want to know whether it is still running or what it printed last.

Why Python prints nothing under nohup

You start a Python script under nohup, tail the log, and the file stays empty even though the script is running. Python block-buffers standard output whenever it is not a terminal, which includes files, pipes and nohup.out, so your print calls arrive in chunks or only when the script exits.

In a test with Python 3.13, a script printing one line per second had written 0 of its 5 lines after 2.5 seconds, while the same script with python -u had written 3. Fix it with -u, the PYTHONUNBUFFERED=1 environment variable or print(..., flush=True):

nohup python -u train.py > train.log 2>&1 &
PYTHONUNBUFFERED=1 nohup python train.py > train.log 2>&1 &

More on how Python buffers its output is in the Python unbuffered output guide.

Stop a nohup job

kill PID sends SIGTERM, so the process can exit cleanly. If it ignores that, kill -9 PID stops it for good. kill %1 also works, but only from the shell that started the job.

nohup, disown, setsid or tmux

nohup has limits. A nohup job does not survive a reboot, and it does not help when systemd-logind on the server runs with KillUserProcesses=yes, which kills every process of your session at logout.

If you already started a command without nohup, the bash builtin disown removes it from the shell's job table, so the shell does not send it SIGHUP when you exit. setsid cmd starts a command in a new session with no controlling terminal at all. tmux or screen keep a whole terminal you can reattach to later. All of them are compared in run a process in the background on Linux.