How to monitor ML training remotely, including from your phone

The four ways to watch a remote training run

When a model trains on a remote GPU box, a rented cloud GPU, Colab or Kaggle, you don't want to sit at your desk to make sure it's alive. You want to check progress from your laptop or your phone while you're away, and get a ping if the run finishes, crashes or silently hangs.

There are four ways to set this up. You can stick to plain SSH plus tmux, run a TensorBoard or MLflow dashboard over an SSH tunnel, log your metrics to a hosted experiment tracker, or use a live progress link with built-in alerts. Each one trades setup time against what you can actually see on a small screen.

SSH and tmux: nothing to install

If you have SSH access, tmux is probably already installed, and it keeps your process running when the SSH connection drops. You start a session with tmux new -s train, kick off your script, detach with Ctrl-b then d, and reattach later to see where things stand.

tmux new -s train
python train.py
# detach with Ctrl-b, then d. Later, from any SSH session:
tmux attach -t train

The catch is that you need a terminal every time you want to check in. From a laptop that's fine, but from a phone you need an SSH client app and your SSH key on the phone, which is a pain when you just want to glance at the epoch count. And since tmux only holds the terminal open, it won't tell you if the script died five minutes after you walked away.

TensorBoard or MLflow through an SSH tunnel

When you need loss curves and metric charts, TensorBoard and MLflow are the standard self-hosted tools. TensorBoard 2.21.0 serves on port 6006 and listens on localhost by default; --bind_all exposes it on every network interface, which also exposes it to anyone who can reach the machine. MLflow 3.17.2 works the same way: mlflow server listens on 127.0.0.1 port 5000, and its own help text says the host setting "is NOT a security setting".

Instead of opening ports to the internet, run an SSH local port forward. The remote dashboard then opens at http://localhost:6006 on your laptop.

# on the GPU box
tensorboard --logdir runs --port 6006

# on your laptop
ssh -N -L 6006:localhost:6006 user@gpu-box

That works on a laptop, but it's awkward from a phone. And neither tool sends an alert when a training run crashes or stops making progress.

Hosted experiment trackers

Hosted trackers give you charts in a normal browser and keep the history of your runs, so checking from your phone is just opening a webpage. Weights & Biases is now part of CoreWeave and sold as CoreWeave Forge, with a Free plan at $0 and Pro starting at $60 per month. Trackio is an open-source (MIT) tracker from Hugging Face with a wandb-compatible API, currently at version 0.44.0.

Both can notify you, but the alert is code inside your training script. In W&B, run.alert() sends an email or Slack message when your code reaches that line (CoreWeave lists Slack and email alerts as a Pro feature), and Trackio's trackio.alert() posts to a Slack or Discord webhook the same way. If the process is killed, that line never runs.

A live progress link with alerts

Sometimes you don't want an account or logging calls just to watch a progress bar from your phone. Runpeek streams your tqdm bars to a read-only link that opens in any phone browser, with no account. Start the script with tqdml run and the link also shows its console output.

tqdml run wraps the command, patches tqdm inside the script, streams stdout and stderr, and closes the run with the exit code. TQDMLIVE_NOTIFY takes a comma-separated list of webhook URLs and pings them when the run finishes or fails. PYTHONUNBUFFERED=1 makes Python print lines as they happen instead of in buffered chunks.

To also get the stall alert, register the webhook from inside the script. Here a run that makes no progress for 15 minutes triggers a ping. Keep the window at 30 minutes or less, because a run's link expires 30 minutes after its last update.

import os
import tqdm_live

tqdm_live.patch()
tqdm_live.start_run("resnet50")
tqdm_live.notify(os.environ["DISCORD_WEBHOOK_URL"], stall_after_minutes=15)

Runpeek does not draw loss curves; it shows the bars with their rate and ETA, plus the log lines under tqdml run. Links expire 30 minutes after the last update, so it is a view of the live run, not a history, and it pairs with any tracker.

Compare the options

Pick by whether you need run history or just want to know tonight's job is still moving.

What to alert on: finished, crashed, stalled

When you set up notifications for a remote run, there are three states to cover. Finished is the easy one: the process exits cleanly and you can go look at the checkpoints or shut down the rented instance. Crashed is nearly as simple, as long as whatever wraps your script watches for an uncaught exception or a non-zero exit code.

Stalled is the one people miss. A hung data loader, a deadlocked distributed job, a Colab runtime that disconnects (free Colab runtimes run for at most 12 hours) or a spot GPU that gets reclaimed never raises an exception. Your process either waits forever or disappears without running its finally block, so an alert call inside your code never fires.

Only a "no progress for N minutes" check that lives outside the machine catches that case. Pick a window a bit longer than your slowest normal gap, such as a long validation pass or a checkpoint save, so it doesn't page you for nothing.