How to stop a GPU instance automatically when training finishes

Why shutdown is the wrong habit

Putting sudo shutdown -h now at the end of a training script is the obvious move when you want to go to sleep and stop paying for idle hours. On some clouds, though, shutting down the guest OS doesn't stop compute billing at all, and on the rest storage or IP charges keep running. What the OS sees as powering off and what the provider bills as releasing the hardware are two different things, so use the provider's own stop or terminate command.

The train-then-stop wrapper

The stop command has to run whether training succeeds, crashes or hangs. Don't chain it with && and don't use set -e: if training dies after twenty minutes with an out-of-memory error, && skips the stop line and the GPU bills all night. A hung job never exits at all, so timeout kills it after a deadline you set above your expected training time.

#!/usr/bin/env bash
set -u

timeout 10h python train.py
echo "train.py exited with code $?"

runpodctl pod stop "$RUNPOD_POD_ID"

Start it with nohup so it survives your SSH session. PYTHONUNBUFFERED=1 makes Python write its output to the log as it happens instead of in buffered chunks.

chmod +x train_then_stop.sh
PYTHONUNBUFFERED=1 nohup ./train_then_stop.sh > train.log 2>&1 &

We tested this wrapper with a fake runpodctl on the PATH: a script that exited with code 1 and a script that hung past a 3-second timeout both reached the stop line. Swap the last line for your provider's command from the sections below.

Runpod

On Runpod you don't need to install anything or copy API keys into the pod. runpodctl comes preinstalled, and the RUNPOD_POD_ID variable is set inside every pod.

As Runpod's docs put it: "Stopping a Pod releases the GPU and preserves data in /workspace (volume disk). Container disk data is cleared." Anything outside /workspace is gone, so write checkpoints and logs there. "You'll still be charged for volume disk storage while stopped." To bring the bill for the pod to zero once your outputs are saved elsewhere, run runpodctl pod delete $RUNPOD_POD_ID instead, which terminates it.

Vast.ai

Vast.ai sets CONTAINER_ID and CONTAINER_API_KEY in every instance. It's a per-instance API key meant for running CLI commands from inside the container. The CLI isn't in every image, so install it first.

curl -fsSL https://vast.ai/install.sh | bash
vastai stop instance "$CONTAINER_ID" --api-key "$CONTAINER_API_KEY"

vastai destroy instance $CONTAINER_ID removes the instance completely. Vast.ai's billing FAQ is clear that you are still charged storage costs for stopped instances, and a stopped instance has no GPU reservation, so someone else may rent the GPU before you restart. Fine for copying checkpoints off in the morning, not for resuming on the same card.

Lambda: terminate, because there is no stop

Lambda has no stopped state. The docs say "Lambda Cloud instances can only be launched, restarted, or terminated," and warn: "Do not use commands such as sudo shutdown -h now or sudo systemctl poweroff to try to terminate or suspend your instances. These commands will not work as expected and will cause your instances to go into Alert status, and billing will continue."

To stop paying, terminate the instance through the Lambda Cloud API. Set LAMBDA_API_KEY and LAMBDA_INSTANCE_ID yourself; the ID is in the Lambda console and in the API's instance list.

curl -fsS -X POST https://cloud.lambda.ai/api/v1/instance-operations/terminate \
  -H "Authorization: Bearer $LAMBDA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"instance_ids\": [\"$LAMBDA_INSTANCE_ID\"]}"

Terminating deletes the instance and its local disk. Copy checkpoints to external storage first, and only terminate if the copy succeeded.

AWS, Google Cloud and Azure

On AWS EC2, a shutdown from inside an EBS-backed instance stops it by default. A stopped instance isn't billed for compute, but its EBS volumes and Elastic IPs still are. To make the same shutdown terminate the instance, run aws ec2 modify-instance-attribute --instance-id i-... --instance-initiated-shutdown-behavior terminate once.

Google Cloud handles a guest shutdown the same way: the VM moves to TERMINATED, which is Google's name for a stopped VM you can start again. Google's docs: "Stopped instances do not incur VM usage charges while they are stopped. However, resources that the VM is using, such as persistent disks and static IP addresses, will continue to be charged until they are deleted."

Azure is the trap. A VM shut down from the guest OS, or with az vm stop, sits in the Stopped state and is still billed. Only Deallocated stops compute charges, and disks and networking keep billing. To deallocate from inside the VM, give its managed identity a role that allows deallocating it.

az login --identity
az vm deallocate --resource-group my-rg --name my-gpu-vm

When the stop command never runs

A stop line only runs if the wrapper gets there. A deadlocked data loader keeps the GPU at 0% until the timeout fires, hours later. A reclaimed spot instance disappears before the script can report anything. To find out while it still matters, the check has to live outside the machine.

Put these lines at the top of train.py, not around the wrapper, and start the wrapper as before. A clean exit marks the run finished and an unhandled exception marks it failed; if your script catches errors and exits with sys.exit(1), call tqdm_live.end_run("failed", exit_code=1) in the except block first. Keep the stall window at 30 minutes or less.