Get notified when model training finishes, crashes or hangs

A send function you can reuse

Start with one small function that posts to a Discord or Slack webhook stored in an environment variable. Give the request a timeout and swallow its errors: a network blip should never crash a 6-hour run at the finish line.

Two limits to keep in mind: Discord content holds up to 2,000 characters, and Slack incoming webhooks take about 1 message per second with the key text instead of content.

import os

import requests


def send(text):
    try:
        requests.post(
            os.environ["DISCORD_WEBHOOK_URL"],
            json={"content": text[:2000]},
            timeout=10,
        )
    except Exception as exc:
        print(f"notification failed: {exc!r}")

Keras: a callback with on_train_end

In Keras (latest 3.15.1), subclass keras.callbacks.Callback and override on_train_end(self, logs=None). logs holds the last epoch's metrics, so the final numbers go straight into the message.

Keras 3's fit() only reaches on_train_end when the epoch loop completes, so an exception during training skips it. Wrap model.fit() in try/except to report crashes too.

import keras


class NotifyCallback(keras.callbacks.Callback):
    def on_train_end(self, logs=None):
        logs = logs or {}
        send(f"Training finished: loss {logs.get('loss', float('nan')):.4f}, "
             f"val_loss {logs.get('val_loss', float('nan')):.4f}")


try:
    model.fit(x_train, y_train, validation_split=0.1, epochs=50, callbacks=[NotifyCallback()])
except Exception as exc:
    send(f"Training crashed: {exc!r}")
    raise

PyTorch: try, except and finally around the loop

A plain PyTorch loop needs no callback class. Wrap the epochs in try/except, send a message when the loop completes and send the exception when it fails. Catching BaseException also reports Ctrl-C.

Track the start time and the best validation loss as you go, so the message says how long it took and how good it got.

import time

started = time.monotonic()
best_val_loss = float("inf")
epoch = 0
try:
    for epoch in range(epochs):
        train_one_epoch(model, train_loader, optimizer)
        best_val_loss = min(best_val_loss, evaluate(model, val_loader))
except BaseException as exc:
    send(f"Training crashed in epoch {epoch} after {(time.monotonic() - started) / 60:.0f}m: {exc!r}")
    raise
send(f"Training finished in {(time.monotonic() - started) / 60:.0f}m, best val loss {best_val_loss:.4f}")

In a distributed script, send only from rank 0 so the other processes stay quiet.

Hugging Face Trainer: TrainerCallback.on_train_end

In Hugging Face transformers (latest 5.22.0, released 2026-10-29), subclass TrainerCallback and implement on_train_end(self, args, state, control, **kwargs). Pass it with callbacks=[...] when you create the Trainer, or add it later with trainer.add_callback().

state is a TrainerState with the numbers you want in the message:

• state.global_step
• state.best_metric and state.best_model_checkpoint, filled in only when you track the best model (metric_for_best_model or load_best_model_at_end=True) and evaluation runs
• state.is_world_process_zero, which is true only on the main process

Check state.is_world_process_zero before sending, so a multi-GPU run sends one message instead of one per GPU. As with Keras, an exception skips the callback, so wrap trainer.train().

from transformers import Trainer, TrainerCallback


class NotifyCallback(TrainerCallback):
    def on_train_end(self, args, state, control, **kwargs):
        if state.is_world_process_zero:
            send(f"Training finished at step {state.global_step}, epoch {state.epoch:.1f}")


trainer = Trainer(model=model, args=training_args, train_dataset=train_ds, callbacks=[NotifyCallback()])
try:
    trainer.train()
except Exception as exc:
    if trainer.is_world_process_zero():
        send(f"Training crashed: {exc!r}")
    raise

Crashes and hangs need an outside check

An end-of-training hook only covers a clean finish, and try/except only works while the process is alive and moving.

When the Linux out-of-memory killer sends SIGKILL, no except, finally or atexit code runs. When a data loader deadlocks or a collective call hangs, the process can sit there without raising anything. Nothing inside a killed or hung process can send a message, so something outside has to notice the silence. The stuck script guide shows how to find where a hung run is waiting.

Keras draws its own progress bar instead of tqdm, so with Keras you get the run link and the alerts, and bars only for loops you wrap in tqdm yourself.