Jupyter notebooks

papermill: run a Jupyter notebook from the command line

Run papermill in.ipynb out.ipynb -p alpha 0.6 to execute every cell and save a copy with all outputs; -p overrides variables in the cell tagged parameters. papermill 2.7.0 needs Python 3.10 or newer, hides cell output unless you add --log-output, waits forever on a slow cell unless you set --execution-timeout, and exits with code 1 when a cell raises.

5 min readLogs and monitoring

At a glance

  • papermill report.ipynb out.ipynb runs every cell and writes a new notebook with the outputs.
  • -p NAME VALUE, -f params.yaml or -y override variables in the cell tagged parameters.
  • --log-output streams cell output to your terminal; it is off by default.
  • jupyter nbconvert --to notebook --execute runs a notebook when you don't need parameters.

Run a notebook with papermill

papermill 2.7.0 (released February 27, 2026) requires Python 3.10 or newer. Install it with pip install papermill, then run papermill INPUT OUTPUT to execute every cell in INPUT and write a new notebook with all outputs to OUTPUT, leaving your original file clean.

Bash
pip install papermill
mkdir -p runs
papermill report.ipynb runs/report-2026-10-30.ipynb

papermill won't create a missing output folder, so mkdir -p runs comes first.

You can pick a specific Jupyter kernel with -k (or --kernel) and set the working directory the notebook runs in with --cwd, which helps when your cells load files using relative paths.

Pass parameters

To parameterize a notebook, put your default variable assignments in one cell and give that cell the parameters tag in Jupyter's cell metadata. papermill inserts a new cell tagged injected-parameters right after it with your overrides, so every cell below uses the new values.

Python
# Cell tagged "parameters"
alpha = 0.1
n_batches = 5
Bash
papermill report.ipynb out.ipynb -p alpha 0.6 -p n_batches 3
papermill report.ipynb out.ipynb -f params.yaml

With -p alpha 0.6 -p n_batches 3, the injected cell contains # Parameters, alpha = 0.6 and n_batches = 3. You can pass values on the command line or load them from YAML:

  • -p NAME VALUE turns integers, floats, True, False and None into Python values and keeps everything else a string.
  • -r NAME VALUE keeps the value a raw string even if it looks like a number.
  • -f PATH reads parameters from a YAML file, and -y TEXT takes a YAML string.

These are the main options and their defaults:

OptionDefaultWhat it does
-p NAME VALUENot setSets one parameter
-f FILENot setReads parameters from a YAML file
--log-outputOffWrites cell output to the terminal as cells run
--progress-barOn, unless --log-output is setShows a bar of executed cells
--execution-timeout SECNo limitFails the run if one cell runs longer
--request-save-on-cell-executeOnSaves the output notebook after each cell
-k NAMEKernel in the notebook metadataPicks the Jupyter kernel

Watch the output while it runs

By default, papermill shows a progress bar of executed cells and keeps your cells' print statements out of the terminal. Pass --log-output to send the notebook's output to the logger as each cell runs; it also turns the cell bar off unless you add --progress-bar.

Terminal
$ papermill report.ipynb out.ipynb -p alpha 0.6 -p n_batches 3 --log-output
Input Notebook:  report.ipynb
Output Notebook: out.ipynb
Executing notebook with kernel: python3
Executing Cell 1---------------------------------------
Ending Cell 1------------------------------------------
Executing Cell 2---------------------------------------
Ending Cell 2------------------------------------------
Executing Cell 3---------------------------------------
batches: 100%|██████████| 3/3 [00:00<00:00,  3.33it/s]
alpha is 0.6
Ending Cell 3------------------------------------------

Text-mode tqdm bars show up between the cell markers. To keep the notebook's streams apart from papermill's own log lines, pass --stdout-file and --stderr-file.

Because --request-save-on-cell-execute is on by default, papermill saves the output notebook after every cell, so you can open out.ipynb mid-run and look at the plots and tables of the cells that already finished.

Run it in the background

For a long notebook on a remote server, run papermill under nohup with the log redirected to a file and follow it with tail -F, or start it inside a tmux session that you can detach from and reattach to later.

Bash
nohup papermill report.ipynb out.ipynb --log-output > run.log 2>&1 &
tail -F run.log

When a cell raises an exception, papermill stops executing further cells, writes the traceback into the output notebook with an error banner at the top, prints a PapermillExecutionError, and exits with code 1. That exit code lets cron, systemd timers and CI pipelines detect the failure. papermill waits forever for a cell by default, so set --execution-timeout for unattended jobs.

nbconvert --execute as the alternative

If you don't need parameters, jupyter nbconvert already ships with Jupyter. jupyter nbconvert --to notebook --execute mynotebook.ipynb executes the cells and saves the result as mynotebook.nbconvert.ipynb.

Bash
jupyter nbconvert --to notebook --execute report.ipynb --output executed.ipynb
jupyter nbconvert --to notebook --execute report.ipynb --ExecutePreprocessor.timeout=600

Use --output to pick the filename or --inplace to overwrite the input notebook. The nbconvert docs still say each cell gets 30 seconds by default, but in a test with nbconvert 7.17.2 a 40-second cell finished without a timeout, so set --ExecutePreprocessor.timeout yourself when you want a limit. nbconvert stops at the first exception unless you pass --allow-errors.

Live progress from inside the notebook

papermill's bar tells you which cell is running, not how far along a multi-hour loop inside that cell is. With tqdm-live installed, from tqdm_live import tqdm inside the notebook streams your loop's bars to a live link, as long as the machine has outbound HTTPS. Widget bars from tqdm.notebook aren't patched, so import the text bar from tqdm_live.

tqdm_live.start_run("name") groups all bars under one link. When papermill finishes, the kernel exits and the run is marked finished. An exception in a cell doesn't mark the run failed on its own, so call tqdm_live.end_run("failed", exit_code=1) in an except block before re-raising. In an interactive notebook, where the kernel stays open, call tqdm_live.end_run("finished") in the last cell.

FAQ

What happens when a cell raises an error?

papermill stops executing the rest of the notebook, saves the traceback in the output notebook, prints it to the terminal and exits with code 1. Open the output notebook to see the error and the outputs of all earlier cells.

How do I set a timeout per cell?

papermill lets cells run forever by default; pass --execution-timeout SECONDS to fail the run when one cell takes longer. With jupyter nbconvert --execute, set --ExecutePreprocessor.timeout=600, or -1 to remove the limit.

Should I use papermill or nbconvert --execute?

Use papermill when you need parameters from the command line or a YAML file, live output with --log-output, or a saved notebook after every cell. Use jupyter nbconvert --to notebook --execute to run a fixed notebook with tools that already ship with Jupyter.

Can I read parameters from a YAML file?

Yes. Pass -f params.yaml and papermill injects every key and value from the file into the injected-parameters cell. You can also pass a YAML string directly with -y.

Why don't I see my print output or tqdm bars?

papermill hides cell output in the terminal unless you pass --log-output. Text-mode tqdm bars then print between the cell markers, but widget bars from tqdm.notebook need a browser front end and don't render as text.

Check on long runs from your phone

Change one import and every tqdm bar prints a link you can open on any device. Get a Discord or Slack message when the run finishes, fails or stalls.

Get a live linkpip install tqdm-live

Sources