Python logging to a file: basicConfig, handlers and rotation

Log to a file with basicConfig

The fastest way to replace print() calls with a log file is logging.basicConfig(). Pass filename, level, format and encoding once when your program starts; encoding was added in Python 3.9.

The root logger's default level is WARNING, so without level your info() and debug() lines are silently dropped. filemode defaults to 'a', so new runs append to the same file; pass filemode="w" to start fresh each run.

import logging

logging.basicConfig(
    filename="app.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    encoding="utf-8",
)
log = logging.getLogger(__name__)
log.info("started")
log.warning("disk at %d%%", 91)

2026-10-30 02:30:04,015 INFO __main__: started
2026-10-30 02:30:04,015 WARNING __main__: disk at 91%

Without format, lines look like INFO:root:message. One catch comes up often in notebooks and larger projects: basicConfig() does nothing if the root logger already has handlers, which happens when a library or the notebook kernel configures logging first. Pass force=True (Python 3.8+) to remove and close the existing handlers before yours are added.

Log to a file and the console at once

For a training script or data pipeline, you usually want a file on disk plus live output in your terminal. Pass a list to handlers: one FileHandler for the file and one StreamHandler, which writes to sys.stderr when you give it no stream.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    handlers=[
        logging.FileHandler("train.log", encoding="utf-8"),
        logging.StreamHandler(),
    ],
)
logging.info("epoch 1 done")

Don't pass filename or stream together with handlers. Python doesn't merge them; it raises ValueError: 'stream' or 'filename' should not be specified together with 'handlers'. Put the path and encoding on the FileHandler instead.

Rotate log files by size or by time

A plain FileHandler grows until you delete the file or fill the disk. The handlers in logging.handlers rotate it for you:

• RotatingFileHandler rolls over when the file is about to exceed maxBytes and keeps backupCount old files named .1, .2 and so on. If either value is 0, it never rotates.
• TimedRotatingFileHandler with when="midnight" and backupCount=7 starts a new file at midnight, adds a %Y-%m-%d suffix to the old one and keeps at most 7.
• WatchedFileHandler reopens the file when an external tool such as logrotate moves it. It isn't meant for Windows.

import logging
from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler("app.log", maxBytes=10_000_000, backupCount=5, encoding="utf-8")
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s"))

log = logging.getLogger("etl")
log.setLevel(logging.INFO)
log.addHandler(handler)

This keeps at most six files of about 10 MB each: app.log plus app.log.1 through app.log.5. For daily files, swap in TimedRotatingFileHandler("app.log", when="midnight", backupCount=7, encoding="utf-8").

Format strings that make logs useful

The default INFO:root:message format doesn't say when an event happened or where it came from. Add any LogRecord attribute to format and set the timestamp layout with datefmt, which uses time.strftime codes:

• %(asctime)s for the time, %(levelname)s for the level and %(message)s for the message.
• %(name)s for the logger name.
• %(process)d and %(threadName)s for the process ID and thread name.
• %(funcName)s and %(lineno)d for the function and line number.

Create one logger per module with logging.getLogger(__name__), so %(name)s shows which module wrote each line. Pass variables as arguments (log.info("processed batch %d of %d", i, total)) instead of f-strings, so formatting only happens when the record is emitted.

logging.basicConfig(
    filename="app.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s %(funcName)s:%(lineno)d %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

2026-10-30 02:30:04 INFO     __main__ run:13 processed batch 1 of 2

Log inside tqdm loops without breaking the bar

If you log to the console while a tqdm bar is running, StreamHandler writes to sys.stderr over the bar and leaves broken fragments. Wrap the loop in logging_redirect_tqdm from tqdm.contrib.logging: console logging goes through tqdm.write, so each message prints cleanly above the bar, and file handlers keep writing as normal.

import logging
from tqdm import tqdm
from tqdm.contrib.logging import logging_redirect_tqdm

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
log = logging.getLogger(__name__)

with logging_redirect_tqdm():
    for step in tqdm(range(3), desc="steps"):
        log.info("step %d loss=%.3f", step, 1 / (step + 1))

A log file on a server records what already happened, but it doesn't show how far along a running job is unless you SSH in and tail it.