Suppressing logging...

Hi peterott, 

When you enable Log Analytics integration for Fabric notebooks, the Spark diagnostic emitter sends all categories by default:

• Log (driver logs)
• EventLog (Spark events)
• Metrics (runtime metrics)

These include verbose platform logs for every Spark job, executor, and metric—not just your custom messages. That’s why ingestion explodes overnight.

You might want to have a look at Python logging in a notebook - Microsoft Fabric | Microsoft Learn on how to use the Python logging library. You can also use the example configuration code below. Added it in a utils-generic notebook and include (%run) it every notebook, so that logging is always consistent.

import logging

logging_config = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {
            "format": "%(asctime)s.%(msecs)03d [%(levelname)-8s] %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        }
    },
    "handlers": {
        "stdout": {
            "class": "logging.StreamHandler",
            "formatter": "simple",
            "stream": "ext://sys.stdout",
        }
    },
    "loggers": {
        "NotebookLogger": {
            "level": "DEBUG",
            "handlers": ["stdout"],
            "propagate": False,
            },
    },
}

logger = logging.getLogger("NotebookLogger")
logging.config.dictConfig(config=logging_config)

Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.