Suppressing logging...

Hi peterott, 

When you enable Log Analytics integration for Fabric notebooks, the Spark diagnostic emitter sends all categories by default:

• Log (driver logs)
• EventLog (Spark events)
• Metrics (runtime metrics)

These include verbose platform logs for every Spark job, executor, and metric—not just your custom messages. That’s why ingestion explodes overnight.

You might want to have a look at Python logging in a notebook - Microsoft Fabric | Microsoft Learn on how to use the Python logging library. You can also use the example configuration code below. Added it in a utils-generic notebook and include (%run) it every notebook, so that logging is always consistent.

import logging

logging_config = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {
            "format": "%(asctime)s.%(msecs)03d [%(levelname)-8s] %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        }
    },
    "handlers": {
        "stdout": {
            "class": "logging.StreamHandler",
            "formatter": "simple",
            "stream": "ext://sys.stdout",
        }
    },
    "loggers": {
        "NotebookLogger": {
            "level": "DEBUG",
            "handlers": ["stdout"],
            "propagate": False,
            },
    },
}

logger = logging.getLogger("NotebookLogger")
logging.config.dictConfig(config=logging_config)

Hope this helps. If so, please give a Kudos 👍 and mark as Accepted Solution ✔️.

Hi peterott, 

My org saw the same thing. We decided to not use Log Analytics and instead use the admin REST APIs to capture audit data. 

SPecifically we use the GetActivityEvents endpoint nightly to pull audit data into a SQL database. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

Hi peterott,

Thank you for reaching out to Microsoft Fabric Community.

Thank you nielsvdc and tayloramy for the prompt response. 

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa