Retrieving Detailed Notebook-Level Error Logs When Triggering Notebooks Directly via REST API/FABCLI

Hello AnushShetty 

At the moment, if you use the REST API or FAB CLI to run a notebook directly, you won’t get all the detailed notebook error logs in-line. You’ll see the job or run status and a general failure message, but not the same detailed logs you’d see in the notebook interface or pipeline Monitoring. This is a limitation when notebooks are run outside a pipeline.

If you want to get driver logs and Spark event logs for a notebook run, you’ll need to find the Livy session ID and Spark application ID first, then use the Spark monitoring tools to access them.

If your workspace is connected to Azure Log Analytics and Spark diagnostic emitters are switched on, notebook logs—including custom Log4j messages—will be sent to Log Analytics - see link below. 

Monitor Apache Spark applications with Azure Log Analytics - Microsoft Fabric | Microsoft Learn