Synapse Spark mssparkutils run multiple and log analytics

[VictorMed]

Hi all,

Context:

We have developed a solution using Synapse Notebooks as Fabric hasn't been approved for production yet. 

The initial approach was to orchestrate each notebook using data pipelines, but it would take c.2-3min to spin up the spark session on each one (46 Notebooks). The reason we wanted to use this approach is to capture the logs with log analytics for each one of the notebooks. The whole solution was taking an average of 1 h 40 min.  Not ideal when the expectation is to refresh the data every 2 hours.

Reviewing Microsoft documentation,Introduction to Microsoft Spark utilities - Azure Synapse Analytics | Microsoft Learn, one suggested way to reduce runtime was to use the mssparkutils.notebook.runtime() function, allowing us to run multiple notebooks within the same spark session and share computing resources. 

With this new approach, we designed a DAG that reduced our execution to 45 min, more aligned with the initial expectation of refresh scheduling we had in mind.

Problem:

With the new implementation, we orchestrate the execution of the notebooks using mssparkutils in one pipeline with one notebook and a trigger. This means that we have lost the ability to monitor individual notebooks with log analytics, as it only monitors the main pipeline/spark session/notebook and not all the executions within it.

Has someone faced a similar issue? Is there a way to send to log analytics information regarding each notebook that is running inside the DAG in runmultiple?

We want to monitor start time, end time, and status (queue, in progress, succeeded, failed) and capture errors if they occur.

Thank you.

[v-gchenna-msft · Accepted solution]

Hi @VictorMed ,

Thanks for using Fabric Community.
I understand your concern about losing individual notebook monitoring with your new implementation. This is a common challenge when switching from separate orchestrations to a single spark session with mssparkutils.notebook.runtime(). While there's no direct way to capture individual notebook logs within the single session, but we have an alternative way to capture logs is - custom Logging within Notebooks

Hope this is helpful. Please let me know incase of further queries.

[VictorMed]

Hi @v-gchenna-msft 

Thank you for your answer and the information provided.

We'll discuss it internally and give it a go, we were trying to find a solution that wouldn't need to maintain customized logging, but it seems it's the only way right now.

Regards,

Victor

[v-gchenna-msft]

Hi @VictorMed ,

Glad to know that your query got answered.
Please continue using Fabric Community for your further queries.

[v-gchenna-msft]

Hi @VictorMed ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .