Forum Discussion
Configuring Spark Diagnostic Emitter using Logs Ingestion API
Hi,
I am trying to configure spark diagnostic emitter using logs ingestion API where the doc states only two methods for authentication where one is the client secret and other is the client cert method. I want to know if we can authenticate using Managed Identity for this purpose.
This is the doc link I am following: Collect logs and metrics with Azure Log Analytics - Microsoft Fabric | Microsoft Learn
10 Replies
Hi ISHPREETKaur !
Thanks for your question here. Straight answer to your question is — there is no way now on using anything other than two options:
Authentication supports both service principal client secret and certificate-based options.
Links to the additional doc, which was updated literally one week ago:
Also, as you may notice, it became generally available relatively recently. Maybe it will be updated later. Also, if it is needed, you may push this as an idea to Microsoft here:
https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas
I am ready to support it, to minimize operational efforts for supporting the logging infra.
BR, Yurri
If this answer added value to you, Kudos and Making it a solution will help me doing that work more & more.
Hello ISHPREETKaur
Managed Identity is not currently supported for authenticating the Fabric Apache Spark Diagnostic Emitter when using the Azure Monitor Logs Ingestion API. The official Fabric documentation explicitly lists only service principal–based authentication, using either a client secret or a client certificate.
Although the Azure Log Ingestion API supports Managed Identity in some Azure scenarios, this capability is not exposed in Fabric’s managed Spark Diagnostic Emitter. As a result, authentication must be configured with an Entra ID app registration, and Managed Identity cannot be used for this integration today.
Hi,
While I try to use my custom env for running jobs so that I can get logs in the LAW i am getting this warning in the spark job logs:
where SparkLogsLAW is the emitter I have created. Not sure why this is being skipped. Can you please help with this.
This is the runtime I am using and the spark properties:
Hi ISHPREETKaur ,
Thank you deborshi_nag , 4iurchenko for your inputs.
We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.
Thank you.Hi ISHPREETKaur ,
May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.
Thank you.
Hi deborshi_nag can you pls help with the class not found issue in log analytics emitter.
Hi ISHPREETKaur ,
Based on the logs you’ve shared, the configuration is being picked up correctly as seen with “configured AzureLogIngestion”, but the warning “Cannot find class for emitter SparkLogsLAW, skip” indicates that the underlying emitter class is not available in the runtime classpath. This typically happens when using a custom environment, as it overrides the default Fabric runtime and does not include certain built in components like the Azure Log Analytics diagnostic emitter. As a result, Spark is unable to resolve the emitter implementation and skips it.
To validate this, I would recommend running the same configuration on the default Fabric runtime without a custom environment, if it works there, it confirms this is an environment limitation rather than a configuration issue. Additionally, you may try removing spark.fabric.pools.skipStarterPools = true, to rule out any initialization impact.
Thank you.Hi v-echaithra , how will I configure my emitter without creating a custom environment? And also i was able to send logs to LAW till 26th March. When I tried to change a few fields in my env two days back I started getting this error. Did something change with the platform as now with the same set of properties I am not able to configure my emitter. I tried running this with the starter pool and removed spark.fabric.pools.skipStarterPools = true then also I get the same logs
Hi ISHPREETKaur ,
To configure the emitter without using a custom environment, you should define the required Spark properties directly at the notebook session level or within the Spark job definition, and ensure that the job is executed on the default Fabric runtime. The AzureLogIngestion emitter is a built-in component of the default runtime.
Since you have already tested with starter pools and removed spark.fabric.pools.skipStarterPools, the key steps to resolve this are to make sure that no custom environment is attached to your notebook or job, verify in Workspace Settings > Spark that a custom environment is not set as the default, and then run the workload explicitly on the default Fabric runtime.
Thank you.Hi ISHPREETKaur ,
Just following up to see if the Response provided by community members were helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.
Best regards,
Chaithra E.
