Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

nsprudhvi — Fri, 04 Oct 2024 12:38:51 GMT

Hi everyone,

I'm encountering an issue while running an Airflow task that interacts with Microsoft Fabric using the apache_airflow_microsoft_fabric_plugin. The task fails with a 401 Unauthorized error, and the logs indicate the following:

invalid_client: The request body must contain the following parameter: 'client_assertion' or 'client_secret'. Trace ID: 54844adb-4acd-4720-a99f-8b71ad921100 Correlation ID: fd79265c-dd63-457b-aa31-50ed665fec2d Timestamp: 2024-10-04 12:18:02Z

It seems the OAuth2 request is missing either a client_assertion or client_secret. I've verified that my Azure app registration has a client secret configured and that my Airflow connection (fabric-airflow-conn) is set up with the correct credentials.

Here's the traceback:

[2024-10-04, 14:18:02 CEST] {base.py:73} INFO - Using connection ID 'fabric-airflow-conn' for task execution. [2024-10-04, 14:18:02 CEST] {taskinstance.py:1824} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/apache_airflow_microsoft_fabric_plugin/hooks/fabric.py", line 132, in _get_token response.raise_for_status() File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://login.microsoftonline.com/117c4e93-3230-40d6-b1cd-57cdee469b2f/oauth2/v2.0/token During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/apache_airflow_microsoft_fabric_plugin/operators/fabric.py", line 122, in execute self.location = self.hook.run_fabric_item( File "/home/airflow/.local/lib/python3.8/site-packages/apache_airflow_microsoft_fabric_plugin/hooks/fabric.py", line 214, in run_fabric_item headers = self.get_headers() File "/home/airflow/.local/lib/python3.8/site-packages/apache_airflow_microsoft_fabric_plugin/hooks/fabric.py", line 161, in get_headers "Authorization": f"Bearer {self._get_token()}", File "/home/airflow/.local/lib/python3.8/site-packages/apache_airflow_microsoft_fabric_plugin/hooks/fabric.py", line 135, in _get_token raise AirflowException(msg) airflow.exceptions.AirflowException: Response: {"error":"invalid_client","error_description":"AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'. Trace ID: 54844adb-4acd-4720-a99f-8b71ad921100 Correlation ID: fd79265c-dd63-457b-aa31-50ed665fec2d Timestamp: 2024-10-04 12:18:02Z","error_codes":[7000218],"timestamp":"2024-10-04 12:18:02Z","trace_id":"54844adb-4acd-4720-a99f-8b71ad921100","correlation_id":"fd79265c-dd63-457b-aa31-50ed665fec2d","error_uri":"https://login.microsoftonline.com/error?code=7000218"} Status Code: 401 [2024-10-04, 14:18:02 CEST] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=Run_Fabric_Item, task_id=run_fabric_notebook, execution_date=20241004T121800, start_date=20241004T121801, end_date=20241004T121802 [2024-10-04, 14:18:02 CEST] {standard_task_runner.py:104} ERROR - Failed to execute job 104 for task run_fabric_notebook (Response: {"error":"invalid_client","error_description":"AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'. Trace ID: 54844adb-4acd-4720-a99f-8b71ad921100 Correlation ID: fd79265c-dd63-457b-aa31-50ed665fec2d Timestamp: 2024-10-04 12:18:02Z","error_codes":[7000218],"timestamp":"2024-10-04 12:18:02Z","trace_id":"54844adb-4acd-4720-a99f-8b71ad921100","correlation_id":"fd79265c-dd63-457b-aa31-50ed665fec2d","error_uri":"https://login.microsoftonline.com/error?code=7000218"} Status Code: 401; 50) [2024-10-04, 14:18:02 CEST] {local_task_job_runner.py:225} INFO - Task exited with return code 1

Has anyone else encountered this issue or have insights into how to resolve it?

Thanks in advance!

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

Anonymous — Mon, 07 Oct 2024 01:52:12 GMT

Hi nsprudhvi

Here are a few steps you can take to resolve this issue:

The error message suggests that the request body must contain either a client_assertion or client_secret. Make sure these parameters are included in your authentication request.

Verify that your configuration settings for basic authentication are correctly set up. This includes checking your tenant ID and other relevant settings.

You can try to re-generate your tokens according to this document:

OAuth 2.0 client credentials flow on the Microsoft identity platform - Microsoft identity platform | Microsoft Learn

Here's a link for your reference:

requests.exceptions.HTTPError: 401 Client Error · Issue #10 · ambika-garg/apache-airflow-microsoft-fabric-plugin · GitHub

Best Regards

Zhengdong Xu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

mattiasdesmet — Tue, 05 Nov 2024 20:35:56 GMT

I came across the same issue as reported above, but couldn't find the answer in these links. I finally resolved it by finding out that the Extra parameter in the Airflow Connector contained (as per documentation on various sites) "client_secret" : "xxxxx" while this should be "clientSecret" : "xxxxxx" Subtle, but time consuming. Even more so because the error message states that "client_secret" is missing.

I wrote a full setup guide on my blog that also explores Semantic model refresh in a DAG : Orchestrate Fabric Data workloads with Airflow - Mattias De Smet

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

BENLTAIEF — Fri, 09 May 2025 16:51:43 GMT

Thank you Mattias, it work for me. I spend a lot of time to find a solution for issu, but like you said "Even more the error message is clear"

Best regards

topic Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client) in Apache Airflow Job

Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)

Re: Unauthorized Error in Airflow Task with Microsoft Fabric Plugin (401 Invalid Client)