undefinedHTTP/1.1 401 Unauthorized when reading or writing to an EventHouse

[mkjit256]

Hello 

i had created the following function to write a pyspark dataframe to a Eventhouse Database table:

def write_pandas_to_kusto(df, KUSTO_URI, KUSTO_DATABASE, KUSTO_TABLE, accessToken, mode = "Append"):
    """
    Converts a Pandas DataFrame to a PySpark DataFrame and writes it to a Kusto destination.

    Args:
        df (pd.DataFrame): The Pandas DataFrame to be converted and written.
        KUSTO_URI (str): The URI of the Kusto cluster.
        KUSTO_DATABASE (str): The name of the Kusto database.
        KUSTO_TABLE (str): The name of the Kusto table.
        accessToken (str): The access token for authentication.
    """

    # Convert Pandas DataFrame to PySpark DataFrame
    df_sp = spark.createDataFrame(df)

    # Write the PySpark DataFrame to Kusto
    df_sp.write \
        .format("com.microsoft.kusto.spark.synapse.datasource") \
        .option("kustoCluster", KUSTO_URI) \
        .option("kustoDatabase", KUSTO_DATABASE) \
        .option("kustoTable", KUSTO_TABLE) \
        .option("accessToken", accessToken) \
        .mode(mode) \
        .save()

When trying to use the function after getting the acessToken:

accessToken = mssparkutils.credentials.getToken(KUSTO_URI)
write_pandas_to_kusto(df, KUSTO_URI, KUSTO_DATABASE, KUSTO_TABLE_OREB, accessToken, mode = "Append")

I am getting the error:

Py4JJavaError: An error occurred while calling o6046.save. : com.microsoft.azure.kusto.data.exceptions.DataServiceException: Http StatusCode='HTTP/1.1 401 Unauthorized'

I had created the eventhouse in the same workspace where i have my notebook running, and i am the workspace admin, so i am not sure why i am Unauthorized. I did get the KUSTO_URI from the eventhouse itself.

did i miss anything? 

[v-pagayam-msft]

Hi @mkjit256 ,
Thank you for reaching out to the Microsoft Fabric Community Forum about the issue you are encountering.

Providing the resolution steps which might assist you in resolving the issue:
1.Check the access token you are using is valid and is not expired.If expired, regenerate the token
2.If you are using an Azure AD  to obtain the token, ensure that it is properly registered and has the correct permissions
3.Ensure that you are using the latest version of the PySpark and Kusto libraries.

If still facing difficulty in resolving the issue, please feel free to reachout.

If this post helps, please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

[mkjit256]

Thanks for your reply,

Can you please specify how i am able to regenerate the token? In my organization we use Azure Active Directory, but i didn't get what should be properly registered and how do we verify if we have the correct permissions?

Thanks

[v-pagayam-msft]

Hi @mkjit256 ,
Thank you for your response.
Providing the document which may help you.
Refresh tokens in the Microsoft identity platform - Microsoft identity platform | Microsoft Learn

If still facing difficulty in resolving the issue, please feel free to reachout.
Thank you for being a part of Microsoft Fabric Community Forum!

If this post helps, please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

[v-pagayam-msft · Accepted solution]

Hi @mkjit256 ,
I just wanted to kindly follow up to see if you had a chance to review the previous response provided by me. I hope it was helpful. If yes, please Accept the answer so that it will be helpful to others to find it quickly.
If you have any queries please feel free to reachout.
Thank you for being part of Microsoft Fabric Community Forum!

[mkjit256]

Hello apparently it was something related to what you said. MS Provided a similar explanation for the issue which was related to the AAD token access failure on the user side.  Please check your AAD user token and confirm if this is expired.  

Please try a new AAD user token registration.

Thanks

[nilendraFabric]

Hi @mkjit256 

Try this and see if token is valid:

try:
token = mssparkutils.credentials.getToken(KUSTO_URI)
is_valid = mssparkutils.credentials.isValidToken(token)
print("Token is valid:", is_valid)
except:
print("Failed to validate token")

[mkjit256]

Thanks for your reply

i had verified the token and apparently my token is not valid, which i have no idea how will i renew it.

Thanks

[nilendraFabric]

Hi @mkjit256 

accessToken = mssparkutils.credentials.getToken("https://kusto.kusto.windows.net")
Note: Use "https://kusto.kusto.windows.net" instead of your specific KUSTO_URI

please give it a try .

[mkjit256]

That didn't work:
An error occurred while calling z:mssparkutils.credentials.getToken. : java.io.IOException: 500

my original KUSTO_URI: has a pattern string.string.kusto.fabric.microsoft.com that i got it from the CLUSTER_URI: