topic Re: Writing to KQL db from notebook - Error in Data Engineering

Writing to KQL db from notebook - Error

VenDaFabricator — Fri, 19 Jul 2024 07:19:07 GMT

Am following this link => Use Fabric notebooks with data from a KQL database - Microsoft Fabric | Microsoft Learn to write directly from notebook to KQL db, but gettting below error;

Py4JJavaError: An error occurred while calling o4856.save. : kusto_connector_shaded.com.microsoft.azure.kusto.data.exceptions.DataServiceException: IOException when trying to retrieve cluster metadata:xxxxxxxx.kusto.windows.net

Here is my code:

from pyspark.sql import SparkSession from pyspark.sql.functions import col, lit, round from datetime import datetime # Welcome to your new notebook # Type here in the cell editor to add code! # The Kusto cluster uri to write the data. The query Uri is of the form https://<>.kusto.data.microsoft.com kustoUri = "https://xxxxxx.kusto.data.microsoft.com" # The database to write the data database = "BRDS" # The table to write the data table = "Metrics" # The access credentials for the write accessToken = mssparkutils.credentials.getToken(kustoUri) print(accessToken) # Generate a range of 5 rows with Id's 5 to 9 data = spark.range(5, 10).withColumnRenamed("id", "Id") # Add columns for ProcessName, CpuUsage, MemoryUsage, and Timestamp data = data.withColumn("ProcessName", lit("Process_") + col("Id").cast("string")) \ .withColumn("CpuUsage", round(lit(10.0) * col("Id") / 10, 2)) \ .withColumn("MemoryUsage", round(lit(20.0) * col("Id") / 10, 2)) \ .withColumn("Timestamp", lit(datetime.now().strftime('%Y-%m-%d %H:%M:%S'))) data.show() # Write data to a Kusto table data.write.\ format("com.microsoft.kusto.spark.synapse.datasource").\ option("kustoCluster",kustoUri).\ option("kustoDatabase",database).\ option("kustoTable", table).\ option("accessToken", accessToken ).\ option("tableCreateOptions", "CreateIfNotExist").mode("Append").save()

Any clues!

Re: Writing to KQL db from notebook - Error

Anonymous — Fri, 19 Jul 2024 09:03:57 GMT

Hi @VenDaFabricator ,

The error message indicates a problem retrieving cluster metadata from the Kusto database.

Consider the following to resolve the problem:

There seems to be a problem with your kustoUri:

After checking this, please verify that the access token was retrieved correctly and is valid. You have done a good job. this statement prints the token to ensure that it is not empty or expired.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Writing to KQL db from notebook - Error

VenDaFabricator — Fri, 19 Jul 2024 23:37:13 GMT

HI Yang,

No, there is no issue with URI, i changed it to 'xxxx' while posting in the forum, actual uri in code is from KQL db in Fabric, which was copied from (see below)

other thing is when you copy url from above:

clusteruri is "https:<KQLdatabaseURI>z1.kusto.fabric.microsoft.com", whereas the codebase (from MS) say's to specify clusteruri as "https://<yourKQLdatabaseURI>.z0.kusto.data.microsoft.com", but the error message says it can't find "<KQLdatabaseURI>.z1.kusto.windows.net"

Re: Writing to KQL db from notebook - Error

VenDaFabricator — Mon, 22 Jul 2024 07:51:31 GMT

Instead of using Ingest URI, when i used Query URI https://<clusterid>kusto.fabric.microsoft.com and it works. Documentation is bit mis-leading.

Re: Writing to KQL db from notebook - Error

Anonymous — Tue, 23 Jul 2024 07:09:48 GMT

Hi @VenDaFabricator ,

It seems that you have gotten a solution. Could you please mark the helpful post as Answered?

It will help the others in the community find the solution easily if they face the same problem as yours.

Thank you.

Best Regards,
Yang
Community Support Team