Writing to KQL db from notebook - Error

[VenDaFabricator]

Am following this link => Use Fabric notebooks with data from a KQL database - Microsoft Fabric | Microsoft Learn to write directly from notebook to KQL db, but gettting below error;

Py4JJavaError: An error occurred while calling o4856.save. : kusto_connector_shaded.com.microsoft.azure.kusto.data.exceptions.DataServiceException: IOException when trying to retrieve cluster metadata:xxxxxxxx.kusto.windows.net

Here is my code:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, lit, round
from datetime import datetime

# Welcome to your new notebook
# Type here in the cell editor to add code!
# The Kusto cluster uri to write the data. The query Uri is of the form https://<>.kusto.data.microsoft.com 
kustoUri = "https://xxxxxx.kusto.data.microsoft.com"
# The database to write the data
database = "BRDS"
# The table to write the data 
table    = "Metrics"
# The access credentials for the write
accessToken = mssparkutils.credentials.getToken(kustoUri)
print(accessToken)
# Generate a range of 5 rows with Id's 5 to 9
data = spark.range(5, 10).withColumnRenamed("id", "Id")

# Add columns for ProcessName, CpuUsage, MemoryUsage, and Timestamp
data = data.withColumn("ProcessName", lit("Process_") + col("Id").cast("string")) \
           .withColumn("CpuUsage", round(lit(10.0) * col("Id") / 10, 2)) \
           .withColumn("MemoryUsage", round(lit(20.0) * col("Id") / 10, 2)) \
           .withColumn("Timestamp", lit(datetime.now().strftime('%Y-%m-%d %H:%M:%S')))
data.show()
# Write data to a Kusto table
data.write.\
    format("com.microsoft.kusto.spark.synapse.datasource").\
    option("kustoCluster",kustoUri).\
    option("kustoDatabase",database).\
    option("kustoTable", table).\
    option("accessToken", accessToken ).\
    option("tableCreateOptions", "CreateIfNotExist").mode("Append").save()

Any clues! 

[VenDaFabricator]

HI Yang, 

No, there is no issue with URI, i changed it to 'xxxx' while posting in the forum, actual uri in code is from KQL db in Fabric, which was copied from (see below)

other thing is when you copy url from above:

clusteruri is "https:<KQLdatabaseURI>z1.kusto.fabric.microsoft.com", whereas the codebase (from MS) say's  to specify clusteruri as "https://<yourKQLdatabaseURI>.z0.kusto.data.microsoft.com", but the error message says it can't find "<KQLdatabaseURI>.z1.kusto.windows.net"

[VenDaFabricator · Accepted solution]

Instead of using Ingest URI, when i used Query URI https://<clusterid>kusto.fabric.microsoft.com and it works.  Documentation is bit mis-leading.

[Anonymous]

Hi @VenDaFabricator ,

It seems that you have gotten a solution. Could you please mark the helpful post as Answered?

It will help the others in the community find the solution easily if they face the same problem as yours.

Thank you.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[Anonymous]

Hi @VenDaFabricator ,

The error message indicates a problem retrieving cluster metadata from the Kusto database.

Consider the following to resolve the problem:

There seems to be a problem with your kustoUri:

After checking this, please verify that the access token was retrieved correctly and is valid. You have done a good job. this statement prints the token to ensure that it is not empty or expired.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!