Issue Writing to OneLake-Enabled KQL Database Delta Tables

[Dinzz]

I enabled OneLake availability for my KQL database so the tables are exposed as Delta or Parquet files. I’m able to read the table using Parquet without any issues:

df = spark.read.format("parquet").load(abfsspath)

However, when I try to overwrite or append data to the table, it fails:

append_df.write.format("delta").mode("append").save(abfsspath)

I receive the following error:

Operation failed: "Forbidden", 403, AuthorizationPermissionMismatch
"This request is not authorized to perform this operation using this permission."

I am the workspace admin, and I don’t want to ingest data using the conventional KQL ingestion methods because they are too slow for my use case. I specifically need to write records directly using Delta or Parquet. Could you help me understand how to enable this write operation for Delta tables?

[v-priyankata · Accepted solution]

Hi @Dinzz 

Thank you for reaching out to the Microsoft Fabric Forum Community.

@kustortininja Thanks for the inputs.

Please try below code, it may helps you.

kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

accessToken = mssparkutils.credentials.getToken(kustoUri)

append_df.write \
    .format("com.microsoft.kusto.spark.synapse.datasource") \
    .option("kustoCluster", kustoUri) \
    .option("kustoDatabase", database) \
    .option("kustoTable", tableName) \
    .option("accessToken", accessToken) \
    .option("ingestionMode", "Direct") \
    .option("tableCreation", "CreateIfNotExists") \
    .option("writeBatchTimeout", "00:10:00") \
    .option("batchSize", "1024") \
    .mode("append") \
    .save()

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Thanks.

[kustortininja]

Onelake availability endpoints are read-only. Eventhouse OneLake Availability - Microsoft Fabric | Microsoft Learn You cannot write back to the parquet table, you would need to write it as a new table in the Lakehouse. What do you mean "traditional ingestion methods are too slow for your use case". RTI is the fastest ingestion mechanism available in Fabric. If you are trying to refine and transform data in real-time, this wouldn't be a best practice. The lowest you can turn Onelake availability down to is 5 minutes, and even then it comes with a large warning about the small files problem in parquet. If you want to transform data on arrival you should look into Eventhouse Update Policies 

[Dinzz]

hi @kustortininja this is the code that iam using to ingest data from notebook 

kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"
accessToken = mssparkutils.credentials.getToken(kustoUri)

append_df.write \

    .format("com.microsoft.kusto.spark.synapse.datasource") \

    .option("kustoCluster", kustoUri) \

    .option("kustoDatabase", database) \

    .option("kustoTable", tableName) \

    .option("accessToken", accessToken) \

    .option("ingestionMode", "Direct") \

    .mode("append") \

    .save()

[v-priyankata · Accepted solution]

Hi @Dinzz 

Thank you for reaching out to the Microsoft Fabric Forum Community.

@kustortininja Thanks for the inputs.

Please try below code, it may helps you.

kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

accessToken = mssparkutils.credentials.getToken(kustoUri)

append_df.write \
    .format("com.microsoft.kusto.spark.synapse.datasource") \
    .option("kustoCluster", kustoUri) \
    .option("kustoDatabase", database) \
    .option("kustoTable", tableName) \
    .option("accessToken", accessToken) \
    .option("ingestionMode", "Direct") \
    .option("tableCreation", "CreateIfNotExists") \
    .option("writeBatchTimeout", "00:10:00") \
    .option("batchSize", "1024") \
    .mode("append") \
    .save()

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Thanks.

[Dinzz]

Hi @v-priyankata The code you shared is taking around 30–40 seconds to append a single record. Could you help me understand why this is taking so long? I’m looking to ingest the data as quickly as possible. Is there any alternative approach that would allow faster ingestion?

[v-priyankata · Accepted solution]

Hi @Dinzz 

That behavior is expected and not related to your code itself. The Kusto Spark connector (even in Direct ingestion mode) is optimized for batch ingestion, not for low-latency, row-by-row writes. Each Spark write triggers to authentication and token validation, Ingestion orchestration in KQL and Extent creation and commit on the KQL side

Because of this, ingesting a single record can easily take 30–40 seconds, which is expected behavior.

Currently, there is no supported way in Fabric to ingest single records into KQL with millisecond-level latency, and direct Delta writes are intentionally blocked.

Thanks.

[Dinzz]

hi @v-priyankata  thank you for the explanation it helped a lot

[v-priyankata]

Hi @Dinzz 

Thank you for reaching out to the Microsoft Fabric Forum Community.

I hope the information provided was helpful. If you still have questions, please don't hesitate to reach out to the community.

[v-priyankata]

Hi @Dinzz 

Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.