Issue Writing to OneLake-Enabled KQL Database Delta Tables

Hi Dinzz 

Thank you for reaching out to the Microsoft Fabric Forum Community.

kustortininja Thanks for the inputs.

Please try below code, it may helps you.

kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

accessToken = mssparkutils.credentials.getToken(kustoUri)

append_df.write \
    .format("com.microsoft.kusto.spark.synapse.datasource") \
    .option("kustoCluster", kustoUri) \
    .option("kustoDatabase", database) \
    .option("kustoTable", tableName) \
    .option("accessToken", accessToken) \
    .option("ingestionMode", "Direct") \
    .option("tableCreation", "CreateIfNotExists") \
    .option("writeBatchTimeout", "00:10:00") \
    .option("batchSize", "1024") \
    .mode("append") \
    .save()

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Thanks.

Hi Dinzz 

That behavior is expected and not related to your code itself. The Kusto Spark connector (even in Direct ingestion mode) is optimized for batch ingestion, not for low-latency, row-by-row writes. Each Spark write triggers to authentication and token validation, Ingestion orchestration in KQL and Extent creation and commit on the KQL side

Because of this, ingesting a single record can easily take 30–40 seconds, which is expected behavior.

Currently, there is no supported way in Fabric to ingest single records into KQL with millisecond-level latency, and direct Delta writes are intentionally blocked.

Thanks.