Forum Discussion

Dinzz's avatar
Dinzz
Frequent Visitor
9 months ago
Solved

Issue Writing to OneLake-Enabled KQL Database Delta Tables

I enabled OneLake availability for my KQL database so the tables are exposed as Delta or Parquet files. I’m able to read the table using Parquet without any issues: df = spark.read.format("parquet")...
KQL Queryset
Real-Time Dashboard
  • v-priyankata's avatar
    v-priyankata
    8 months ago

    Hi Dinzz 

    Thank you for reaching out to the Microsoft Fabric Forum Community.

    kustortininja Thanks for the inputs.

    Please try below code, it may helps you.


    kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

    accessToken = mssparkutils.credentials.getToken(kustoUri)

    append_df.write \
        .format("com.microsoft.kusto.spark.synapse.datasource") \
        .option("kustoCluster", kustoUri) \
        .option("kustoDatabase", database) \
        .option("kustoTable", tableName) \
        .option("accessToken", accessToken) \
        .option("ingestionMode", "Direct") \
        .option("tableCreation", "CreateIfNotExists") \
        .option("writeBatchTimeout", "00:10:00") \
        .option("batchSize", "1024") \
        .mode("append") \
        .save()

    If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

    Thanks.

     

  • v-priyankata's avatar
    v-priyankata
    7 months ago

    Hi Dinzz 

    That behavior is expected and not related to your code itself. The Kusto Spark connector (even in Direct ingestion mode) is optimized for batch ingestion, not for low-latency, row-by-row writes. Each Spark write triggers to authentication and token validation, Ingestion orchestration in KQL and Extent creation and commit on the KQL side

    Because of this, ingesting a single record can easily take 30–40 seconds, which is expected behavior.

    Currently, there is no supported way in Fabric to ingest single records into KQL with millisecond-level latency, and direct Delta writes are intentionally blocked.


    Thanks.

Dinzz's avatar
Dinzz
Frequent Visitor
8 months ago

Hi v-priyankata The code you shared is taking around 30–40 seconds to append a single record. Could you help me understand why this is taking so long? I’m looking to ingest the data as quickly as possible. Is there any alternative approach that would allow faster ingestion?

v-priyankata's avatar
v-priyankata
Community Support
7 months ago

Hi Dinzz 

That behavior is expected and not related to your code itself. The Kusto Spark connector (even in Direct ingestion mode) is optimized for batch ingestion, not for low-latency, row-by-row writes. Each Spark write triggers to authentication and token validation, Ingestion orchestration in KQL and Extent creation and commit on the KQL side

Because of this, ingesting a single record can easily take 30–40 seconds, which is expected behavior.

Currently, there is no supported way in Fabric to ingest single records into KQL with millisecond-level latency, and direct Delta writes are intentionally blocked.


Thanks.