Forum Discussion

Dinzz's avatar
Dinzz
Frequent Visitor
9 months ago
Solved

Issue Writing to OneLake-Enabled KQL Database Delta Tables

I enabled OneLake availability for my KQL database so the tables are exposed as Delta or Parquet files. I’m able to read the table using Parquet without any issues:

df = spark.read.format("parquet").load(abfsspath)

However, when I try to overwrite or append data to the table, it fails:

append_df.write.format("delta").mode("append").save(abfsspath)

I receive the following error:

Operation failed: "Forbidden", 403, AuthorizationPermissionMismatch
"This request is not authorized to perform this operation using this permission."

 

I am the workspace admin, and I don’t want to ingest data using the conventional KQL ingestion methods because they are too slow for my use case. I specifically need to write records directly using Delta or Parquet. Could you help me understand how to enable this write operation for Delta tables?

 

KQL Queryset
Real-Time Dashboard
  • v-priyankata's avatar
    v-priyankata
    8 months ago

    Hi Dinzz 

    Thank you for reaching out to the Microsoft Fabric Forum Community.

    kustortininja Thanks for the inputs.

    Please try below code, it may helps you.


    kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

    accessToken = mssparkutils.credentials.getToken(kustoUri)

    append_df.write \
        .format("com.microsoft.kusto.spark.synapse.datasource") \
        .option("kustoCluster", kustoUri) \
        .option("kustoDatabase", database) \
        .option("kustoTable", tableName) \
        .option("accessToken", accessToken) \
        .option("ingestionMode", "Direct") \
        .option("tableCreation", "CreateIfNotExists") \
        .option("writeBatchTimeout", "00:10:00") \
        .option("batchSize", "1024") \
        .mode("append") \
        .save()

    If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

    Thanks.

     

  • v-priyankata's avatar
    v-priyankata
    7 months ago

    Hi Dinzz 

    That behavior is expected and not related to your code itself. The Kusto Spark connector (even in Direct ingestion mode) is optimized for batch ingestion, not for low-latency, row-by-row writes. Each Spark write triggers to authentication and token validation, Ingestion orchestration in KQL and Extent creation and commit on the KQL side

    Because of this, ingesting a single record can easily take 30–40 seconds, which is expected behavior.

    Currently, there is no supported way in Fabric to ingest single records into KQL with millisecond-level latency, and direct Delta writes are intentionally blocked.


    Thanks.

8 Replies

  • kustortininja's avatar
    kustortininja
    Microsoft Employee
    9 months ago

    Onelake availability endpoints are read-only. Eventhouse OneLake Availability - Microsoft Fabric | Microsoft Learn You cannot write back to the parquet table, you would need to write it as a new table in the Lakehouse. What do you mean "traditional ingestion methods are too slow for your use case". RTI is the fastest ingestion mechanism available in Fabric. If you are trying to refine and transform data in real-time, this wouldn't be a best practice. The lowest you can turn Onelake availability down to is 5 minutes, and even then it comes with a large warning about the small files problem in parquet. If you want to transform data on arrival you should look into Eventhouse Update Policies 

    • Dinzz's avatar
      Dinzz
      Frequent Visitor
      8 months ago

      hi kustortininja this is the code that iam using to ingest data from notebook 

       

      kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"
      accessToken = mssparkutils.credentials.getToken(kustoUri)

       

      append_df.write \
          .format("com.microsoft.kusto.spark.synapse.datasource") \
          .option("kustoCluster", kustoUri) \
          .option("kustoDatabase", database) \
          .option("kustoTable", tableName) \
          .option("accessToken", accessToken) \
          .option("ingestionMode", "Direct") \
          .mode("append") \
          .save()
      • v-priyankata's avatar
        v-priyankata
        Community Support
        8 months ago

        Hi Dinzz 

        Thank you for reaching out to the Microsoft Fabric Forum Community.

        kustortininja Thanks for the inputs.

        Please try below code, it may helps you.


        kustoUri = "https://trd-test_dummy.kusto.fabric.microsoft.com"

        accessToken = mssparkutils.credentials.getToken(kustoUri)

        append_df.write \
            .format("com.microsoft.kusto.spark.synapse.datasource") \
            .option("kustoCluster", kustoUri) \
            .option("kustoDatabase", database) \
            .option("kustoTable", tableName) \
            .option("accessToken", accessToken) \
            .option("ingestionMode", "Direct") \
            .option("tableCreation", "CreateIfNotExists") \
            .option("writeBatchTimeout", "00:10:00") \
            .option("batchSize", "1024") \
            .mode("append") \
            .save()

        If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

        Thanks.