Forum Discussion

dzfamy's avatar
dzfamy
Icon for Microsoft Employee rankMicrosoft Employee
17 hours ago

[Open Mirroring] Incremental CSV Upload Overwrites Existing File and Changes Are Not Replicated

Hi Team,

I'd like to report a possible issue with Microsoft Fabric Open Mirroring (manual CSV upload).

A customer reported that the initial CSV upload works, but subsequent uploads (UPDATE/INSERT) are not reflected in the mirrored table.

I was able to reproduce the same behavior in my own Fabric test environment.

Steps to reproduce:

  1. Create a new Open Mirrored Database.
  2. Upload 00000000000000000001.csv with 3 records and create a table with a primary key.
  3. Confirm that all 3 records are successfully replicated.
  4. Upload a second file, 00000000000000000002.csv, using Upload files → Update existing table.
  5. Test both UPDATE (__rowMarker__ = 1) and INSERT (__rowMarker__ = 0) operations.

Actual behavior:

  • Initial replication works correctly (3 rows).
  • Subsequent UPDATE/INSERT operations are not reflected.
  • Replication status remains Running, but the replicated row count stays at 3.
  • Workspace Monitoring logs show the initial Snapshotting and StartReplicating operations, but no subsequent incremental replication activity.
  • I checked the Landing Zone using Azure Storage Explorer and confirmed that the second CSV file was not created. Instead, the first CSV file (000...001.csv) was overwritten with the content of the second CSV, despite uploading a different filename (000...002.csv).
  • The same behavior was reproduced across 3 different tables.

Expected behavior:

The incremental CSV should be processed as a new change file, and the changes should be reflected in the mirrored table.

According to the Microsoft documentation, manual incremental uploads are supported:

https://learn.microsoft.com/en-us/fabric/mirroring/open-mirroring-tutorial

Questions:

  1. Is this a known issue with the Fabric portal's Upload files feature?
  2. Is there any additional configuration required for incremental CSV uploads?
  3. Is there a recommended workaround?

Any guidance from the Fabric Mirroring team would be greatly appreciated.

Thanks!

Mirroring

2 Replies

  • v-saisrao-msft's avatar
    v-saisrao-msft
    Icon for Community Support rankCommunity Support
    14 hours ago

    Hi dzfamy​,
    According to the Open Mirroring documentation, published files in the Landing Zone should have unique, immutable paths and should not be overwritten. Please verify that the incremental files follow the required sequential naming convention and that the configured file detection strategy is appropriate. As an alternative publishing approach, Microsoft recommends uploading files with temporary names and using the ADLS Gen2 rename operation to publish them under their final filenames. Also, note that overwriting an already processed file path does not trigger reprocessing.
    Open Mirroring Landing Zone Requirements and Formats - Microsoft Fabric | Microsoft Learn
    Best Practices for Open Mirroring in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

    Thank you.

    • dzfamy's avatar
      dzfamy
      Icon for Microsoft Employee rankMicrosoft Employee
      13 hours ago

      Hi v-saisrao-msft​ 

      Thanks for your response.
      The issue is with Fabric's built-in Upload files feature, not the file naming convention.

      When I upload a second CSV (000...002.csv) using Update existing table, Fabric overwrites the first CSV (000...001.csv) instead of creating a new file.

      I confirmed this in Azure Storage Explorer and reproduced it across three different tables. None of the incremental changes were replicated.

      According to Microsoft documentation, incremental uploads through the Fabric portal should work.

      Can you confirm whether this is a known issue or a bug in the Fabric portal?