Forum Discussion

JuliaMeneley's avatar
JuliaMeneley
Regular Visitor
2 months ago
Solved

Delta Table Maintenance for Mirrored DB Tables

Hello,   We have a Lakehouse set up with some regular delta tables as well as shortcuts to tables in a mirrored DB (using CDC from on-prem SQL server). I know there are delta table maintenance task...
  • Parchitect's avatar
    2 months ago

    Hi JuliaMeneley ,

    Short answer:
    I would not treat mirrored tables the same way as regular Lakehouse Delta tables for manual maintenance.

    For regular Lakehouse Delta tables, Fabric supports maintenance operations such as OPTIMIZE, V-Order and VACUUM.

    However, for mirrored tables, I would treat the mirrored output as Fabric-managed unless Microsoft confirms otherwise.

    Microsoft documentation for Lakehouse table maintenance:
    https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-table-maintenance

     

    Why this likely happens:
    Mirroring is a managed replication feature. Microsoft describes Mirroring as continuously replicating data into OneLake and automatically keeping the data in sync. That makes the mirrored table different from a normal Delta table that your Lakehouse owns and writes to directly.

    Microsoft documentation on Mirroring:
    https://learn.microsoft.com/en-us/fabric/mirroring/overview

     

    Also, if you are accessing the mirrored table through a Lakehouse shortcut, the shortcut is just a reference to data without copying it. So the table appears in the Lakehouse, but the underlying files are not physically owned by that Lakehouse.

    Microsoft documentation on Lakehouse shortcuts:
    https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-shortcuts

     

    Maintenance commands such as OPTIMIZE and VACUUM are not just read operations. OPTIMIZE compacts/reorganizes Delta files, and VACUUM removes obsolete files that are no longer referenced by the Delta log. So they require write/delete access to the underlying Delta storage.

    Your result also seems consistent with this:

    numFilesAdded: 0
    numFilesRemoved: 0
    totalFilesSkipped: 1

    That suggests Spark did not compact or rewrite the mirrored table files.

     

    What I would do:
    If you need full control over table maintenance, I would create a downstream curated Delta table:

    Mirrored table
    → read from it
    → write to your own Delta table in the Lakehouse
    → run OPTIMIZE / VORDER / VACUUM on that curated table

     

    Practical rule:

    • Regular Lakehouse Delta tables: maintain them yourself.
    • Mirrored tables / shortcuts: treat them as Fabric-managed or source-owned.
    • If you need custom optimization: create a curated Delta copy and maintain that.

     

    Best regards,
    Solutions Architect · Microsoft Fabric Specialist · Parchitect

    💡Did my response help you? Clicking Kudos is a small gesture that goes a long way, it encourages contributors and helps the community thrive!
    ✔️Did I answer your question? Please mark my post as a Solution, it helps others find the answer faster.