It was nice to see Microsoft add an out of the box mirror solution for the SAP Datasphere replication flow output. However, there are some limitations and issue I have found. Each entity that is replicated from SAP Datasphere to Azure Gen2 Storage produces a Top Fold matching the table name and sub-folders of initial and delta. In the initial folder, SAP generates a _SUCCESS file indicating that the initial load is completed and all parquet files are now ready for consumption. The time-stamp on the file will not change unless the entitie's replication is restarted in SAP Datasphere. The mirror db solution needs to keep track of this file and automaticaly reset the individual entity if the _SUCCESS file is modified or recreated. This should trigger the mirror db to "restart" replication on the entity. The restart process shoud delete all records in the target delta lake table and reload everything again. This bring up another point in that their is no way to restart replication on a single entity within the mirror database management interface. Their are times where I need to restart just one entity and not all of them.
on:
Add a Git-tracked “resources” folder to Lakehouses
on:
Add native Dev/Test/UAT/Prod semantic model compar...
on:
Computed Columns in Fabric Data Warehouse
LiLiu_Fabric
on:
Deployment Pipeline Run in Fabric monitor