Advocate IUltra Epic Idea: Allow Direct Delta Log & Parquet Access for Mirrored Databases
We (and I'm sure many others) need to be able to incrementally transform data from fabric mirrored databases into elevated tiers of medallion archetecture. We do not want to make any alterations at source to leverage a watermarking approach (that requires more work). If we could simply hit the delta log and files of the mirrored database (you can see them exposed via a shortcut from a lakehouse but can't actually interact with them from a notebook), we could efficiently and incrementally transform data into higher tiers. Please expose delta logs and parquet files of fabric mirrored databases! Wack and load approaches are not sustainable!
Thanks!
6 Comments
- This would be a huge benefit to our processes
Markpm_msftMicrosoft Employee
This already exists, you do have direct access to the underlying delta tables.
Some people create a seperate lakehouse and create shortcuts to the Mirrored database, but you dont need to.The only downside, you need to access via the ABFS path. You can look at the files (delta logs and parquet files, delete vectors,etc) using Azure storage explorer or Onelake data explorer.
- dbWizard
Markpm_msft- can you please direct me to any example of anyone leveraging the abfss path to a delta log of a mirrored fabric database to incrementally transform only what is new changed or removed into a silver layer with a notebook? Have you done or seen this personally? Also please see this post: https://community.fabric.microsoft.com/t5/Data-Engineering/Is-it-possible-yet-to-leverage-delta-log-of-a-mirrored-fabric/m-p/4719324/highlight/true#M9884
- Markpm_msft
I put together a demo using timetravel in delta, to get the differences between two different versions. (I would have to dig it out)
Delta Lake Time Travel | Delta Lake
What you really want is the delta change feed, which is on the roadmap. Planned for Q3.
But to answer your orginal question, below is how to find the path and use it in spark.
If you goto SQL Endpoint, find the table, look at the properties. You can get the ABFS path.
Create a notebook and just paste the ABFS path in.
df = spark.read.format("delta").load("<paste in ABFS path>")display(df) - SimplyWilsonStatus changed:NewtoAbandoned
This feature already exists as part of Onelake. Thus, closing this as decline.
