Subject: Understanding OneLake in Microsoft Fabric and Managing Data Copies
Hello Everyone,
I understand that OneLake in Fabric is designed to maintain a single copy of the data. However, I’m trying to fully grasp how this works in practice.
I have a real-life scenario:
I need to pull Dataverse data, implement the medallion architecture, and provide the final dataset to the reporting team. Here’s how I’ve structured it:
DataEngineer_Workspace
- Created a shortcut in the Bronze_Lakehouse (under the table section) to access Dataverse tables.
- Used a pipeline and notebook to cleanse, transform, and load the data into Silver_Warehouse.
- Applied business rules, aggregation, and mapping, then loaded the data into Gold_Warehouse via a pipeline.
Reporting_Workspace
- I attempted to create a shortcut to Gold_Warehouse from the Reporting Workspace, but there’s no direct option for this.
- I shared the Gold_Warehouse and granted permissions, but the reporting team can only read the data when directly accessing Gold_Warehouse. They cannot query it from Reporting_Warehouse.
- The only available option for them is to create a shortcut into a lakehouse, but this converts Gold_Warehouse into a lakehouse instead of a warehouse.
- Other options include Deployment Pipelines or Data Pipeline Copy Activity.
The Issue
Currently, I end up with three copies of the data:
- Silver_Warehouse
- Gold_Warehouse
- A duplicate of Gold_Warehouse in Reporting_Workspace
Question
How can I ensure that there is only a single copy of the data across the tenancy while allowing the reporting team to access and query it from their own workspace?
Any insights or best practices would be greatly appreciated.
Best regards,
Hemant Gosai
