Forum Discussion
How do I organize everything?
There are a lot of factors we found to consider when moving to Fabric structures. For example, if you are doing any sort of GIT control on your environment, GIT doesn't always play nice with lakehouse / warehouse objects. It's destructive and changes to schema can result in dropping and recreating an entire lakehouse / warehouse with data loss.
We do have GIT versioning, so we elected to NOT use git versioning on any data elements.
Secondly we found some strong challenges with choosing between lakehouse and warehouse. While Warehouse allowed us to use traditional SQL procedures, they don't work great with Notebooks. YES, you can use a JDBC driver to connect spark to a SQL endpoint, but it's not as out-of-the-box as the defaultLakehouse experience and you get non of the "onelake" benefits. Lakehouses conversely don't have "stored procedures". So while a hybrid solution is possible, we found it to be functionally more of a fork in the road. We chose lakehouse + notebooks since it's WAAAY more performant, and lakehouse has much more options (though you are sacrificing VARCHAR(MAX).
The setup:
Workspace 1 (Bronze):
Lakehouses for each source system
Workspace 2 (Silver):
Single primary lakehouse to be the central "data warehouse"
Anciliary lakehouse for things like static files, materialized views used as "staging" for ingestion to the primary lakehouse, etc.
Workspace 3 (Gold):
We elected to do all of our gold layer as semantic models. Use PBIR format and it GIT versions well (be prepared for a lot of JSON)
Workspace 4 (Orchestration):
This is where we put all of our pipelines, notebooks, and 1 environment. These elements GIT version well and there is no risk of data loss. We have subfolders in this workspace named for the workspaces they affect. So a folder for Workspace 1, a folder for workspace 2, and we have a special _schedules folder for our master pipelines that are scheduled.
I'm not saying this is the "right" solution. It's "A" Solution that we have found to work well. It's been quite the journey along the way and we've found soooo many gatchas. Like environments for example. Something they don't make obvious is that if you create your own whl file and slap it in an environment you "can" do this, but at the cost of utilizing pre-warmed spark nodes. We found jobs that took less than 5 minutes to run would spend 35 minutes just spinning up a custom spark node to run. Now we avoid Fabric Environments like the plague.