Inspired by this blog entry, I've been looking into using predefined Spark resource profiles: Supercharge your workloads: write-optimized default Spark configurations in Microsoft Fabric | Micro...
The use cases seem quite straightforward, and I don't see any reason not to use ReadHeavyForPBI for our Gold layer.
But, how do you decide between ReadHeavyForSpark or WriteHeavy for Bronze and Silver layers?
For Bronze and Silver tables that will end up as facts in our Gold layer, should you use WriteHeavy?
But for tables that will end up as slowly changing dimensions, would it be best to use ReadHeavyForSpark, as we will spend more time reading them than writing to them?
Has anyone measured any of these scenarios, and come up with recommendations?
A quick description of our architecture, for context:
- We are using a medallion architecture with Bronze, Silver, Gold Lakehouses, each in their own workspaces.
- We store Notebooks and Data pipelines in their own workspace that we call "process".
- We process fact and dimension data for multiple business areas.
- Volumes vary between hundreds of records, and 100M records per month, depending on the data source.
