Forum Discussion
Create data model/dataflow over the large volume of data
Hello, I need help if it is possible somehow to build a PBI report over a really large volume of data.
Sources and requirements:
- I have a usecase to build reports over a delta table with data from heatpumps.
- That table with data is stored as a delta table in Azure blob storage and now I can read it as databrics managed table running on databrics sql warehouse or cluster. (I really don`t know what is a better way how to connect the data).
- Every day it comes around 3000 000 new rows into this table. This fact table is also partioned.
- Data in report visuals (linechart and table) should be visible at minute granularity.
- The requirement is to get new data to report every 15 minutes and visuals can refresh till 3-5 minutes.
I thought about hybrid fact table with last 3 years in import mode , incremental refresh and last day in direct query mode. But I found out that the limit for import mode is 2 bilion of rows
The first question is how to build such a model or what would be the best way how to do it?... maybe dataflow or use fabric?
The second question is what is the best and cheapest way in this use case to connect the data?
Thank you for all your ideas
1 Reply
None of this will be cheap. Depending on the compressibility of your data you are looking at a P4 (or higher) or equivalent F SKU.
I would recommend you engage with a Microsoft Certified Partner for a scoping and quoting exercise.
