Our largest fact table is > 15M rows, lives in an Azure SQL DB, and feeds into a Dataflow in a Premium Capacity Workspace.
To make development a bit easier, I'd like to create a version of the fact table that only has the last week's worth of data - a few hundred thousand rows at most. Then people could import that smaller table from the Dataflow, create their model, then when ready, switch to the main table in the Dataflow that has all the data.
I could:
Create a 2nd table in the dataflow, that's just the same query, but limited (via SQL) to the last week. But then say, the transformations of each table in the Dataflow, have to be kept in sync.
I was thinking instead:
Have a computed entity, that references the full table in the Dataflow, and then limits to the last week of data. That way it's always the exact same columns as the main Dataflow table.
But seemingly it's going to load all 16M rows into that computed entiry first - thus using up a lot of our capacity's processing.
So - is there a better way, or something I'm missing?
