Hi
I'm currently dealing with a large dataset.
My dataset consists of historical data for the past 5 years (2018, 2019, 2020, 2021, 2022) and one for the current year (2023)
Each year has almost 10 million rows.
What I did is that I created a dataflow for Historical data. I created a different table for reach historical year in this dataflow. I dont' refresh this dataflow. The reason for creating different table for each year is because Power BI online times out as it takes more than 5 hours to pull the data if I pull more than 1 year.
I created also one YTD dataflow where I can apply incremental refresh.
My planned next steps are:
1) Append all historical data together in one table.
2) Create new Dataflow where I will link the historical data and YTD data. Also, I'll append the who data (Historical + YTD).
My question is: Is this the right approach? Or is there a better way to achieve this with the best performance?
