Forum Discussion
Helper IIIPower BI Dataflow - response too large to return
Can't use direct connection to BQ for access reasonsChallenge that assumption. Tell them that 650 GB is too much.
If they don't budge, use Incremental Refresh. Start VERY small, with a day or so. See how many rows you get. Recommended partition size is around 8 to 20 million rows.
Please explain "import does not work" - are you getting an error message?
Direct Query on a dataflow = datamart = lipstick on a pig (Azure SQL db being the lipstick and the dataflow being the pig). Possible, sure, but a travesty nevertheless.
You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.
You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.
Helper IIIOkay, i just ran the refresh for store 1 day and refresh last 3 months. Also just from the dataset info on BQ there's an avg of about 4M rows per day.
So this should potentially have all the data for the past 90 days and more at the moment. Am just waiting for it to finish the run.
Once this is successful, i'll have to change the refresh schedule to be store x months and refresh 1 day, that way i get the latest day's data added, right?
Only concern here - the new data keeps on getting added with daily refresh but there is historic data present which as days pass will be much much more than 90 days. Will i run into storage/processing issues at that point?
Is there a way of automatic data pruning to get rid of historic data past 90 days?
Thanks!
Super UserWill i run into storage/processing issues at that point?
Is there a way of automatic data pruning to get rid of historic data past 90 days?
keep "x" at 3 and the Power BI service will do the pruning for you automatically.