Forum Discussion
Helper IIIPower BI Dataflow - response too large to return
Can't use direct connection to BQ for access reasonsChallenge that assumption. Tell them that 650 GB is too much.
If they don't budge, use Incremental Refresh. Start VERY small, with a day or so. See how many rows you get. Recommended partition size is around 8 to 20 million rows.
Please explain "import does not work" - are you getting an error message?
Direct Query on a dataflow = datamart = lipstick on a pig (Azure SQL db being the lipstick and the dataflow being the pig). Possible, sure, but a travesty nevertheless.
You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.
You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.
Helper IIIPlease help me understand this step. My aim was to get data for the past 90 days - so period being = 9 April to 9 July or so. The current dataflow has data for last 30 days meaning - 9 June to 9 july.
What am i supposed to be inputting on the incremental refresh because the refresh rows days can't be higher than the store rows days.
And if i am successful to even bring all the past 90 days somwhow, i still have to figure out a way to keep that as a rolling 90 day.
Thanks and i appreciate all your help.
Super UserSet it to refresh 1 day and store 10 days. Then evaluate the refresh log to see how many rows are in each partition. If there are substantially fewer than 8M rows per day then you can change your settings to 1 day and 3 months.
You will get the rolling 90 days automatically (plus some extra days to complete the months, as the actual month partitions will include not three but four months). You could change your setting to 1 day and 90 days but that would mean you would have 90 partitions. Only do that if necessary.