Forum Discussion

Sam_Jain's avatar
Sam_Jain
Icon for Helper III rankHelper III
2 years ago
Solved

Power BI Dataflow - response too large to return

Hello Community,   I have run into a PowerBI dataflow issue. My source table is in Big Query and is a table that gets updated daily and has data for the last 90 days in a rolling fashion.   What ...
dataflow
Need Help
Tips and Tricks
  • lbendlin's avatar
    lbendlin
    2 years ago
    Can't use direct connection to BQ for access reasons

    Challenge that assumption.  Tell them that 650 GB is too much.

     

    If they don't budge, use Incremental Refresh. Start VERY small, with a day or so.  See how many rows you get.  Recommended partition size is around 8 to 20 million rows.

  • lbendlin's avatar
    lbendlin
    2 years ago

    Please explain "import does not work"  - are you getting an error message?

     

    Direct Query on a dataflow = datamart = lipstick on a pig  (Azure SQL db being the lipstick and the dataflow being the pig).  Possible, sure,  but a travesty nevertheless.

     

    You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.

     

    You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.

lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
2 years ago

Please explain "import does not work"  - are you getting an error message?

 

Direct Query on a dataflow = datamart = lipstick on a pig  (Azure SQL db being the lipstick and the dataflow being the pig).  Possible, sure,  but a travesty nevertheless.

 

You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.

 

You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.

Sam_Jain's avatar
Sam_Jain
Icon for Helper III rankHelper III
2 years ago

So, i think the dataflow may work - i connected the PowerBI report to the dataflow and had it imported and refreshed- sits at about 4.5 GB of pbix.
However, the only issue now is the data that is flowing in on that is not of the last 90 days.
So ideally it should start with 05/14/2024 and up to 08/12/2024 and that's exactly how it is on the source as well as dataflow. But the moment i bring it into the report the data now starts from 05/01/2024 and ends at 07/08/2024.
I suspect it's because i kept the incremental refresha t 3 months and 1 day and should switch to 90 days and 1 day.
I can't think of any other reasons why the data is falling short.
There are no other filters applied on the report, dataflow has the auto appiled parameters for incremental refresh and the source has data for last 90 days.