Forum Discussion

Sam_Jain's avatar
Sam_Jain
Helper III
2 years ago
Solved

Power BI Dataflow - response too large to return

Hello Community,   I have run into a PowerBI dataflow issue. My source table is in Big Query and is a table that gets updated daily and has data for the last 90 days in a rolling fashion.   What ...
Dataflow
Need Help
Tips and Tricks
  • lbendlin's avatar
    lbendlin
    2 years ago
    Can't use direct connection to BQ for access reasons

    Challenge that assumption.  Tell them that 650 GB is too much.

     

    If they don't budge, use Incremental Refresh. Start VERY small, with a day or so.  See how many rows you get.  Recommended partition size is around 8 to 20 million rows.

  • lbendlin's avatar
    lbendlin
    2 years ago

    Please explain "import does not work"  - are you getting an error message?

     

    Direct Query on a dataflow = datamart = lipstick on a pig  (Azure SQL db being the lipstick and the dataflow being the pig).  Possible, sure,  but a travesty nevertheless.

     

    You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.

     

    You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.

SaiTejaTalasila's avatar
SaiTejaTalasila
Super User
2 years ago

Hi Sam_Jain ,

 

Let's assume If you were able to bring your 90days to dataflow 1 and on the dataflow 2 apply incremental refresh and append 90+new data on it or you can take dataflow 3 and append that dataflow 1 and dataflow 2 on it.

Or 

Option 2 using Fabric 

You can try this bring your data to datalake and save it as a parquet file.

You can refer it and you can try it-

Loading data from Google Big Query into Microsoft Fabric using Python Notebooks: A Simple Guide
https://www.linkedin.com/pulse/loading-data-from-google-big-query-microsoft-fabric-using-jakub-vala-1nlue?utm_source=share&utm_medium=member_android&utm_campaign=share_via

 

I hope it will be helpful 

 

Thanks,

Sai Teja 

  • Sam_Jain's avatar
    Sam_Jain
    Helper III
    2 years ago

    Hello SaiTejaTalasila 
    Thanks for your repsonse. Both are good approaches, only challenge is i need to maintain that 90 day rolling window. And if i have a consolidated dataflow my source would be the other two dataflows and i would have to somehow make them all run on an incremental refresh.

     

    I figured out the solution with another member's help. Start small and then use incremental refresh!

     

    Thanks again for your input.