Forum Discussion
Power BI Dataflow - response too large to return
Can't use direct connection to BQ for access reasonsChallenge that assumption. Tell them that 650 GB is too much.
If they don't budge, use Incremental Refresh. Start VERY small, with a day or so. See how many rows you get. Recommended partition size is around 8 to 20 million rows.
Please explain "import does not work" - are you getting an error message?
Direct Query on a dataflow = datamart = lipstick on a pig (Azure SQL db being the lipstick and the dataflow being the pig). Possible, sure, but a travesty nevertheless.
You can consider switching the incremental refresh to the dataset instead and to completely remove the dataflow from the equation.
You can consider dataflow Gen2 and store the results as Delta Lake in Fabric.
Sam_Jain Sorry I'm not fully sure what and how your current data movement process looks like. If data has already landed in the lakehouse, you should have many options for providing rolling 90 days of data to end users. You can create a view on the SQL endpoint of the lakehouse with a 90-day rolling method and let the user use that view in their model, not sure why you want to have another dataflow on top of lakehouse. You have a lot more options available if data is already in the lakehouse.
Oh i wasn't planning on a lake house. I just looked up Gen2 Dataflow and the method it suggested me was to create a lakehouse and then you will find the GEN2 dataflow option.
If there is a different method, i'd like to try that.
And the answer to your first question so basically the data itself is housed in Big Query and the original dataset runs on a daily referesh and has a data size of about 650 GB and the solution here was to create a subset so i created a new table with just the last 90 days of data and a stored procedure alongside with it to have the oldest day's data deleted and new day's data added from the original table which also runs on a daily basis to maintian that 90 day rolling window.
My only goal here is to somehow provide this data without actually giving direct access to BQ and for the team to be able to use this data on their own datasets/reports as just another source.
Thanks