Forum Discussion
Incremental refresh on Data Flow failing with Error: PipelineException: Access is denied
HI Anonymous,
I think the buffer functions may suitable for your scenario but I'm not so sure if it has enough permission to execute on the power bi premium server.
How and When to use List & Table Buffer?
For your requirement, you can take a look at the following steps if they works.
Steps:
1, Create a static table to store large amounts of old records and use on-demand refresh instead of configuring the scheduler refresh on this data source.
2, Create a dynamic table loading less amount of records, you need to exclude the ranges of first table records to reduce the processing data amount. (you can configure incremental refresh on it)
3, Add a custom query table that merges the above two tables records without any other advanced operations.Regards,
Xiaoxin Sheng
HI Anonymous,
I think the buffer functions may suitable for your scenario but I'm not so sure if it has enough permission to execute on the power bi premium server.
How and When to use List & Table Buffer?
For your requirement, you can take a look at the following steps if they works.
Steps:
1, Create a static table to store large amounts of old records and use on-demand refresh instead of configuring the scheduler refresh on this data source.
2, Create a dynamic table loading less amount of records, you need to exclude the ranges of first table records to reduce the processing data amount. (you can configure incremental refresh on it)
3, Add a custom query table that merges the above two tables records without any other advanced operations.
Regards,
Xiaoxin Sheng
However, I had to create 30 staging tables as the size of data was extremely large. I wanted to avoid taking this approach (and hence thought of breaking up the tables by BU/ dates. .etc).
Not an ideal solution.. but it worked for me!!