Forum Discussion
Data refresh size makes no sense
Hello
Problem discription:
I have made a dataset with an active and a passive load of a very large table. The point of this is to reduce the refresh data load time and load quantity. I have used the technique described in this youtube video. (timestamp: 36:10)
The issue i'm facing, and i haven't been able to figure out why this happens, is that when i refresh the active table, which only contains 3000 rows and 9 columns still refreshes 200 MB of data. If i refresh the passive table, which contains 200.000+ rows, the data refresh is also around 200 MB.
How is the model built:
In power query i load the source table, which is called "Finansposter". Finansposter is a very large table, so that's why i want to split it to a passive and an active table. The Advanced Editor code for Finansposter is: (the source has been removed due to confidentiality, and the source is not important here)
let
Source = OData.Feed("xxxxx')/Finansposter? &$filter= Posting_Date gt datetime'2020-09-30T00:00:00Z' and G_L_Account_No le '26010' and G_L_Account_No ge '11010'", null, [Implementation="2.0"]),
#"Filtered Rows1" = Table.SelectRows(Source, each Text.Contains([G_L_Account_No], "110") or Text.Contains([G_L_Account_No], "210") or Text.Contains([G_L_Account_No], "26010")),
#"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows1",{{"Posting_Date", type date}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"Document_Type", "Document_No", "Description", "Job_No", "IC_Partner_Code", "Gen_Posting_Type", "Gen_Bus_Posting_Group", "Gen_Prod_Posting_Group", "Quantity", "Additional_Currency_Amount", "VAT_Amount", "Bal_Account_Type", "Bal_Account_No", "User_ID", "Reason_Code", "Reversed", "Reversed_by_Entry_No", "Reversed_Entry_No", "FA_Entry_Type", "FA_Entry_No", "Person_Hours", "Dim_Corrected", "ETag"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"G_L_Account_No", type text}, {"Amount", type number}, {"Dimension_Set_ID", type text}, {"Global_Dimension_2_Code", type text}, {"Global_Dimension_1_Code", type text}, {"Entry_No", type text}, {"G_L_Account_Name", type text}, {"Source_Code", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type1", each [Posting_Date] >= StartDate),
#"Changed Type2" = Table.TransformColumnTypes(#"Filtered Rows",{{"G_L_Account_No", Int64.Type}, {"Entry_No", Int64.Type}, {"Global_Dimension_1_Code", Int64.Type}, {"Dimension_Set_ID", Int64.Type}})
in
#"Changed Type2"
This table has disabled load, so it is not included in the report.
From here two other queries are created: Finansposter (Active) and Finansposer (Static)
Two parameters has been created:
StartDate: 01-10-2019
CutOffDate: 28-06-2021
The advanced editor code for Finansposter (Active) is:
let
Source = Finansposter,
#"Filtered Rows" = Table.SelectRows(Source, each [Posting_Date] >= CutOffDate)
in
#"Filtered Rows"
The advanced editor code for Finansposter (Static) is:
let
Source = Finansposter,
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Posting_Date", type date}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each [Posting_Date] < CutOffDate)
in
#"Filtered Rows"
So the active table only loads rows with a date after or equal to the CutoffDate.
The static table is not included in the report refresh.
The output works, as in the static table shows rows between the startdate and the cutoffdate, and the active table only shows rows after the cutoffdate.
When the queries are loaded into the report, the static and the active table are combined using DAX.
Question:
Can anyone explain to me why the active table has such a large load quantity, when it is only 3000 rows?
Can i change the queires in any way to change this?
Because if the active and static tables have the same load time and quantity, the whole purpose of making passive and active tables is defeated.
Thank you in advance to anyone who can help me with this issue.
2 Replies
- v-angzheng-msft
Community Support
Hi, Anonymous
Please make sure that your 2 parameters are created correctly, they are case sensitive and cannot be changed. RangeStart and RangeEnd.
Please refer to the link below for more information
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-configure
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-troubleshoot
Best Regards,
Community Support Team _ Zeon Zheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.- AnonymousNot applicable
Hi
Thank you for your reply. However, i am not using the incremental refresh feature.
With the model i have created, power bi should only load the data after the CutOffDate (which is the result that i get) when i press refresh data, but i do not understand why it still takes so long time to refresh the data, and why it is refreshing such a large amount of data. So i am looking to find out more about how power bi reads the queries i have made and if i can change my query so it doesn't load such large quantities. Because in theory this model should work, and it outputs the desired result, its just the amount of data it has to load to produce this output that baffles me, since the output is just a couple of thousand rows.