Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago

Data refresh size makes no sense

Hello

 

Problem discription:

I have made a dataset with an active and a passive load of a very large table. The point of this is to reduce the refresh data load time and load quantity. I have used the technique described in this youtube video. (timestamp: 36:10)

 

The issue i'm facing, and i haven't been able to figure out why this happens, is that when i refresh the active table, which only contains 3000 rows and 9 columns still refreshes 200 MB of data. If i refresh the passive table, which contains 200.000+ rows, the data refresh is also around 200 MB. 

How is the model built:

In power query i load the source table, which is called "Finansposter". Finansposter is a very large table, so that's why i want to split it to a passive and an active table. The Advanced Editor code for Finansposter is: (the source has been removed due to confidentiality, and the source is not important here)

 

let

Source = OData.Feed("xxxxx')/Finansposter? &$filter= Posting_Date gt datetime'2020-09-30T00:00:00Z' and G_L_Account_No le '26010' and G_L_Account_No ge '11010'", null, [Implementation="2.0"]),
#"Filtered Rows1" = Table.SelectRows(Source, each Text.Contains([G_L_Account_No], "110") or Text.Contains([G_L_Account_No], "210") or Text.Contains([G_L_Account_No], "26010")),
#"Changed Type" = Table.TransformColumnTypes(#"Filtered Rows1",{{"Posting_Date", type date}}),
#"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"Document_Type", "Document_No", "Description", "Job_No", "IC_Partner_Code", "Gen_Posting_Type", "Gen_Bus_Posting_Group", "Gen_Prod_Posting_Group", "Quantity", "Additional_Currency_Amount", "VAT_Amount", "Bal_Account_Type", "Bal_Account_No", "User_ID", "Reason_Code", "Reversed", "Reversed_by_Entry_No", "Reversed_Entry_No", "FA_Entry_Type", "FA_Entry_No", "Person_Hours", "Dim_Corrected", "ETag"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"G_L_Account_No", type text}, {"Amount", type number}, {"Dimension_Set_ID", type text}, {"Global_Dimension_2_Code", type text}, {"Global_Dimension_1_Code", type text}, {"Entry_No", type text}, {"G_L_Account_Name", type text}, {"Source_Code", type text}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type1", each [Posting_Date] >= StartDate),
#"Changed Type2" = Table.TransformColumnTypes(#"Filtered Rows",{{"G_L_Account_No", Int64.Type}, {"Entry_No", Int64.Type}, {"Global_Dimension_1_Code", Int64.Type}, {"Dimension_Set_ID", Int64.Type}})

in

#"Changed Type2"

 

This table has disabled load, so it is not included in the report. 

 

From here two other queries are created: Finansposter (Active) and Finansposer (Static)

 

Two parameters has been created: 

StartDate: 01-10-2019

CutOffDate: 28-06-2021

 

The advanced editor code for Finansposter (Active) is:

let

Source = Finansposter,
#"Filtered Rows" = Table.SelectRows(Source, each [Posting_Date] >= CutOffDate)

in

#"Filtered Rows"

 

The advanced editor code for Finansposter (Static) is:

let

Source = Finansposter,
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Posting_Date", type date}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each [Posting_Date] < CutOffDate)

in

#"Filtered Rows"

 

So the active table only loads rows with a date after or equal to the CutoffDate. 

The static table is not included in the report refresh. 

 

The output works, as in the static table shows rows between the startdate and the cutoffdate, and the active table only shows rows after the cutoffdate. 

 

When the queries are loaded into the report, the static and the active table are combined using DAX. 

 

Question:

Can anyone explain to me why the active table has such a large load quantity, when it is only 3000 rows?

Can i change the queires in any way to change this?

Because if the active and static tables have the same load time and quantity, the whole purpose of making passive and active tables is defeated. 

 

Thank you in advance to anyone who can help me with this issue. 

2 Replies