Forum Discussion
Solved
Shared dataset with incremental refresh growing rapidly
Dear community members, I have an issue regarding shared dataset with incremental refresh. Dataset contains 1 table with incremental refresh - set by date refreshing last month of data. The...
I mananged to solve the issue with modifying query for the table. Following two steps were removed:
#"Inserted Date" = Table.AddColumn(TableData, "Date", each DateTime.Date([DatetimeLocal]), type date), #"Inserted Hour" = Table.AddColumn(#"Inserted Date", "Hour", each Time.Hour([DatetimeLocal]), Int64.Type),Instead of power query, these are now calculated in DAX (after data is loaded from database). Hint for this change was warning message (shown on previous screenshot) saying Unable to confirm if the M query can be folded. Now the query is clean and dataset growth is smaller. Final query:
let Source = Sql.Databases(DB_Server), Table = Source{[Name=DB]}[Data], TableData = Table{[Schema="someschema",Item="sometable"]}[Data], #"Filtered Rows" = Table.SelectRows(TableData, each [DatetimeUtc] > RangeStart and [DatetimeUtc] <= RangeEnd) in #"Filtered Rows"
Thanks for your response v-xiaotang , though I did not find any mention why dataset size should grow so fast.
Table is set up like this:
History is set for 10 years, table actually contains data just for the last two years.
This is query for this table:
let
Source = Sql.Databases(DB_Server),
Table = Source{[Name=DB]}[Data],
TableData = Table{[Schema="someschema",Item="sometable"]}[Data],
#"Inserted Date" = Table.AddColumn(TableData, "Date", each DateTime.Date([DatetimeLocal]), type date),
#"Inserted Hour" = Table.AddColumn(#"Inserted Date", "Hour", each Time.Hour([DatetimeLocal]), Int64.Type),
#"Filtered Rows" = Table.SelectRows(#"Inserted Hour", each [DatetimeUtc] > RangeStart and [DatetimeUtc] <= RangeEnd)
in
#"Filtered Rows"
Why do you think this dataset grows by 5MB each day when it should grow just by 1MB?