Re: Expand Dates in Large Data Set - PowerQuery an...

lerecie · ‎03-29-2021

I have a large data set in a proper star schema. However, i'm struggling with some reports out of it.

I'll explain what i'm currently doing, and hopefully someone can tell me a better way (or more elegant) way to accomplish something similar.

Data set:

There's many more tables that have proper relationships (Aprox 30 more), but for the purpose of this exercise, that's all we need.

Most of the work happens in the ResourceActualDetail table it looks like this (with some columns deleted):

ResourceAllocationId	AllocationPercentage	EndDate	StartDate
794	1	4/1/2018 0:00	2/25/2018 0:00
795	1.15	3/4/2018 0:00	2/25/2018 0:00
795	1.0875	3/11/2018 0:00	3/4/2018 0:00
795	1	4/1/2018 0:00	3/11/2018 0:00
796	1.05	3/4/2018 0:00	2/25/2018 0:00
797	0.725	3/4/2018 0:00	2/25/2018 0:00
1531	1	3/11/2018 0:00	2/25/2018 0:00
1721	1	4/1/2018 0:00	2/25/2018 0:00

Desired Output:

ResourceAllocationId	AllocationPercentage	EndDate	StartDate	Date
794	1	4/1/2018 0:00	2/25/2018 0:00	2/26/2018 0:00
794	1	4/1/2018 0:00	2/25/2018 0:00	3/4/2018 0:00
794	1	4/1/2018 0:00	2/25/2018 0:00	…
794	1	4/1/2018 0:00	2/25/2018 0:00	4/1/2018 0:00

Notice the 7 day increments.

And the same for every ResourceAllocationID so that I can graph it like so:

(error in Graph dates, should be with 7 day increments).

Where there is a different line for each Resource (connected through relationships to ResourceAllocation and ResourceActualDetail). Even if we can get it to a point where we can do that for each ResourceAllocationId I can figure out the rest 🙂

Current Approach:

These data tables are quite large already(~1M rows) , and my approach only has bloated that.

Trying to do everything as close to the source, this what I came up with:

Let

Source=PowerBI.Dataflows(null),

...

#"Changed Type" = Table.TransformColumnTypes(ResourceActualDetail1,{{"EndDate", type number}, {"StartDate", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Dates", each List.Numbers([StartDate],([EndDate]-[StartDate])/7,7)),
#"Expanded Dates" = Table.ExpandListColumn(#"Added Custom", "Dates"),
#"Changed Type1" = Table.TransformColumnTypes(#"Expanded Dates",{{"Dates", type datetime}, {"EndDate", type datetime}, {"StartDate", type datetime}})
in
#"Changed Type1"

Essentially, I change the StartDate and EndDate fields to numeric Values and then I calculate the beginning of each week before i switch back all those values to Date type.

This part of the query doesn't fold, so I cannot take the advantage of the PowerFlow to do it. It also makes each refresh very slow (should be refresed multiple times a day).

Any suggestions or ideas?? Obviously looking for the most efficient way to achieve this so that it won't take many resources (I do have a premium instance).

Thanks!

lerecie · ‎03-30-2021

I have tried both approaches and there's something that is not quite working.

I have focused in using your Value by End of the Month formula, since it looks the most complete for what I am trying to do, but it doesn't work.. i don't get the continuos dates and get breaks in the data still.