Forum Discussion

cassidy's avatar
cassidy
Power Participant
6 years ago
Solved

Fill in rows between dates

I've been trying to figure out this puzzle in power query for a week or so, not finding any examples that apply to my situation.  

 

Our database only records the change in price of a product on the day it happens, which is great, but I also need a historical output with every date for at least the last year to see change over time.  Final output would be every product, every date, with appropriate values. 

 

Looks like the table below.  For example, Line 1 (orange), I need additional rows below it repeating the same values until it gets to Line 2 (blue). 

 

So far, I have been able to merge in a full calendar to create new lines with null values and then use Fill Down to copy the values down.  This is almost perfect.  Problem is that when the product changes ( Line 5 & 6 / red & green  ), I end up filling down the prior products values in as the new products start date values.  Only when I get to the new products first real value does it correct itself.

 

Thanks for your help

 

ProductMSRPCurrent_PriceTypeDate
15233-1287272Full Price3/4/2020
15233-1285858Full Price2/27/2020
15233-1287272Full Price2/27/2020
15233-1287272Full Price2/20/2020
15233-1285858Full Price1/2/2020
15233-EH27272Full Price3/4/2020
15233-EH27272Full Price2/27/2020
15233-EH27272Full Price2/20/2020
15233-EH25858Full Price1/31/2020
  • See if this works. I think it does, but not 100% sure I've properly accounted for the duplicate dates you have, like Feb 27.

     

    Paste this into a blank query:

    1) In Power Query, select New Source, then Blank Query
    2) On the Home ribbon, select "Advanced Editor" button
    3) Remove everything you see, then paste the M code I've given you in that box.
    4) Press Done

     

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQ1MjbWNTSyUNJRMjeCEW6lOTkKAUWZyalAjrG+ib6RgZGBUqwOqnpTOIGi3kjfyBy7BhwWkKPBgCQnGeoboat39TAiyct41WPzASENGD6AaMDpA2NDqIZYAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Product = _t, MSRP = _t, Current_Price = _t, Type = _t, Date = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Product", type text}, {"MSRP", Int64.Type}, {"Current_Price", Int64.Type}, {"Type", type text}, {"Date", type date}}),
        #"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
        #"Added Index1" = Table.AddIndexColumn(#"Added Index", "Index.1", 1, 1),
        #"Merged Queries" = Table.NestedJoin(#"Added Index1", {"Index.1"}, #"Added Index1", {"Index"}, "Added Index1", JoinKind.LeftOuter),
        #"Expanded Added Index1" = Table.ExpandTableColumn(#"Merged Queries", "Added Index1", {"Product", "Date"}, {"Product.1", "Date.1"}),
        #"Filled Down" = Table.FillDown(#"Expanded Added Index1",{"Product.1", "Date.1"}),
        #"Added Date Range" = Table.AddColumn(#"Filled Down", "Date Range", each if [Product] = [Product.1] and [Date] > [Date.1] then 
    {Number.From(Date.AddDays([Date.1], 1))..Number.From([Date])}
    else {Number.From([Date])}),
        #"Expanded Date Range" = Table.ExpandListColumn(#"Added Date Range", "Date Range"),
        #"Changed Type1" = Table.TransformColumnTypes(#"Expanded Date Range",{{"Date Range", type date}}),
        #"Removed Other Columns" = Table.SelectColumns(#"Changed Type1",{"Product", "MSRP", "Current_Price", "Type", "Date Range"}),
        #"Sorted Rows" = Table.Sort(#"Removed Other Columns",{{"Product", Order.Ascending}, {"Date Range", Order.Descending}})
    in
        #"Sorted Rows"

     

    The sorting in the last step isn't necessary for the computer, but it is for me to read the data and ensure the result looked reasonable. 😁

     

    The end result has 98 records, but here is a sample of it:

8 Replies

  • Greg_Deckler's avatar
    Greg_Deckler
    Community Champion

    ImkeF and edhans probably know better but I'm guessing that this will involve List.Generate in order to get a full list of Dates and then some sort of Merge and a custom column.

  • edhans's avatar
    edhans
    Community Champion

    See if this works. I think it does, but not 100% sure I've properly accounted for the duplicate dates you have, like Feb 27.

     

    Paste this into a blank query:

    1) In Power Query, select New Source, then Blank Query
    2) On the Home ribbon, select "Advanced Editor" button
    3) Remove everything you see, then paste the M code I've given you in that box.
    4) Press Done

     

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQ1MjbWNTSyUNJRMjeCEW6lOTkKAUWZyalAjrG+ib6RgZGBUqwOqnpTOIGi3kjfyBy7BhwWkKPBgCQnGeoboat39TAiyct41WPzASENGD6AaMDpA2NDqIZYAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Product = _t, MSRP = _t, Current_Price = _t, Type = _t, Date = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Product", type text}, {"MSRP", Int64.Type}, {"Current_Price", Int64.Type}, {"Type", type text}, {"Date", type date}}),
        #"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
        #"Added Index1" = Table.AddIndexColumn(#"Added Index", "Index.1", 1, 1),
        #"Merged Queries" = Table.NestedJoin(#"Added Index1", {"Index.1"}, #"Added Index1", {"Index"}, "Added Index1", JoinKind.LeftOuter),
        #"Expanded Added Index1" = Table.ExpandTableColumn(#"Merged Queries", "Added Index1", {"Product", "Date"}, {"Product.1", "Date.1"}),
        #"Filled Down" = Table.FillDown(#"Expanded Added Index1",{"Product.1", "Date.1"}),
        #"Added Date Range" = Table.AddColumn(#"Filled Down", "Date Range", each if [Product] = [Product.1] and [Date] > [Date.1] then 
    {Number.From(Date.AddDays([Date.1], 1))..Number.From([Date])}
    else {Number.From([Date])}),
        #"Expanded Date Range" = Table.ExpandListColumn(#"Added Date Range", "Date Range"),
        #"Changed Type1" = Table.TransformColumnTypes(#"Expanded Date Range",{{"Date Range", type date}}),
        #"Removed Other Columns" = Table.SelectColumns(#"Changed Type1",{"Product", "MSRP", "Current_Price", "Type", "Date Range"}),
        #"Sorted Rows" = Table.Sort(#"Removed Other Columns",{{"Product", Order.Ascending}, {"Date Range", Order.Descending}})
    in
        #"Sorted Rows"

     

    The sorting in the last step isn't necessary for the computer, but it is for me to read the data and ensure the result looked reasonable. 😁

     

    The end result has 98 records, but here is a sample of it:

    • cassidy's avatar
      cassidy
      Power Participant

      Thank you, I think this is very close, but the price changes are going the opposite direction.  I played around with the indexes a bit, but wasn't able to make it reverse.  

       

      Here is a one product example using your query (Left, Current) and what the goal output would be (Right, Expected).  Thanks for the help.

       

      Current     Expected     
      ProductMSRPCurrent_PriceTypeDate Range ProductMSRPCurrent_PriceTypeDate Range 
      60399-G432819Clearance3/16/2020 60399-G432819Clearance3/16/2020*Date Price Changed
      60399-G432819Clearance3/15/2020 60399-G432828Full Price3/15/2020 
      60399-G432819Clearance3/14/2020 60399-G432828Full Price3/14/2020 
      60399-G432819Clearance3/13/2020 60399-G432828Full Price3/13/2020 
      60399-G432819Clearance3/12/2020 60399-G432828Full Price3/12/2020 
      60399-G432819Clearance3/11/2020 60399-G432828Full Price3/11/2020 
      60399-G432828Full Price3/10/2020 60399-G432828Full Price3/10/2020*Date Price Changed
      60399-G432828Full Price3/9/2020 60399-G432814Discount3/9/2020 
      60399-G432828Full Price3/8/2020 60399-G432814Discount3/8/2020 
      60399-G432828Full Price3/7/2020 60399-G432814Discount3/7/2020 
      60399-G432814Discount3/6/2020 60399-G432814Discount3/6/2020*Date Price Changed
      • edhans's avatar
        edhans
        Community Champion

        You asked for this:
        "Looks like the table below.  For example, Line 1 (orange), I need additional rows below it repeating the same values until it gets to Line 2 (blue). "

         

        so I took that to mean you wanted the data in orange copying down until it got to the blue row, then stopping. Is that not accurate?

  • dax's avatar
    dax
    Community Support

    Hi cassidy , 

    I am not clear  about your requirement, if possible could you please inform me more detailed information(such as your expected output and your sample data (by OneDrive for Business))? Then I will help you more correctly.

    Please do mask sensitive data before uploading.

    Thanks for your understanding and support.
    Best Regards,
    Zoe Zhi

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.