Forum Discussion

Dan80's avatar
Dan80
Icon for Helper II rankHelper II
7 years ago
Solved

PQ Matching data that doesn’t exactly match!

Hi all,
I have 2 data sets and I need to merge them together based on date and time. However the date and time quite often don’t match. What I need to do is take first table date and time and find a match in second table within a four hour window (2 hours before and 2 hours after). For example 2 April 2019 10:00 needs to look up any matches between 2 April 2019 08:00 and 12:00.

Appreciate any help! Thanks. Dan
  • Anonymous's avatar
    Anonymous
    7 years ago

    In that case, go through the following example based on your sample data set.

     

    Modify it as necessary. But open a "BlankQuery" in power bi desktop, paste the following codes and run it to get some idea.

     

    let
    
    //your sample data set of "Seats" table
    // Replace the following codes with your actual power query code that fetches the Seats table.
    
        Seats = #table(
            {"Train","Origin","Date","Time","Seats"},
            {
            {"A", "X",  #date(2018,4,1),#time(10,5,0), 186},
            {"B", "Y",  #date(2018,4,1),#time(11,40,0), 147},
            {"C", "Z",  #date(2018,4,1),#time(13,48,0), 167},
            {"A", "X",  #date(2018,4,1),#time(22,10,0), 191},
            {"B", "Y",  #date(2018,4,1),#time(23,55,0), 146},
            {"C", "Z",  #date(2018,4,1),#time(1,30,0), 170}
            }
    
        ),
    
    //your sample data set of "Passengers" table.
    // Replace the following codes with your actual power query code that fetches the Passengers table.
    
        Passengers = #table(
            {"Train","Origin","Date","Time","Passengers"},
            {
                {"A", "X",  #date(2018,4,1),#time(10,9,0), 149},
                {"B", "Y",  #date(2018,4,1),#time(11,51,0), 118},
                {"C", "Z",  #date(2018,4,1),#time(13,59,0), 134},
                {"A", "X",  #date(2018,4,1),#time(22,2,0), 153},
                {"B", "Y",  #date(2018,4,2),#time(0,2,0), 117},
                {"C", "Z",  #date(2018,4,1),#time(1,29,0), 136}
            }
    
        ),
    
    // Add the date field and time field together to form a datetime column for operations
    
    AddDateTimeColumnS = Table.AddColumn(Seats,"DateTimeColumn",each [Date] & [Time],type datetime),
    
    // Reduce 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowFrom"
    
    AddTimeWindowFrom = Table.AddColumn(AddDateTimeColumnS,"TimeWindowFrom",each [DateTimeColumn] - #duration(0,2,0,0),type datetime),
    
    // Add 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowTo"
    
    AddTimeWindowTo = Table.AddColumn(AddTimeWindowFrom,"TimeWindowTo",each [DateTimeColumn] + #duration(0,2,0,0),type datetime),
    
    // Add the date field and time field together to form a datetime column for operations - To Passengers Table.
    
    AddDateTimeColumnP = Table.AddColumn(Passengers,"DateTimeColumn",each [Date] & [Time],type datetime),
    
    // Custom Function
    // This function takes and Train Number, Origin, TimeWindowFrom, TimeWindowTo from Seats table and the entire Passengers table as input.
    // It matches the details and returns the number of passengers as the list. 
    // IF there are two or more trains matching all the criteria (a rare scenario), it returns the first match from the list
    
    MatchTrain = (p as table,t as text,o as text,dtf as datetime,dtt as datetime) as number => 
        List.First(
            Table.Column(
                Table.SelectRows(p, each [Train]=t and [Origin]=o and dtf <= [DateTimeColumn] and dtt >=[DateTimeColumn]),
                "Passengers")
        ),
    
    // Adds a column "Passengers" to the "Seats Table" after matching the details using the custom function "MatchTrain"
    
    MatchedTable = Table.AddColumn(AddTimeWindowTo,
        "Passengers",
        each MatchTrain(AddDateTimeColumnP,[Train],[Origin],[TimeWindowFrom],[TimeWindowTo])
        ),
    
    // Removes the unnecessary columns.
    
    RemoveUnnecessaryColumns = Table.RemoveColumns(MatchedTable,{"DateTimeColumn","TimeWindowFrom","TimeWindowTo"}),
    
    Output=RemoveUnnecessaryColumns
    
    in
        Output

    If you want to see the step by step operations, change the "Output=RemoveUnnecessaryColumns" line to each of the following and execute it. You will be able to see step by step output.

     

    Output = Seats

    Output = Passengers

    Output = AddDateTimeColumnS

    Output = AddTimeWindowFrom

    Output = AddTimeWindowTo

    Output = AddDateTimeColumnP

    Output = MatchedTable

    Output = RemoveUnnecessaryColumns

8 Replies

  • v-juanli-msft's avatar
    v-juanli-msft
    Icon for Community Support rankCommunity Support

    Hi Dan80 

    If Table 1 is 

    datetime1
    2/2/2019 10:00
    2/2/2019 13:00

    Table 2 is

    datetime2
    2/2/2019 7:00
    2/2/2019 8:00
    2/2/2019 9:00
    2/2/2019 10:00
    2/2/2019 11:00
    2/2/2019 0:00
    2/2/2019 13:00

    Do you want to crossjoin two tables as below

    datetime1 datetime2
    2/2/2019 10:00 2/2/2019 8:00
    2/2/2019 10:00 2/2/2019 9:00
    2/2/2019 10:00 2/2/2019 10:00
    2/2/2019 10:00 2/2/2019 11:00
    2/2/2019 13:00 2/2/2019 11:00
    2/2/2019 13:00 2/2/2019 13:00

     

    Best Regards
    Maggie

  • v-juanli-msft's avatar
    v-juanli-msft
    Icon for Community Support rankCommunity Support

    Hi Dan80 

    I create a test file according to my assumption.

    If my understanding is correct or there is any confustion of my solution, please let me know.

    Best Regards
    Maggie

     

    Community Support Team _ Maggie Li
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • Anonymous's avatar
    Anonymous
    Not applicable

    Without looking at a sample of your actual dataset, it's difficult to give a ready to use solution. But I will approach this in the following way.

     

    Internally, DAX represents datetime datatype as a floating point number where the integer part corresponds to a day and the decimal part corresponds to the hours, minutes, and seconds. i.e.

     

    1 day = 1

    1 hour = 1/24

    1 minute = 1 / (24*60)

    1 second = 1 / (24*60*60)

     

    If you have to match your datetime value with a plus or minus 2-hour window, I think you should add two columns as follows..

     

    StartTime = <datetime1> - (2/24)

    EndTime = <datetime1> + (2/24)

     

    But set the data type of StartTime and EndTime to datetime although it does not really matter.

     

    Now from the second table, you can match your datetime2 value with the following expression.

     

     

    datetime2 >= StartTime AND datetime2 <= EndTime

     

     

    If if the result of this expression is true, you will have a match.

     

    For every second, within a span of 4 hours window, there are 14400 distinct datetime values possible (4 * 60 * 60). 

    In a day there are 86400 distinct datetime values (24*60*60). I think instead of maintaining a table of (14400 * 86400) = 1244160000 records per day for matching, you will be better off with adding the start range and end range columns to your table1 and use the boolean condition to match.

     

    Moreover, this method will also handle the datetime window that spans across days. for example 1:00 AM will have the StartTime on the previous day and EndTime on the same day. 

     

    • Dan80's avatar
      Dan80
      Icon for Helper II rankHelper II

      Thanks for your responses so far, however, the scenario is a little more complicated and the solution needs to be in PQ and not using DAX.  

       

      Example below, there are 2 tables regarding train operations.  One table has the number of seats that each train had and the other the number of passengers on board each train.  Both tables have the following details:

       

      1  Train company

      2  Origin of train

      3  Date of arrival

      4  Time of arrival

       

      However, the data is captured by 2 different systems and so the time of arrival doesn't match but to get a match we need to use the time (give or take ~ an hour or so) to join the two tables together.  Any suggestions would be really apprecaited!  Thanks,

       

      Dan


      • Anonymous's avatar
        Anonymous
        Not applicable

        In that case, go through the following example based on your sample data set.

         

        Modify it as necessary. But open a "BlankQuery" in power bi desktop, paste the following codes and run it to get some idea.

         

        let
        
        //your sample data set of "Seats" table
        // Replace the following codes with your actual power query code that fetches the Seats table.
        
            Seats = #table(
                {"Train","Origin","Date","Time","Seats"},
                {
                {"A", "X",  #date(2018,4,1),#time(10,5,0), 186},
                {"B", "Y",  #date(2018,4,1),#time(11,40,0), 147},
                {"C", "Z",  #date(2018,4,1),#time(13,48,0), 167},
                {"A", "X",  #date(2018,4,1),#time(22,10,0), 191},
                {"B", "Y",  #date(2018,4,1),#time(23,55,0), 146},
                {"C", "Z",  #date(2018,4,1),#time(1,30,0), 170}
                }
        
            ),
        
        //your sample data set of "Passengers" table.
        // Replace the following codes with your actual power query code that fetches the Passengers table.
        
            Passengers = #table(
                {"Train","Origin","Date","Time","Passengers"},
                {
                    {"A", "X",  #date(2018,4,1),#time(10,9,0), 149},
                    {"B", "Y",  #date(2018,4,1),#time(11,51,0), 118},
                    {"C", "Z",  #date(2018,4,1),#time(13,59,0), 134},
                    {"A", "X",  #date(2018,4,1),#time(22,2,0), 153},
                    {"B", "Y",  #date(2018,4,2),#time(0,2,0), 117},
                    {"C", "Z",  #date(2018,4,1),#time(1,29,0), 136}
                }
        
            ),
        
        // Add the date field and time field together to form a datetime column for operations
        
        AddDateTimeColumnS = Table.AddColumn(Seats,"DateTimeColumn",each [Date] & [Time],type datetime),
        
        // Reduce 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowFrom"
        
        AddTimeWindowFrom = Table.AddColumn(AddDateTimeColumnS,"TimeWindowFrom",each [DateTimeColumn] - #duration(0,2,0,0),type datetime),
        
        // Add 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowTo"
        
        AddTimeWindowTo = Table.AddColumn(AddTimeWindowFrom,"TimeWindowTo",each [DateTimeColumn] + #duration(0,2,0,0),type datetime),
        
        // Add the date field and time field together to form a datetime column for operations - To Passengers Table.
        
        AddDateTimeColumnP = Table.AddColumn(Passengers,"DateTimeColumn",each [Date] & [Time],type datetime),
        
        // Custom Function
        // This function takes and Train Number, Origin, TimeWindowFrom, TimeWindowTo from Seats table and the entire Passengers table as input.
        // It matches the details and returns the number of passengers as the list. 
        // IF there are two or more trains matching all the criteria (a rare scenario), it returns the first match from the list
        
        MatchTrain = (p as table,t as text,o as text,dtf as datetime,dtt as datetime) as number => 
            List.First(
                Table.Column(
                    Table.SelectRows(p, each [Train]=t and [Origin]=o and dtf <= [DateTimeColumn] and dtt >=[DateTimeColumn]),
                    "Passengers")
            ),
        
        // Adds a column "Passengers" to the "Seats Table" after matching the details using the custom function "MatchTrain"
        
        MatchedTable = Table.AddColumn(AddTimeWindowTo,
            "Passengers",
            each MatchTrain(AddDateTimeColumnP,[Train],[Origin],[TimeWindowFrom],[TimeWindowTo])
            ),
        
        // Removes the unnecessary columns.
        
        RemoveUnnecessaryColumns = Table.RemoveColumns(MatchedTable,{"DateTimeColumn","TimeWindowFrom","TimeWindowTo"}),
        
        Output=RemoveUnnecessaryColumns
        
        in
            Output

        If you want to see the step by step operations, change the "Output=RemoveUnnecessaryColumns" line to each of the following and execute it. You will be able to see step by step output.

         

        Output = Seats

        Output = Passengers

        Output = AddDateTimeColumnS

        Output = AddTimeWindowFrom

        Output = AddTimeWindowTo

        Output = AddDateTimeColumnP

        Output = MatchedTable

        Output = RemoveUnnecessaryColumns