Forum Discussion
PQ Matching data that doesn’t exactly match!
I have 2 data sets and I need to merge them together based on date and time. However the date and time quite often don’t match. What I need to do is take first table date and time and find a match in second table within a four hour window (2 hours before and 2 hours after). For example 2 April 2019 10:00 needs to look up any matches between 2 April 2019 08:00 and 12:00.
Appreciate any help! Thanks. Dan
- Anonymous7 years ago
In that case, go through the following example based on your sample data set.
Modify it as necessary. But open a "BlankQuery" in power bi desktop, paste the following codes and run it to get some idea.
let //your sample data set of "Seats" table // Replace the following codes with your actual power query code that fetches the Seats table. Seats = #table( {"Train","Origin","Date","Time","Seats"}, { {"A", "X", #date(2018,4,1),#time(10,5,0), 186}, {"B", "Y", #date(2018,4,1),#time(11,40,0), 147}, {"C", "Z", #date(2018,4,1),#time(13,48,0), 167}, {"A", "X", #date(2018,4,1),#time(22,10,0), 191}, {"B", "Y", #date(2018,4,1),#time(23,55,0), 146}, {"C", "Z", #date(2018,4,1),#time(1,30,0), 170} } ), //your sample data set of "Passengers" table. // Replace the following codes with your actual power query code that fetches the Passengers table. Passengers = #table( {"Train","Origin","Date","Time","Passengers"}, { {"A", "X", #date(2018,4,1),#time(10,9,0), 149}, {"B", "Y", #date(2018,4,1),#time(11,51,0), 118}, {"C", "Z", #date(2018,4,1),#time(13,59,0), 134}, {"A", "X", #date(2018,4,1),#time(22,2,0), 153}, {"B", "Y", #date(2018,4,2),#time(0,2,0), 117}, {"C", "Z", #date(2018,4,1),#time(1,29,0), 136} } ), // Add the date field and time field together to form a datetime column for operations AddDateTimeColumnS = Table.AddColumn(Seats,"DateTimeColumn",each [Date] & [Time],type datetime), // Reduce 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowFrom" AddTimeWindowFrom = Table.AddColumn(AddDateTimeColumnS,"TimeWindowFrom",each [DateTimeColumn] - #duration(0,2,0,0),type datetime), // Add 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowTo" AddTimeWindowTo = Table.AddColumn(AddTimeWindowFrom,"TimeWindowTo",each [DateTimeColumn] + #duration(0,2,0,0),type datetime), // Add the date field and time field together to form a datetime column for operations - To Passengers Table. AddDateTimeColumnP = Table.AddColumn(Passengers,"DateTimeColumn",each [Date] & [Time],type datetime), // Custom Function // This function takes and Train Number, Origin, TimeWindowFrom, TimeWindowTo from Seats table and the entire Passengers table as input. // It matches the details and returns the number of passengers as the list. // IF there are two or more trains matching all the criteria (a rare scenario), it returns the first match from the list MatchTrain = (p as table,t as text,o as text,dtf as datetime,dtt as datetime) as number => List.First( Table.Column( Table.SelectRows(p, each [Train]=t and [Origin]=o and dtf <= [DateTimeColumn] and dtt >=[DateTimeColumn]), "Passengers") ), // Adds a column "Passengers" to the "Seats Table" after matching the details using the custom function "MatchTrain" MatchedTable = Table.AddColumn(AddTimeWindowTo, "Passengers", each MatchTrain(AddDateTimeColumnP,[Train],[Origin],[TimeWindowFrom],[TimeWindowTo]) ), // Removes the unnecessary columns. RemoveUnnecessaryColumns = Table.RemoveColumns(MatchedTable,{"DateTimeColumn","TimeWindowFrom","TimeWindowTo"}), Output=RemoveUnnecessaryColumns in OutputIf you want to see the step by step operations, change the "Output=RemoveUnnecessaryColumns" line to each of the following and execute it. You will be able to see step by step output.
Output = Seats
Output = Passengers
Output = AddDateTimeColumnS
Output = AddTimeWindowFrom
Output = AddTimeWindowTo
Output = AddDateTimeColumnP
Output = MatchedTable
Output = RemoveUnnecessaryColumns
8 Replies
- v-juanli-msft
Community Support
Hi Dan80
If Table 1 is
datetime1 2/2/2019 10:00 2/2/2019 13:00 Table 2 is
datetime2 2/2/2019 7:00 2/2/2019 8:00 2/2/2019 9:00 2/2/2019 10:00 2/2/2019 11:00 2/2/2019 0:00 2/2/2019 13:00 Do you want to crossjoin two tables as below
datetime1 datetime2 2/2/2019 10:00 2/2/2019 8:00 2/2/2019 10:00 2/2/2019 9:00 2/2/2019 10:00 2/2/2019 10:00 2/2/2019 10:00 2/2/2019 11:00 2/2/2019 13:00 2/2/2019 11:00 2/2/2019 13:00 2/2/2019 13:00 Best Regards
Maggie - v-juanli-msft
Community Support
Hi Dan80
I create a test file according to my assumption.
If my understanding is correct or there is any confustion of my solution, please let me know.
Best Regards
MaggieCommunity Support Team _ Maggie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. - AnonymousNot applicable
Without looking at a sample of your actual dataset, it's difficult to give a ready to use solution. But I will approach this in the following way.
Internally, DAX represents datetime datatype as a floating point number where the integer part corresponds to a day and the decimal part corresponds to the hours, minutes, and seconds. i.e.
1 day = 1
1 hour = 1/24
1 minute = 1 / (24*60)
1 second = 1 / (24*60*60)
If you have to match your datetime value with a plus or minus 2-hour window, I think you should add two columns as follows..
StartTime = <datetime1> - (2/24)
EndTime = <datetime1> + (2/24)
But set the data type of StartTime and EndTime to datetime although it does not really matter.
Now from the second table, you can match your datetime2 value with the following expression.
datetime2 >= StartTime AND datetime2 <= EndTime
If if the result of this expression is true, you will have a match.
For every second, within a span of 4 hours window, there are 14400 distinct datetime values possible (4 * 60 * 60).
In a day there are 86400 distinct datetime values (24*60*60). I think instead of maintaining a table of (14400 * 86400) = 1244160000 records per day for matching, you will be better off with adding the start range and end range columns to your table1 and use the boolean condition to match.
Moreover, this method will also handle the datetime window that spans across days. for example 1:00 AM will have the StartTime on the previous day and EndTime on the same day.
- Dan80
Helper II
Thanks for your responses so far, however, the scenario is a little more complicated and the solution needs to be in PQ and not using DAX.
Example below, there are 2 tables regarding train operations. One table has the number of seats that each train had and the other the number of passengers on board each train. Both tables have the following details:
1 Train company
2 Origin of train
3 Date of arrival
4 Time of arrival
However, the data is captured by 2 different systems and so the time of arrival doesn't match but to get a match we need to use the time (give or take ~ an hour or so) to join the two tables together. Any suggestions would be really apprecaited! Thanks,
Dan
- AnonymousNot applicable
In that case, go through the following example based on your sample data set.
Modify it as necessary. But open a "BlankQuery" in power bi desktop, paste the following codes and run it to get some idea.
let //your sample data set of "Seats" table // Replace the following codes with your actual power query code that fetches the Seats table. Seats = #table( {"Train","Origin","Date","Time","Seats"}, { {"A", "X", #date(2018,4,1),#time(10,5,0), 186}, {"B", "Y", #date(2018,4,1),#time(11,40,0), 147}, {"C", "Z", #date(2018,4,1),#time(13,48,0), 167}, {"A", "X", #date(2018,4,1),#time(22,10,0), 191}, {"B", "Y", #date(2018,4,1),#time(23,55,0), 146}, {"C", "Z", #date(2018,4,1),#time(1,30,0), 170} } ), //your sample data set of "Passengers" table. // Replace the following codes with your actual power query code that fetches the Passengers table. Passengers = #table( {"Train","Origin","Date","Time","Passengers"}, { {"A", "X", #date(2018,4,1),#time(10,9,0), 149}, {"B", "Y", #date(2018,4,1),#time(11,51,0), 118}, {"C", "Z", #date(2018,4,1),#time(13,59,0), 134}, {"A", "X", #date(2018,4,1),#time(22,2,0), 153}, {"B", "Y", #date(2018,4,2),#time(0,2,0), 117}, {"C", "Z", #date(2018,4,1),#time(1,29,0), 136} } ), // Add the date field and time field together to form a datetime column for operations AddDateTimeColumnS = Table.AddColumn(Seats,"DateTimeColumn",each [Date] & [Time],type datetime), // Reduce 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowFrom" AddTimeWindowFrom = Table.AddColumn(AddDateTimeColumnS,"TimeWindowFrom",each [DateTimeColumn] - #duration(0,2,0,0),type datetime), // Add 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowTo" AddTimeWindowTo = Table.AddColumn(AddTimeWindowFrom,"TimeWindowTo",each [DateTimeColumn] + #duration(0,2,0,0),type datetime), // Add the date field and time field together to form a datetime column for operations - To Passengers Table. AddDateTimeColumnP = Table.AddColumn(Passengers,"DateTimeColumn",each [Date] & [Time],type datetime), // Custom Function // This function takes and Train Number, Origin, TimeWindowFrom, TimeWindowTo from Seats table and the entire Passengers table as input. // It matches the details and returns the number of passengers as the list. // IF there are two or more trains matching all the criteria (a rare scenario), it returns the first match from the list MatchTrain = (p as table,t as text,o as text,dtf as datetime,dtt as datetime) as number => List.First( Table.Column( Table.SelectRows(p, each [Train]=t and [Origin]=o and dtf <= [DateTimeColumn] and dtt >=[DateTimeColumn]), "Passengers") ), // Adds a column "Passengers" to the "Seats Table" after matching the details using the custom function "MatchTrain" MatchedTable = Table.AddColumn(AddTimeWindowTo, "Passengers", each MatchTrain(AddDateTimeColumnP,[Train],[Origin],[TimeWindowFrom],[TimeWindowTo]) ), // Removes the unnecessary columns. RemoveUnnecessaryColumns = Table.RemoveColumns(MatchedTable,{"DateTimeColumn","TimeWindowFrom","TimeWindowTo"}), Output=RemoveUnnecessaryColumns in OutputIf you want to see the step by step operations, change the "Output=RemoveUnnecessaryColumns" line to each of the following and execute it. You will be able to see step by step output.
Output = Seats
Output = Passengers
Output = AddDateTimeColumnS
Output = AddTimeWindowFrom
Output = AddTimeWindowTo
Output = AddDateTimeColumnP
Output = MatchedTable
Output = RemoveUnnecessaryColumns