Forum Discussion
PQ Matching data that doesn’t exactly match!
- Anonymous7 years ago
In that case, go through the following example based on your sample data set.
Modify it as necessary. But open a "BlankQuery" in power bi desktop, paste the following codes and run it to get some idea.
let //your sample data set of "Seats" table // Replace the following codes with your actual power query code that fetches the Seats table. Seats = #table( {"Train","Origin","Date","Time","Seats"}, { {"A", "X", #date(2018,4,1),#time(10,5,0), 186}, {"B", "Y", #date(2018,4,1),#time(11,40,0), 147}, {"C", "Z", #date(2018,4,1),#time(13,48,0), 167}, {"A", "X", #date(2018,4,1),#time(22,10,0), 191}, {"B", "Y", #date(2018,4,1),#time(23,55,0), 146}, {"C", "Z", #date(2018,4,1),#time(1,30,0), 170} } ), //your sample data set of "Passengers" table. // Replace the following codes with your actual power query code that fetches the Passengers table. Passengers = #table( {"Train","Origin","Date","Time","Passengers"}, { {"A", "X", #date(2018,4,1),#time(10,9,0), 149}, {"B", "Y", #date(2018,4,1),#time(11,51,0), 118}, {"C", "Z", #date(2018,4,1),#time(13,59,0), 134}, {"A", "X", #date(2018,4,1),#time(22,2,0), 153}, {"B", "Y", #date(2018,4,2),#time(0,2,0), 117}, {"C", "Z", #date(2018,4,1),#time(1,29,0), 136} } ), // Add the date field and time field together to form a datetime column for operations AddDateTimeColumnS = Table.AddColumn(Seats,"DateTimeColumn",each [Date] & [Time],type datetime), // Reduce 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowFrom" AddTimeWindowFrom = Table.AddColumn(AddDateTimeColumnS,"TimeWindowFrom",each [DateTimeColumn] - #duration(0,2,0,0),type datetime), // Add 2 hours from the DateTimeColumn added in the previous step as a new column "TimeWindowTo" AddTimeWindowTo = Table.AddColumn(AddTimeWindowFrom,"TimeWindowTo",each [DateTimeColumn] + #duration(0,2,0,0),type datetime), // Add the date field and time field together to form a datetime column for operations - To Passengers Table. AddDateTimeColumnP = Table.AddColumn(Passengers,"DateTimeColumn",each [Date] & [Time],type datetime), // Custom Function // This function takes and Train Number, Origin, TimeWindowFrom, TimeWindowTo from Seats table and the entire Passengers table as input. // It matches the details and returns the number of passengers as the list. // IF there are two or more trains matching all the criteria (a rare scenario), it returns the first match from the list MatchTrain = (p as table,t as text,o as text,dtf as datetime,dtt as datetime) as number => List.First( Table.Column( Table.SelectRows(p, each [Train]=t and [Origin]=o and dtf <= [DateTimeColumn] and dtt >=[DateTimeColumn]), "Passengers") ), // Adds a column "Passengers" to the "Seats Table" after matching the details using the custom function "MatchTrain" MatchedTable = Table.AddColumn(AddTimeWindowTo, "Passengers", each MatchTrain(AddDateTimeColumnP,[Train],[Origin],[TimeWindowFrom],[TimeWindowTo]) ), // Removes the unnecessary columns. RemoveUnnecessaryColumns = Table.RemoveColumns(MatchedTable,{"DateTimeColumn","TimeWindowFrom","TimeWindowTo"}), Output=RemoveUnnecessaryColumns in OutputIf you want to see the step by step operations, change the "Output=RemoveUnnecessaryColumns" line to each of the following and execute it. You will be able to see step by step output.
Output = Seats
Output = Passengers
Output = AddDateTimeColumnS
Output = AddTimeWindowFrom
Output = AddTimeWindowTo
Output = AddDateTimeColumnP
Output = MatchedTable
Output = RemoveUnnecessaryColumns
Thanks for your responses so far, however, the scenario is a little more complicated and the solution needs to be in PQ and not using DAX.
Example below, there are 2 tables regarding train operations. One table has the number of seats that each train had and the other the number of passengers on board each train. Both tables have the following details:
1 Train company
2 Origin of train
3 Date of arrival
4 Time of arrival
However, the data is captured by 2 different systems and so the time of arrival doesn't match but to get a match we need to use the time (give or take ~ an hour or so) to join the two tables together. Any suggestions would be really apprecaited! Thanks,
Dan
Hi Dan80 ,
Here is the way I went about
1. The time column being totally differen in two tables, it is not possible to do merge of the two tables.
2. So based on the value of time in the tables create a column called Shift.
06:00 AM to 18:59 PM let us assign shift with value 1.
And time from 19:00 (07:00 PM) to 05:59 AM let us assign shift with value 2.
3. This shift column is computed for both SeatData and PassengerData table.
4. Now you can merge the SeatDate with PassengerData on Train+Origin+Date+Shift
5. The M-Script for PassengerSeatData
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUYoA4pCM0qLilMRKHQUTBa/EvNLEokoFIwNDC6CUoYGVgSWINrFUitWJVnICsiMJaDG0MjUE0xZgLc5AdhQBLcZWpmBbjE3AWohwmJGRlZEBSIupMbEOA3rFCOwuc2LdZWBoZQxxl5lSbCwA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Train = _t, Origin = _t, Date = _t, Time = _t, Seats = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Train", type text}, {"Origin", type text}, {"Date", type date}, {"Time", type time}, {"Seats", Int64.Type}}),
#"Range" = Table.AddColumn(#"Changed Type", "Shift", each if
Time.Hour([Time]) >= 06 and Time.Hour([Time]) < 19 then 1 else 2)
in
Range
6. The M-Script for SeatData
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("dY5LDoAgDETv0jUJLZSP3amXUAn3v4alLpFFM9PkZWZagx0cXHpIHtkHpKoPoWAaWjN01+BQf08QCeNQLgad6p8JisKm+YN+60IQsqSN1nUhSrJNnNd1SBItqSD0/gI=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Train = _t, Origin = _t, Date = _t, Time = _t, Seats = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Train", type text}, {"Origin", type text}, {"Date", type date}, {"Time", type time},
{"Seats", Int64.Type}}),
#"Range" = Table.AddColumn(#"Changed Type", "Shift", each if
Time.Hour([Time]) >= 06 and Time.Hour([Time]) < 19
then 1 else 2),
#"Merged Queries" = Table.NestedJoin(Range, {"Train", "Origin", "Date", "Shift"}, PassengerData, {"Train", "Origin", "Date", "Shift"}, "PassengerData", JoinKind.LeftOuter),
#"Expanded PassengerData" = Table.ExpandTableColumn(#"Merged Queries", "PassengerData", {"Seats"}, {"PassengerData.Seats"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded PassengerData",{{"PassengerData.Seats", "PassengerSeats"}})
in
#"Renamed Columns"
7. After the merging of SeatData with PassengerData you can find the number of empty seats by Train/Origin/Date/TIme(seattable).
The only caveat what if a train from an origin operates on a 2 hour shuttle. How to identify such train services and the logic for computed Column - Shift.
Cheers
CheenuSing
CheenuSing