Forum Discussion
Lookup multiple items in a list and merge matching data
Dear All
Kindly help me for below mentioned situation
I have two tables having data like as under:
Table1 :
ID_NO1,OTHER_COLUMN1,OTHER_COLUMN2,OTHER_COLUMN3,OTHER_COLUMN4,OTHER_COLUMN5,OTHER_COLUMN6,OTHER_COLUMN7,ID_NO2,ID_NO3,OTHER_COLUMN8,OTHER_COLUMN9,OTHER_COLUMN10,OTHER_COLUMN11
A,,,,,,,,B,null,,,,
C,,,,,,,,E,D,,,,
F,,,,,,,,G,null,,,,
X,,,,,,,,Y,Z,,,,
Table2:
ID_NO,INFO1,INFO2,INFO3,INFO4,INFO5
A,,,,,
B,,,,,
C,,,,,
D,,,,,
E,,,,,
F,,,,,
G,,,,,
H,,,,,
I,,,,,
J,,,,,
My target is as under:
- I want to obtain all rows from Table1 in which columns Named ID_NO1,ID_NO2,ID_NO3 contain any one or more ID_NOs existing in column named ID_NO in Table2. Eg. Rows No 1,2 and 3 in Table1 contain IDs found in Table2. ID_NO1, ID_NO2 and ID_NO3 each contain only 1 ID or a null. The Row No 4 in the Table1 has to be filtered out, as it does not contain any of the ID_NOs in Table2.
- After obtaining rows as above, I want to attach columns named INFO1 to INFO5 (from Table2) in Table1 to respective ID_NO found in Table1. Eg. In Row 1, I want to attach INFO1 to INFO5 columns of ID_NO A to ID_NO A found in Row No 1 and similarly attach INFO1 to INFO5 columns of ID_NO B to ID_NO B found in ID_NO2 in Row 1.
- To obtain rows as mentioned in Point - 1, I created list of each items in ID_NO1,ID_NO2, ID_NO3 for each row in Table1 like this..... {[ID_NO1],[ID_NO2],[ID_NO3]}. Then I created a list of all items in column named ID_NO in Table2 and I also buffered this list with List.Buffer. Then for each row I created a custom column using List.Intersect to obtain matching Items in both of the lists for each row. So, list of ID_NOs of Table2 would be same for each row in Table1 and list of 3 ID_NOs of Table1 will be different for each row. Then List.Intersect could give me one or more matching Items in the form of a List.
- Thereafter, when I tried to expand list obtained for each row in custom column as obtained in Point No 3, my system became very slow. I wanted to expand list of each row to multiple rows with each item inside that list. But system stopped working and I did not get any result. So I need an alternate or faster method to do this.
- After obtaining result as expected in Point No 4, I am stuck on how to attach INFO1 to INFO5 to each item expanded as mentioned in Point No 4. Because, merely expanding my lists in each row halted the PC altogether. So how would I attach INFO1 to INFO5 to each item without making my PC to take a grinding halt. I have thought of simply merging 2 queries based on ID_NO but it is possible only after I obtain a list as mentioned in Point No 4.
- For information, it may be noted that my actual data in Table1 is spanning across around 45 columns including 3 columns ID_NO1, ID_NO2, ID_NO3 and there are around 150000 rows in Table1. Out of the 45 columns, I will be keeping only 12-14 columns. But 5 columns as mentioned in Table2 i.e. INFO1 to INFO5 will be added for each row for each matching ID_NO.
- The Table2 contains around 270 distinct ID_NOs which have to be matched with ID_NOs in Table1.
I request you all to suggest a way which can work faster so that I can accomplish my task.
Regards
let Table1 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Tc65DcMwDEDRXVS7iG679CkfCyQxvIGRLvuH+mxS8LOg8KDzNL1pTN9b6TA46Th66TQF6TxH6bIkaSm5vpH5fO9b1rq20m3rpPtuH7KOw1pzNacZQR2oBw2gETSBZtBqzPWG2CFiCWmVdJALpIcMkBEyQWbIFrIi5f+feMJaZZ2yHvYJG2AjbILNsC0sYEF5ybwVtYo6Rb2iwVzXDw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO1 = _t, OTHER_COLUMN1 = _t, OTHER_COLUMN2 = _t, OTHER_COLUMN3 = _t, OTHER_COLUMN4 = _t, OTHER_COLUMN5 = _t, OTHER_COLUMN6 = _t, OTHER_COLUMN7 = _t, ID_NO2 = _t, ID_NO3 = _t, OTHER_COLUMN8 = _t, OTHER_COLUMN9 = _t, OTHER_COLUMN10 = _t, OTHER_COLUMN11 = _t]), Table2= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("HdC5FYRADATRXGRjiBtMkMQeKcyb/NPYrXa+U161ZpcNdo1/bghIqNH60OwmT2QISKhJOcgzGQISalZO8kKGgIRalIu8kiEgoVblh7yRISChNuUXeSdDQELtym/yQYaAhDqUP+STDAEJdSp/tcX1xTXGdca1xq33Hw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO = _t, INFO1 = _t, INFO2 = _t, INFO3 = _t, INFO4 = _t, INFO5 = _t]), Custom1 = let ColumnListToTransform={"ID_NO1","ID_NO2","ID_NO3"} in Table.FromRecords(List.TransformMany(Table.ToRecords(Table1),each List.RemoveNulls(List.Transform(Record.ToList(Record.SelectFields(_,ColumnListToTransform,1)),each Table2{[ID_NO=_]}?)),(x,y)=>Record.RemoveFields(x,ColumnListToTransform)&y)) in Custom1
6 Replies
- wdx223_Daniel
Community Champion
it will be better to show some sample data in table and the final result you want.
- CMMaliniJoshi
Helper I
Dear wdx223_Daniel
As required, I am appending hereunder sample data. Please suggest.
Table1:
ID_NO1
OTHER_COLUMN1
OTHER_COLUMN2
OTHER_COLUMN3
OTHER_COLUMN4
OTHER_COLUMN5
OTHER_COLUMN6
OTHER_COLUMN7
ID_NO2
ID_NO3
OTHER_COLUMN8
OTHER_COLUMN9
OTHER_COLUMN10
OTHER_COLUMN11
A
AA1
BB2
CC3
DD4
EE5
FF6
GG7
B
null
HH8
II9
JJ10
KK11
C
AA2
BB3
CC4
DD5
EE6
FF7
GG8
E
D
HH9
II10
JJ11
KK12
F
AA3
BB4
CC5
DD6
EE7
FF8
GG9
G
null
HH10
II11
JJ12
KK13
X
AA4
BB5
CC6
DD7
EE8
FF9
GG10
Y
Z
HH11
II12
JJ13
KK14
Table2:
ID_NO
INFO1
INFO2
INFO3
INFO4
INFO5
A
A1
B1
C1
D1
E1
B
A2
B2
C2
D2
E2
C
A3
B3
C3
D3
E3
D
A4
B4
C4
D4
E4
E
A5
B5
C5
D5
E5
F
A6
B6
C6
D6
E6
G
A7
B7
C7
D7
E7
H
A8
B8
C8
D8
E8
I
A9
B9
C9
D9
E9
J
A10
B10
C10
D10
E10
Output:
OTHER_COLUMN1
OTHER_COLUMN2
OTHER_COLUMN3
OTHER_COLUMN4
OTHER_COLUMN5
OTHER_COLUMN6
OTHER_COLUMN7
ID_NO
INFO1
INFO2
INFO3
INFO4
INFO5
AA1
BB2
CC3
DD4
EE5
FF6
GG7
A
A1
B1
C1
D1
E1
AA1
BB2
CC3
DD4
EE5
FF6
GG7
B
A2
B2
C2
D2
E2
AA2
BB3
CC4
DD5
EE6
FF7
GG8
C
A3
B3
C3
D3
E3
AA2
BB3
CC4
DD5
EE6
FF7
GG8
E
A5
B5
C5
D5
E5
AA2
BB3
CC4
DD5
EE6
FF7
GG8
D
A4
B4
C4
D4
E4
AA3
BB4
CC5
DD6
EE7
FF8
GG9
F
A6
B6
C6
D6
E6
AA3
BB4
CC5
DD6
EE7
FF8
GG9
G
A7
B7
C7
D7
E7
Regards
- wdx223_Daniel
Community Champion
let Table1 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Tc65DcMwDEDRXVS7iG679CkfCyQxvIGRLvuH+mxS8LOg8KDzNL1pTN9b6TA46Th66TQF6TxH6bIkaSm5vpH5fO9b1rq20m3rpPtuH7KOw1pzNacZQR2oBw2gETSBZtBqzPWG2CFiCWmVdJALpIcMkBEyQWbIFrIi5f+feMJaZZ2yHvYJG2AjbILNsC0sYEF5ybwVtYo6Rb2iwVzXDw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO1 = _t, OTHER_COLUMN1 = _t, OTHER_COLUMN2 = _t, OTHER_COLUMN3 = _t, OTHER_COLUMN4 = _t, OTHER_COLUMN5 = _t, OTHER_COLUMN6 = _t, OTHER_COLUMN7 = _t, ID_NO2 = _t, ID_NO3 = _t, OTHER_COLUMN8 = _t, OTHER_COLUMN9 = _t, OTHER_COLUMN10 = _t, OTHER_COLUMN11 = _t]), Table2= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("HdC5FYRADATRXGRjiBtMkMQeKcyb/NPYrXa+U161ZpcNdo1/bghIqNH60OwmT2QISKhJOcgzGQISalZO8kKGgIRalIu8kiEgoVblh7yRISChNuUXeSdDQELtym/yQYaAhDqUP+STDAEJdSp/tcX1xTXGdca1xq33Hw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO = _t, INFO1 = _t, INFO2 = _t, INFO3 = _t, INFO4 = _t, INFO5 = _t]), Custom1 = let ColumnListToTransform={"ID_NO1","ID_NO2","ID_NO3"} in Table.FromRecords(List.TransformMany(Table.ToRecords(Table1),each List.RemoveNulls(List.Transform(Record.ToList(Record.SelectFields(_,ColumnListToTransform,1)),each Table2{[ID_NO=_]}?)),(x,y)=>Record.RemoveFields(x,ColumnListToTransform)&y)) in Custom1