Forum Discussion

CMMaliniJoshi's avatar
3 years ago
Solved

Lookup multiple items in a list and merge matching data

Dear All

Kindly help me for below mentioned situation

I have two tables having data like as under:

Table1 :

ID_NO1,OTHER_COLUMN1,OTHER_COLUMN2,OTHER_COLUMN3,OTHER_COLUMN4,OTHER_COLUMN5,OTHER_COLUMN6,OTHER_COLUMN7,ID_NO2,ID_NO3,OTHER_COLUMN8,OTHER_COLUMN9,OTHER_COLUMN10,OTHER_COLUMN11

A,,,,,,,,B,null,,,,

C,,,,,,,,E,D,,,,

F,,,,,,,,G,null,,,,

X,,,,,,,,Y,Z,,,,

Table2:

ID_NO,INFO1,INFO2,INFO3,INFO4,INFO5

A,,,,,

B,,,,,

C,,,,,

D,,,,,

E,,,,,

F,,,,,

G,,,,,

H,,,,,

I,,,,,

J,,,,,

My target is as under:

  1. I want to obtain all rows from Table1 in which columns Named ID_NO1,ID_NO2,ID_NO3 contain any one or more ID_NOs existing in column named ID_NO in Table2. Eg. Rows No 1,2 and 3 in Table1 contain IDs found in Table2. ID_NO1, ID_NO2 and ID_NO3 each contain only 1 ID or a null.  The Row No 4 in the Table1 has to be filtered out, as it does not contain any of the ID_NOs in Table2.
  2. After obtaining rows as above, I want to attach columns named INFO1 to INFO5 (from Table2) in Table1 to respective ID_NO found in Table1. Eg. In Row 1, I want to attach INFO1 to INFO5 columns of ID_NO A to ID_NO A found in Row No 1 and similarly attach INFO1 to INFO5 columns of ID_NO B to ID_NO B found in ID_NO2 in Row 1.
  3. To obtain rows as mentioned in Point - 1, I created list of each items in ID_NO1,ID_NO2, ID_NO3 for each row in Table1 like this..... {[ID_NO1],[ID_NO2],[ID_NO3]}. Then I created a list of all items in column named ID_NO in Table2 and I also buffered this list with List.Buffer. Then for each row I created a custom column using List.Intersect to obtain matching Items in both of the lists for each row.  So, list of ID_NOs of Table2 would be same for each row in Table1 and list of 3 ID_NOs of Table1 will be different for each row.  Then List.Intersect could give me one or more matching Items in the form of a List.
  4. Thereafter, when I tried to expand list obtained for each row in custom column as obtained in Point No 3, my system became very slow.  I wanted to expand list of each row to multiple rows with each item inside that list.  But system stopped working and I did not get any result.  So I need an alternate or faster method to do this.
  5. After obtaining result as expected in Point No 4, I am stuck on how to attach INFO1 to INFO5 to each item expanded as mentioned in Point No 4.  Because, merely expanding my lists in each row halted the PC altogether.   So how would I attach INFO1 to INFO5 to each item without making my PC to take a grinding halt.  I have thought of simply merging 2 queries based on ID_NO but it is possible only after I obtain a list as mentioned in Point No 4.
  6. For information, it may be noted that my actual data in Table1 is spanning across around 45 columns including 3 columns ID_NO1, ID_NO2, ID_NO3 and there are around 150000 rows in Table1.  Out of the 45 columns, I will be keeping only 12-14 columns.  But 5 columns as mentioned in Table2 i.e. INFO1 to INFO5 will be added for each row for each matching ID_NO.
  7. The Table2 contains around 270 distinct ID_NOs which have to be matched with ID_NOs in Table1.

I request you all to suggest a way which can work faster so that I can accomplish my task.

Regards

  • wdx223_Daniel's avatar
    wdx223_Daniel
    3 years ago
    let
        Table1 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Tc65DcMwDEDRXVS7iG679CkfCyQxvIGRLvuH+mxS8LOg8KDzNL1pTN9b6TA46Th66TQF6TxH6bIkaSm5vpH5fO9b1rq20m3rpPtuH7KOw1pzNacZQR2oBw2gETSBZtBqzPWG2CFiCWmVdJALpIcMkBEyQWbIFrIi5f+feMJaZZ2yHvYJG2AjbILNsC0sYEF5ybwVtYo6Rb2iwVzXDw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO1 = _t, OTHER_COLUMN1 = _t, OTHER_COLUMN2 = _t, OTHER_COLUMN3 = _t, OTHER_COLUMN4 = _t, OTHER_COLUMN5 = _t, OTHER_COLUMN6 = _t, OTHER_COLUMN7 = _t, ID_NO2 = _t, ID_NO3 = _t, OTHER_COLUMN8 = _t, OTHER_COLUMN9 = _t, OTHER_COLUMN10 = _t, OTHER_COLUMN11 = _t]),
        Table2= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("HdC5FYRADATRXGRjiBtMkMQeKcyb/NPYrXa+U161ZpcNdo1/bghIqNH60OwmT2QISKhJOcgzGQISalZO8kKGgIRalIu8kiEgoVblh7yRISChNuUXeSdDQELtym/yQYaAhDqUP+STDAEJdSp/tcX1xTXGdca1xq33Hw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO = _t, INFO1 = _t, INFO2 = _t, INFO3 = _t, INFO4 = _t, INFO5 = _t]),
        Custom1 = let ColumnListToTransform={"ID_NO1","ID_NO2","ID_NO3"} in Table.FromRecords(List.TransformMany(Table.ToRecords(Table1),each List.RemoveNulls(List.Transform(Record.ToList(Record.SelectFields(_,ColumnListToTransform,1)),each Table2{[ID_NO=_]}?)),(x,y)=>Record.RemoveFields(x,ColumnListToTransform)&y))
    in
        Custom1

6 Replies

  • wdx223_Daniel's avatar
    wdx223_Daniel
    Icon for Community Champion rankCommunity Champion

    it will be better to show some sample data in table and the final result you want. 

    • CMMaliniJoshi's avatar
      CMMaliniJoshi
      Icon for Helper I rankHelper I

      Dear wdx223_Daniel

       

      As required, I am appending hereunder sample data.  Please suggest.

       

      Table1:

       

      ID_NO1

      OTHER_COLUMN1

      OTHER_COLUMN2

      OTHER_COLUMN3

      OTHER_COLUMN4

      OTHER_COLUMN5

      OTHER_COLUMN6

      OTHER_COLUMN7

      ID_NO2

      ID_NO3

      OTHER_COLUMN8

      OTHER_COLUMN9

      OTHER_COLUMN10

      OTHER_COLUMN11

      A

      AA1

      BB2

      CC3

      DD4

      EE5

      FF6

      GG7

      B

      null

      HH8

      II9

      JJ10

      KK11

      C

      AA2

      BB3

      CC4

      DD5

      EE6

      FF7

      GG8

      E

      D

      HH9

      II10

      JJ11

      KK12

      F

      AA3

      BB4

      CC5

      DD6

      EE7

      FF8

      GG9

      G

      null

      HH10

      II11

      JJ12

      KK13

      X

      AA4

      BB5

      CC6

      DD7

      EE8

      FF9

      GG10

      Y

      Z

      HH11

      II12

      JJ13

      KK14

       

      Table2:

       

      ID_NO

      INFO1

      INFO2

      INFO3

      INFO4

      INFO5

      A

      A1

      B1

      C1

      D1

      E1

      B

      A2

      B2

      C2

      D2

      E2

      C

      A3

      B3

      C3

      D3

      E3

      D

      A4

      B4

      C4

      D4

      E4

      E

      A5

      B5

      C5

      D5

      E5

      F

      A6

      B6

      C6

      D6

      E6

      G

      A7

      B7

      C7

      D7

      E7

      H

      A8

      B8

      C8

      D8

      E8

      I

      A9

      B9

      C9

      D9

      E9

      J

      A10

      B10

      C10

      D10

      E10

       

      Output:

       

      OTHER_COLUMN1

      OTHER_COLUMN2

      OTHER_COLUMN3

      OTHER_COLUMN4

      OTHER_COLUMN5

      OTHER_COLUMN6

      OTHER_COLUMN7

      ID_NO

      INFO1

      INFO2

      INFO3

      INFO4

      INFO5

      AA1

      BB2

      CC3

      DD4

      EE5

      FF6

      GG7

      A

      A1

      B1

      C1

      D1

      E1

      AA1

      BB2

      CC3

      DD4

      EE5

      FF6

      GG7

      B

      A2

      B2

      C2

      D2

      E2

      AA2

      BB3

      CC4

      DD5

      EE6

      FF7

      GG8

      C

      A3

      B3

      C3

      D3

      E3

      AA2

      BB3

      CC4

      DD5

      EE6

      FF7

      GG8

      E

      A5

      B5

      C5

      D5

      E5

      AA2

      BB3

      CC4

      DD5

      EE6

      FF7

      GG8

      D

      A4

      B4

      C4

      D4

      E4

      AA3

      BB4

      CC5

      DD6

      EE7

      FF8

      GG9

      F

      A6

      B6

      C6

      D6

      E6

      AA3

      BB4

      CC5

      DD6

      EE7

      FF8

      GG9

      G

      A7

      B7

      C7

      D7

      E7

       

      Regards

       

      • wdx223_Daniel's avatar
        wdx223_Daniel
        Icon for Community Champion rankCommunity Champion
        let
            Table1 = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Tc65DcMwDEDRXVS7iG679CkfCyQxvIGRLvuH+mxS8LOg8KDzNL1pTN9b6TA46Th66TQF6TxH6bIkaSm5vpH5fO9b1rq20m3rpPtuH7KOw1pzNacZQR2oBw2gETSBZtBqzPWG2CFiCWmVdJALpIcMkBEyQWbIFrIi5f+feMJaZZ2yHvYJG2AjbILNsC0sYEF5ybwVtYo6Rb2iwVzXDw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO1 = _t, OTHER_COLUMN1 = _t, OTHER_COLUMN2 = _t, OTHER_COLUMN3 = _t, OTHER_COLUMN4 = _t, OTHER_COLUMN5 = _t, OTHER_COLUMN6 = _t, OTHER_COLUMN7 = _t, ID_NO2 = _t, ID_NO3 = _t, OTHER_COLUMN8 = _t, OTHER_COLUMN9 = _t, OTHER_COLUMN10 = _t, OTHER_COLUMN11 = _t]),
            Table2= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("HdC5FYRADATRXGRjiBtMkMQeKcyb/NPYrXa+U161ZpcNdo1/bghIqNH60OwmT2QISKhJOcgzGQISalZO8kKGgIRalIu8kiEgoVblh7yRISChNuUXeSdDQELtym/yQYaAhDqUP+STDAEJdSp/tcX1xTXGdca1xq33Hw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID_NO = _t, INFO1 = _t, INFO2 = _t, INFO3 = _t, INFO4 = _t, INFO5 = _t]),
            Custom1 = let ColumnListToTransform={"ID_NO1","ID_NO2","ID_NO3"} in Table.FromRecords(List.TransformMany(Table.ToRecords(Table1),each List.RemoveNulls(List.Transform(Record.ToList(Record.SelectFields(_,ColumnListToTransform,1)),each Table2{[ID_NO=_]}?)),(x,y)=>Record.RemoveFields(x,ColumnListToTransform)&y))
        in
            Custom1