Forum Discussion

devaraj's avatar
devaraj
Regular Visitor
1 year ago
Solved

Creating custom column to count rows from 2nd table in power query

Hello, 

I am looking of a powerquery to create a custom column to acheive below column from two different tables, without using dax formulas or merging the tables 

 

table 1  
idrefrence idcustom column for count of valid value id
1232152
1237582
1236482
4565361
4564291
7897640

 

table 2
value idstate
215valid
758valid
648invalid
536invalid
429valid
764invalid

Custom column for count of valid value is the columns I want to acheive using power query without merging table 2 to table 1, as I have millions of records and its quite perfomamce issue when merging the tables and sont want to use dax formulas too. Is there a possiblity to get this column using powerquery ?

thanks

Devaraj

  • Hi devaraj 

     

    The code below doesn't use merge

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQyVtJRMjI0BZFKsTowEXNTCzQRMxOEiImpGZBtagwiDZFETIws4SLmFiC2uZkJkDRQio0FAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [id = _t, #"reference id" = _t, #"custom column for count of valid value id" = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"id", Int64.Type}, {"reference id", Int64.Type}, {"custom column for count of valid value id", Int64.Type}}),
        #"Added Custom" = Table.AddColumn(#"Changed Type", "Reference IDs", each let 
    PrevStep = #"Changed Type",
    CurrentID = [id],
    FilteredRows = Table.SelectRows(PrevStep, each [id] = CurrentID)[reference id]
    in FilteredRows),
        #"Added Custom1" = Table.AddColumn(#"Added Custom", "Count", each let 
    t2 = table2, //actual table2 name
    filteredT2 = Table.SelectRows(t2, each [state] = "valid"),
    refID = filteredT2[value id],
    count =List.Count(List.Intersect({[Reference IDs],refID}))
    in 
    count),
        #"Removed Columns" = Table.RemoveColumns(#"Added Custom1",{"Reference IDs"})
    in
        #"Removed Columns"

     

    As I mentioned in my initial response, whether you perform a merge or not, this process will be very slow and could even result in a timeout if applied to millions of records. You’ll need to either handle this in DAX (though it may still be slow) or perform the transformation at the source, rather than within Power BI.

8 Replies

    • devaraj's avatar
      devaraj
      Regular Visitor

      hi Ashish_Mathur the numbers 2,2,2,1,1,0 are  the count values are coreesponding to id column, id 123 has two valid id state,

  • Hi devaraj 

     

    It is unclear how you came up with the expected result column - reference id 215 has one valid row in table2 but your result is 2. Also, whether you merge or not, this is going to be slow if you have millions of records. This transformation is best done at the source. Alternatively, I would use DAX for this but still expect for some hiccups.

    • devaraj's avatar
      devaraj
      Regular Visitor

      hi danextian the count values are coreesponding to id column, id 123 has two valid id state,

      • danextian's avatar
        danextian
        Icon for Super User rankSuper User

        Hi devaraj 

         

        The code below doesn't use merge

        let
            Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQyVtJRMjI0BZFKsTowEXNTCzQRMxOEiImpGZBtagwiDZFETIws4SLmFiC2uZkJkDRQio0FAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [id = _t, #"reference id" = _t, #"custom column for count of valid value id" = _t]),
            #"Changed Type" = Table.TransformColumnTypes(Source,{{"id", Int64.Type}, {"reference id", Int64.Type}, {"custom column for count of valid value id", Int64.Type}}),
            #"Added Custom" = Table.AddColumn(#"Changed Type", "Reference IDs", each let 
        PrevStep = #"Changed Type",
        CurrentID = [id],
        FilteredRows = Table.SelectRows(PrevStep, each [id] = CurrentID)[reference id]
        in FilteredRows),
            #"Added Custom1" = Table.AddColumn(#"Added Custom", "Count", each let 
        t2 = table2, //actual table2 name
        filteredT2 = Table.SelectRows(t2, each [state] = "valid"),
        refID = filteredT2[value id],
        count =List.Count(List.Intersect({[Reference IDs],refID}))
        in 
        count),
            #"Removed Columns" = Table.RemoveColumns(#"Added Custom1",{"Reference IDs"})
        in
            #"Removed Columns"

         

        As I mentioned in my initial response, whether you perform a merge or not, this process will be very slow and could even result in a timeout if applied to millions of records. You’ll need to either handle this in DAX (though it may still be slow) or perform the transformation at the source, rather than within Power BI.

  • v-sathmakuri's avatar
    v-sathmakuri
    Icon for Community Support rankCommunity Support

    Hi devaraj ,

     

    Thank you for reaching out to Microsoft Fabric Community.

     

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

     

    Thank you.

  • v-sathmakuri's avatar
    v-sathmakuri
    Icon for Community Support rankCommunity Support

    Hi devaraj ,

     

    I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the responses has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

     

    Thank you!!

  • v-sathmakuri's avatar
    v-sathmakuri
    Icon for Community Support rankCommunity Support

    Hi devaraj ,

     

    I hope this information provided is helpful. Feel free to reach out if you have any further questions or would like to discuss this in more detail. If responses provided answers your question, please accept it as a solution so other community members with similar problems can find a solution faster.

     

    Thank you!!