How to extract continuous string set from a column with multiple criteria

Greg_DecklerI have attached the current list of client names for your reference. You could use this list in your working file if it is convenient.

Drive Link 

Anonymous Here is an implementation of the logic I proposed. PBIX is attached below sig with your full dataset. It has the CleanMatchThreshold set to 5 but I think it works better at 4

Fuzzy Column = 
    VAR __MatchThreshold = 3
    VAR __CleanMatchThreshold = 5
    VAR __WordSearchTable = 
        GENERATE(
            'Ongoing Projects',
                VAR __Word = MAXX(FILTER('Ongoing Projects',[Index]=EARLIER('Ongoing Projects'[Index])),[Ongoing Projects])
            RETURN ADDCOLUMNS(GENERATESERIES(3,LEN(__Word),1),"Search",LEFT(__Word,[Value]))
        )
    VAR __Table =
        FILTER(
            ADDCOLUMNS(
                __WordSearchTable,
                "Match",SEARCH([Search],[Column1],,BLANK())
            ),
            NOT(ISBLANK([Match]))
        )
    VAR __Max = MAXX(__Table,[Value])
    VAR __Match = MAXX(FILTER(__Table,[Value]=__Max),[Search])
    VAR __Clean1 = IF(RIGHT(__Match,1)="(",LEFT(__Match,LEN(__Match)-1),__Match)
    VAR __Clean2 = IF(RIGHT(__Clean1,1)=" ",LEFT(__Clean1,LEN(__Clean1)-1),__Clean1)
RETURN
    IF(
        LEN(__Clean2)<=__CleanMatchThreshold,
        IF(
            LEN(__Clean2) = LEN([Column1]),
            __Clean2,
            BLANK()
        ),
        __Clean2
    )

Anonymous - Further improvement

Fuzzy Column = 
    VAR __MatchThreshold = 3
    VAR __CleanMatchThreshold = 4
    VAR __WordSearchTable = 
        GENERATE(
            'Ongoing Projects',
                VAR __Word = MAXX(FILTER('Ongoing Projects',[Index]=EARLIER('Ongoing Projects'[Index])),[Ongoing Projects])
            RETURN ADDCOLUMNS(GENERATESERIES(3,LEN(__Word),1),"Search",LEFT(__Word,[Value]))
        )
    VAR __Table =
        FILTER(
            ADDCOLUMNS(
                __WordSearchTable,
                "Match",SEARCH([Search],[Column1],,BLANK())
            ),
            NOT(ISBLANK([Match]))
        )
    VAR __Max = MAXX(__Table,[Value])
    VAR __Match = MAXX(FILTER(__Table,[Value]=__Max),[Search])
    VAR __Clean1 = IF(RIGHT(__Match,1)="(",LEFT(__Match,LEN(__Match)-1),__Match)
    VAR __Clean2 = IF(RIGHT(__Clean1,1)=" ",LEFT(__Clean1,LEN(__Clean1)-1),__Clean1)
RETURN
    IF(
        LEN(__Clean2)<=__CleanMatchThreshold,
        SWITCH(TRUE(),
            LEN(__Clean2) = LEN([Column1]),__Clean2,
            SEARCH(__Clean2,[Column1],,0)=1,__Clean2,
            BLANK()
        ),
        __Clean2
    )

Anonymous - How about this version? I have implemented a smarter matching algorthim taking into account how much of the total client name is matched. Also, there is an exception process in the SWITCH statement where you can call out specific matches, such as the example, "ABB". So if ABB is matched, it returns it. You could add additional rows to the SWITCH statement for other exceptions. I have the exception commented out because it is not needed with the current thresholds but left it there as an example. Updated PBIX attached.

So, basically the new rules are FuzzyThrehold1 = 30%, if 30% of the length is matched and it starts at the beginning, match. FuzzyThreshold2, if more than 80% of the total length is matched, it's a match.

Fuzzy Column = 
    VAR __MatchThreshold = 3
    VAR __CleanMatchThreshold = 5
    VAR __FuzzyThreshold1 = .3
    VAR __FuzzyThreshold2 = .8
    VAR __WordSearchTable = 
        GENERATE(
            'Ongoing Projects',
                VAR __Word = MAXX(FILTER('Ongoing Projects',[Index]=EARLIER('Ongoing Projects'[Index])),[Ongoing Projects])
            RETURN ADDCOLUMNS(GENERATESERIES(3,LEN(__Word),1),"Search",LEFT(__Word,[Value]))
        )
    VAR __Table =
        FILTER(
            ADDCOLUMNS(
                __WordSearchTable,
                "Match",SEARCH([Search],[Client Name],,BLANK())
            ),
            NOT(ISBLANK([Match]))
        )
    VAR __Max = MAXX(__Table,[Value])
    VAR __Match = MAXX(FILTER(__Table,[Value]=__Max),[Search])
    VAR __Clean1 = IF(RIGHT(__Match,1)="(",LEFT(__Match,LEN(__Match)-1),__Match)
    VAR __Clean2 = IF(RIGHT(__Clean1,1)=" ",LEFT(__Clean1,LEN(__Clean1)-1),__Clean1)
RETURN
    IF(
        LEN(__Clean2)<=__CleanMatchThreshold,
        SWITCH(TRUE(),
            //__Clean2 = "ABB",__Clean2,
            LEN(__Clean2) = LEN([Client Name]),__Clean2,
            LEN(__Clean2)/LEN([Client Name])>__FuzzyThreshold1 && SEARCH(__Clean2,[Client Name],,0)=1,__Clean2,
            LEN(__Clean2)/LEN([Client Name])>__FuzzyThreshold2,__Clean2,
            BLANK()
        ),
        __Clean2
    )

Anonymous - I would have to think about how to implement that within the program, but what about Ernst & Young, that would only match Ernst then? Perhaps I am not understanding something.

Anonymous - I do not see how that logic is going to prevent something like International from being mapped to National for example. International will map to National Grid up until national, no space or G and so...

With the setting of .18 for threshold 1, is the only outstanding exceptions these:

Anonymous - Yes, but "blah blah International blah" is still going to match the "National" in "National Grid" this way and be designated "National" in the match. I don't see that logic preventing this or explain where my thinking is flawed.

Are you saying that if the Ongoing Project has a space in it to only match if the match goes beyond the space? Because I'm not sure how to implement that although I have some thoughts. However, even so that still doesn't fix the problem below with WOLTERSKLUWERS.

I was able to resolve the National issue though with some additional matching threholding checks. The only thing that I see that is an exception at this point is that the client "WOLTERSKLUWERS" matches the "Wolters" in "Wolters Kluwer" when you probably want it to say "Wolters Kluwer". I don't know of a way around that problem. 

Updated PBIX attached. Here is the latest:

Fuzzy Column = 
    VAR __MatchThreshold = 3
    VAR __CleanMatchThreshold = 4
    VAR __FuzzyThreshold1 = .18
    VAR __FuzzyThreshold2 = .8
    VAR __WordSearchTable = 
        GENERATE(
            'Ongoing Projects',
                VAR __Word = MAXX(FILTER('Ongoing Projects',[Index]=EARLIER('Ongoing Projects'[Index])),[Ongoing Projects])
            RETURN ADDCOLUMNS(GENERATESERIES(3,LEN(__Word),1),"Search",LEFT(__Word,[Value]))
        )
    VAR __Table =
        FILTER(
            ADDCOLUMNS(
                __WordSearchTable,
                "Match",SEARCH([Search],[Client Name],,BLANK())
            ),
            NOT(ISBLANK([Match]))
        )
    VAR __Max = MAXX(__Table,[Value])
    VAR __Match = MAXX(FILTER(__Table,[Value]=__Max),[Search])
    VAR __Clean1 = IF(RIGHT(__Match,1)="(",LEFT(__Match,LEN(__Match)-1),__Match)
    VAR __Clean2 = IF(RIGHT(__Clean1,1)=" ",LEFT(__Clean1,LEN(__Clean1)-1),__Clean1)
RETURN
    IF(
        LEN(__Clean2)<=__CleanMatchThreshold,
        SWITCH(TRUE(),
            //__Clean2 = "ABB",__Clean2,
            LEN(__Clean2) = LEN([Client Name]),__Clean2,
            LEN(__Clean2)/LEN([Client Name])>__FuzzyThreshold1 && SEARCH(__Clean2,[Client Name],,0)=1,__Clean2,
            LEN(__Clean2)/LEN([Client Name])>__FuzzyThreshold2,__Clean2,
            BLANK()
        ),
        SWITCH(TRUE(),
            __Clean2 = "Blue Cross",__Clean2,
            LEN(__Clean2)/LEN([Client Name])<__FuzzyThreshold2 && SEARCH(__Clean2,[Client Name],,0)<>1,BLANK(),
            __Clean2
        )
    )

Anonymous Except it doesn't work that way. To implement the original algorithm in DAX I had to do this:

Let's take just the first 2 client names

In order to emulate looping, which is impossible in DAX, this becomes:

and so on, the entire list of projects gets blown out into a big table like this. Then, I go about adding a column to this table that says whether it matches the client or not, basically a 1 or a 0. Then, I filter out the zeros. Then I grab the MAX of the second column, that is my longest match.

This is looping in DAX.

So, explain again how your logic fits into this?

Steelwave is a Client Name and it matches Sterling up until 3 characters. The only match returned is Sterling, not Sterling and Steelwave because Steelwave is not an Ongoing Project name. So, you are left with Steelwave matching Sterling up until Ste. You can lower the thresholds but then you start running into issues with Dell, etc.

Anonymous - OK, this might actually be the one. I added a KillThreshold that was possible because I moved the cleaning to the end to maximum the length of the match. I got rid of the "Unit" match also. So, basically once I adjusted the cleaning of the special characters and spaces at the end, I was able to increase Threshold1 so that ABB is included but Unit is excluded. The only special exception is for Blue Cross and I'm not sure how else to handle that one quite honestly. Let me know if you find any obvious issues. Updated PBIX attached.

Fuzzy Column = 
    VAR __MatchWord = [Client Name]
    VAR __CleanMatchThreshold = 4
    VAR __KillThreshold = 3
    VAR __FuzzyThreshold1 = .4
    VAR __FuzzyThreshold2 = .8
    VAR __WordSearchTable = 
        GENERATE(
            'Ongoing Projects',
                VAR __Word = MAXX(FILTER('Ongoing Projects',[Index]=EARLIER('Ongoing Projects'[Index])),[Ongoing Projects])
            RETURN ADDCOLUMNS(GENERATESERIES(3,LEN(__Word),1),"Search",LEFT(__Word,[Value]),"Original",__Word)
        )
    VAR __Table =
        FILTER(
            ADDCOLUMNS(
                __WordSearchTable,
                "Match",SEARCH([Search],__MatchWord,,BLANK())
            ),
            NOT(ISBLANK([Match]))
        )
    VAR __Max = MAXX(__Table,[Value])
    VAR __Match = MAXX(FILTER(__Table,[Value]=__Max),[Search])
    VAR __Proposed =
        IF(
            LEN(__Match)<=__CleanMatchThreshold,
            SWITCH(TRUE(),
                //__Clean2 = "ABB",__Clean2,
                COUNTROWS(FILTER(__Table,[Value]=__Max))>1,BLANK(),
                LEN(__Match) <= __KillThreshold,BLANK(),
                LEN(__Match) = LEN(__MatchWord),__Match,
                LEN(__Match)/LEN(__MatchWord)>__FuzzyThreshold1 && SEARCH(__Match,__MatchWord,,0)=1,__Match,
                LEN(__Match)/LEN(__MatchWord)>__FuzzyThreshold2,__Match,
                BLANK()
            ),
            SWITCH(TRUE(),
                __Match = "Blue Cross" || __Match = "Blue Cross ",__Match,
                LEN(__Match)/LEN(__MatchWord)<__FuzzyThreshold2 && SEARCH(__Match,__MatchWord,,0)<>1,BLANK(),
                __Match
            )
        )
        VAR __Clean1 = IF(RIGHT(__Proposed,1)="(",LEFT(__Proposed,LEN(__Proposed)-1),__Proposed)
        VAR __Clean2 = IF(RIGHT(__Clean1,1)=" ",LEFT(__Clean1,LEN(__Clean1)-1),__Clean1)
RETURN
    __Clean2

Anonymous - So, yes, we can return the entire matching project, that we can do. I will see what I can do with that. The problem with Blue Cross is that the client names are so long that it ends up not meeting the threshold requirements for a match. I may be able to adjust those. So, for example 

Blue Cross and Blue Shield of Arizona I

Matches 11 out of like 40 characters or .275% This makes it fall below thresholds set to filter out other stuff. The Ernst & Young entry for Blue Cross is even worse.

Anonymous - I will take a look.

Anonymous Glad we got there, was an interesting problem. There are still replies in this thread you haven't kudo'd... 🙂

This may be my record for longest thread before solution, 6 pages!!!

Greg_DecklerI have attached a final Client name file. The operation needs to be done on this fine. Ongoing project List remains the same. My results on this final client file returns some unwanted data like Anders,Unit,Con,Comm,National,Nor,Tri,Ste,Mar.
This is where I see that the threshold does not be effective. Not able to think of a way to improvise on the formula to eliminate these errors.

The good note is that the retuned blank values are perfectly the way they should be since there are no matches for them in the Project list.

Drive Link 

Greg_DecklerThe logic I think will work is , instead of giving the matches by taking both the Client name and project columns into consideration, can we only take Project column into consideration and provide the maximum match term with project column from the client name?

Example: 

Client Name       Project Name

National Grid      National Grid

First National      ABB

Operation to follow as below:

Take National Grid for consideration.

Check for 'N' in all cells of Client Name and save this list in a variable.

Check for 'Na' in this saved list and update this variable with the new lesser list.

Check for 'Nat' in this newly saved list and update the variable......... and so on until we check for 'National G'.
We have to stop with the first character after the first occurance of space because if not then the return values will need to be an exact match with the client name.
This above logic will ensure that National Grid is mapped only to National Grid and not First National.

Now, The above step is iterated by taking ABB into consideration. 

This way we can ensure that every Project Name is mapped exactly to one item from the Client Name and the retun value will be the Client Name and not the matching strings.

This was my initial proposal for the PHP code but I am not sure how I lost the course on this. Probably I got overwhelmed with the results I recieved with DAX.

Is there a way to try the above logic with PBI? This seems to be fool proof atleast for the current set of Client Names and considering the way they appear, it could work on any additional clients that may add in to this list.

I need to submit this presentation in another 12 hours , i.e 12:00 PM. Hope this method is possible.

Greg_Deckleryou almost got it but the below two are a few exceptions or are getting mapped wrongly.

1. Client Name:  'Bose'
                          'Dell'
                          'Casey'   

The above three Projects are not mapped to any clients even though these clients are present in the client list. Probably because of the minimum threshold set.

2. National is being mapped to more than 'National Grid' client (there are three more client names with the term National in their name)

I do not want to create any exceptions unless they are exhaustive in nature, because I am planning on having a readily automated report pushed to the service.

I still feel the earlier logic that I mentioned in my previous post is pretty much fool proof. considering the given conditions and appearance of the client and project data.

Greg_DecklerIt will match 'Ernst &' and if the match is present in the Client Name then it will return that Client Name which will be the whole name.
The match reference should be picked from the project list and when the string matches with a value in Client NAme, The Client Name should be returned. 

Let me know , I am also trying to improvise on your formula as we speak.

Greg_Deckler 
So for 'National Grid',  search operation up until 'National ' there will be all 5 mappings as seen in the screenshot you shared.
Until now the retun values for the increment of characters to Project Name are stored in a variable.
Now increment to the next character 'G' to search for 'National G' in this variable list consisting the 5 mappings.

Since 'National G' matches only with 2 mappings now , These two mappings will overwrite the earlier 5 mappings or the operation eliminates the not matching 3 mappings (whichever is efficient) and the variable list now returns National Grid only for these 2 client names which is fine because both are National Grid Client.

This way, the search can be performed on all the cells in Project List up until 'The first character after the first space'. 

Greg_DecklerThe latest file still have some flaws. Fuzzy Column: 'Tri', "Ste', 'Unit' these three have no ongoing projects, They need to be blank.

If we implement the logic that I proposed, then these would return blank. Since no projects are there with the string 'Tri P' , Search for characters in Project Name 'Sterling' will return only Sterling and 'Ste' will not get mapped to Steelwave, 'Unit' will not get mapped to 'United' -> similar to Sterling.
Explanation on Sterling:
While searching for the characters in cell 'Sterling' , up until 'Ste' the retun values will be Sterling and Steelwave. Now add the next character and search again. 'Ster' will return only 'Sterling' and eliminate Steelwave leaving it as a blank.
From what I understand it is a teeny bit of reverse mapping from that what we are following now.
Instead of searching for the whole string of Project Name, and returning all matches with the matching strings, why dont we set a criteria to search only until first character after first space in the Project Name, return all matching client names to the string set increment. 

Basically as the increment of the character for the cell by cell Project Name happens, the number of matching clients to this string will reduce leaving just 1 or 2 matching client names. 

Basically consider this, A is Project name.

A = "Barclays L"  --> taking only the characters until the first one after first space. Increment (i++) should stop after the first character after first space if found or end of string without any spaces.

for (A[0]='B', search for matches in client name,i++)

B=All client names matching A[0].
Now for (A[0][1]='Ba', search for matches in client name, i++)

B=All client names matching A[0][1]
.

.

.

Now for (A[0][1]...[8],search for matches,i++)

B=Barclays Corporation -> Client name matching to the searched project Name (retuning Client Name instead of matching strings)

The loop ends at [8] because A0]...[9] = 'Barclays L' is not present in the list of client names. Hence matching Barclays is mapped until A[8] and the mapped client name is Barclays Corporation is returned.

I hope I was able to convey the logic. My apologies for the above way of mentioning the logic.

And Dont worry about Wolters. It is the client name error. There has to be a space after Wolter.

Greg_DecklerYou are right. My logic has some flaws in case where the Project name does not appear in the client list

Greg_DecklerThank you Greg. You are truly a genius. I will go ahead and use this for my presentation today. This is really helpful. Thank you for all your time.
However I do not understand just one thing. Why was the exception given to Blue Cross?

Even National Grid has two matches with client names similar to Blue cross. While this works for National Grid, why is it not working for Blue Cross?

(See -Ernst and YoungNATIONAL Grid     and Ernst and YoungBLUE Cross   under client names for which blue cross exception was added). While National grip is getting mapped to YoungNATIONAL Grid, Blue Cross should get mapped to YoungBLUE Cross since the client names are of similar nature or structure. Any thoughts on why this is happening? Then may be we can figure out to eliminate giving this exception as well and have a perfectly working formula without any exceptions 🙂
Let me know you thoughts. I will go ahead and accept this as a solution post your confirmation on this piece.

Greg_Deckler  Presented today with the exception but obviously I did not mentioned about the exception in the code. However I was thinking, instead of retunring only matched characters and then clean it, can we have the formula return the entire cell value from the Project list?

For example, instead of 'Casey' which has been matched, since 'Casey' is matched, the return value will be the entire cell value of 'Casey' i.e; Casey's General Store (CARES 2019) --> From the project list.

This should be similar for other matches.

Now I am aware of multiple names in project list (For example: Barclays has 3 variations in the project list). In such cases let the retun value be the first occurance od this match term. i.e; Barclays(VDA).
Sorry for trying to improvise on the initial requirement but could you suggest me a formula for the above?

We can mark the response as a solution since I am satisfied with the results. (Except for my initial concern to avoid giving exception to 'Blue Cross'). You could also clarify why Blue Cross is behaving differently than 'National Grid'. 

Thanl you so much for all the time Greg_Deckler  I really appreciate. You are the saviour 🙂 

Greg_Deckler  For some wierd reason, I added data to my datadump today and after refresh I dont see Blue Cross and Blue Shield of Arizona I as well as The Ernst & Young entry for Blue Cross . I only see BLUE CROSS OF IDAHO CARE PLUS, INC in the data dump (excel). However when I filter in the data view of the BI desktop under list of projects, there are two entries for Blue cross. Idaho care plus and the Ernst and Young Entry. I am not sure what could be a reason for this. I dont see Blue Shield entry in my entire data set nor in the data view. Have you come across such odd things?

On the return value, we could either return the Project Name for the match or the Client Name for the Match. (either works - Better to know when to change what for convenience).

I am still trying to understand some part of the code. May be I can play around with the threshold values and get a match for Blue cross also for all kind of entries currently present. That is if you dont come up with one earlier than me 🙂

Please share the latest improvements you could make on the existing one to have better results. Hopefully the next post from you could be our very much needed near to perfect solution or even better - A perfect Solution 😉

Greg_DecklerGenius!! Thank you so much for this. You have literally spared me a grave headache. Wish I could give you a dozen Kudos haha!! Peace!

Hi Greg, pleasure in greeting you... 2 queries:

1. Does the Microsoft Power BI Cookcbook (2nd Edition) have a Spanish version?

2. In the book, will you have cases to identify ranges of text and texts with momodines?

Example, if I have the value "AC23", "AJ80" and "KRZ600".

AC23 = Case1 and Case2

AJ80 = Caso2

KRZ600 = Caso3

Thanks and greetings...