SQL? DAX? Machine Learning? or what?

[EricM]

I have a little problem for which I am wondering what would be the right tool...

I have several versions of a dataset.

Each version contains between 500,000 rows and a million.

Each dataset is divided in three hierarchies

• Time (Year/Quarter/Month)
• Geography (Region/Country)
• Attributes (Level 1, Level 2, Level 3, Level 4, etc)

I would like to compare 2 versions of the dataset and spot whether there are differences of more than X percent at any level.

I could do that in SQL but it would be quite slow...

I'd have to loop through all potential combinations of the three hierarchies, etc... 

Feasible but tedious and slow.

I could also do this in DAX but I might also have to loop through all possible hierachy levels.

Feels as tedious even if probably quicker.

What about machine learning?

I don't have experience of it but it "feels" like it should be able to do that sort of thing.

Or should I use an entirely different approach?

Thanks

Eric

[v-sihou-msft · Accepted solution]

@EricM

In this scenario, since your two datasets same metadata and same number of rows, you can build the relatioinship between two tables and add a calculated column to tag if the fact data is different or not.

IfDifferent = IF(Table1[Column]=RELATED(Table2[Column]),0,1)

Then you just sum above column together, divide by count of total rows. To calculated it on different level, you just need to use ALLEXCEPT() as filters to group your calculation on different level.

Diff Pct On Year = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Year]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Year]))

Diff Pct On Month = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Month]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Month]))

Regards,

[Greg_Deckler]

Could potentially be machine learning falling under "Anomoly Detection" but I would think that you could create a Measure that calculates your % and then through that into a matrix with your hierarchy. Could potentially use the new Top N filter to view highest % differences. Could you supply some mock data and expected result?