Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
EricM
Helper I
Helper I

SQL? DAX? Machine Learning? or what?

I have a little problem for which I am wondering what would be the right tool...

 

I have several versions of a dataset.

Each version contains between 500,000 rows and a million.

 

Each dataset is divided in three hierarchies

  • Time (Year/Quarter/Month)
  • Geography (Region/Country)
  • Attributes (Level 1, Level 2, Level 3, Level 4, etc)

I would like to compare 2 versions of the dataset and spot whether there are differences of more than X percent at any level.

 

I could do that in SQL but it would be quite slow...

I'd have to loop through all potential combinations of the three hierarchies, etc... 

Feasible but tedious and slow.

 

I could also do this in DAX but I might also have to loop through all possible hierachy levels.

Feels as tedious even if probably quicker.

 

What about machine learning?

I don't have experience of it but it "feels" like it should be able to do that sort of thing.

 

Or should I use an entirely different approach?

 

 

Thanks

 

Eric

1 ACCEPTED SOLUTION
v-sihou-msft
Employee
Employee

@EricM

 

In this scenario, since your two datasets same metadata and same number of rows, you can build the relatioinship between two tables and add a calculated column to tag if the fact data is different or not.

 

IfDifferent = IF(Table1[Column]=RELATED(Table2[Column]),0,1)

 

Then you just sum above column together, divide by count of total rows. To calculated it on different level, you just need to use ALLEXCEPT() as filters to group your calculation on different level.

 

Diff Pct On Year = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Year]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Year]))
Diff Pct On Month = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Month]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Month]))

Regards,

 

View solution in original post

2 REPLIES 2
v-sihou-msft
Employee
Employee

@EricM

 

In this scenario, since your two datasets same metadata and same number of rows, you can build the relatioinship between two tables and add a calculated column to tag if the fact data is different or not.

 

IfDifferent = IF(Table1[Column]=RELATED(Table2[Column]),0,1)

 

Then you just sum above column together, divide by count of total rows. To calculated it on different level, you just need to use ALLEXCEPT() as filters to group your calculation on different level.

 

Diff Pct On Year = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Year]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Year]))
Diff Pct On Month = 
CALCULATE(SUM(Table1[IfDifferent]),ALLEXCEPT(Table1,Table1[Month]))
/
CALCULATE(COUNTROWS(Table1),ALLEXCEPT(Table1,Table1[Month]))

Regards,

 

Greg_Deckler
Super User
Super User

Could potentially be machine learning falling under "Anomoly Detection" but I would think that you could create a Measure that calculates your % and then through that into a matrix with your hierarchy. Could potentially use the new Top N filter to view highest % differences. Could you supply some mock data and expected result?


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
The Definitive Guide to Power Query (M)

DAX is easy, CALCULATE makes DAX hard...

Helpful resources

Announcements
LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.