Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Be one of the first to start using Fabric Databases. View on-demand sessions with database experts and the Microsoft product team to learn just how easy it is to get started. Watch now

Reply
robarivas
Post Patron
Post Patron

Efficiently create junk dimensions in power query

I have a large fact table (many millions of rows). It has a bunch of columns in it that I'd like to break out into a few junk dimension tables. However, I have no keys to work with so I was planning on creating surrogate primary keys fo each junk dimension table. However, this means I then have to merge these tables back on to the fact table so that the fact table can obtain the proper foreign keys. I suspect this sort of multiple (self) joins/merges might bring the loading and refresh process to its knees. So is there a best practice way to acheive this whole procedure in an efficient (hopefully quick) way? I'd like to do it all in power query if possible. The fact table lives in a remote SQL Server.

1 ACCEPTED SOLUTION

Hi @robarivas 

yes, doing everything on the server-side would probably be the fastest version.

It's not that PQ cannot do it, but SQL would be faster with it. Just create views for it that you then query.

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

View solution in original post

3 REPLIES 3
ImkeF
Super User
Super User

Hi @robarivas 

Multiple levels of merge tend to become slow in PQ. Why not generate some views on the SQL Server already?

Or write them as SQL-queries in PQ.

 

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

Thank you @ImkeF . Just to clarify: Are you saying I should create the junk dimensions on SQL server? That seems easy and logical. However, wouldn't I still need to merge those tables back to the fact table so that the fact table can obtain the keys. Or are you saying that this step should also be done on the SQL Server side?

Hi @robarivas 

yes, doing everything on the server-side would probably be the fastest version.

It's not that PQ cannot do it, but SQL would be faster with it. Just create views for it that you then query.

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.

ArunFabCon

Microsoft Fabric Community Conference 2025

Arun Ulag shares exciting details about the Microsoft Fabric Conference 2025, which will be held in Las Vegas, NV.

December 2024

A Year in Review - December 2024

Find out what content was popular in the Fabric community during 2024.