Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Ben-Dev
Helper II
Helper II

PQ in Azure Data Factory - Mashup Engine or Transposition

Hello,

The documentation for Azure Data Factory's 'Execute Power Query activity' states:
> To achieve scale with your Power Query activity, Azure Data Factory translates your M script into a data flow script so that you can execute your Power Query at scale using the Azure Data Factory data flow Spark environment.

Does this mean that, outside of ADF's Query Editor, there's no PQ mashup engine involved? In other words, that the saved M script is transposed into a Spark-powered data flow script and run entirely by Spark without a mashup engine executing?

 

Thanks!

 

1 ACCEPTED SOLUTION

so, when using PQ from ADF, instead of my M expression being run by a mashup engine, it is never touched by a mashup engine but instead the M expression is simply translated into an ADF data flow (Spark) script and run by Spark

 

That's correct.

View solution in original post

4 REPLIES 4
v-stephen-msft
Community Support
Community Support

Hi @Ben-Dev ,

 

The engine makes it easy for data movement and transformation. You could refer to the following article which has a good introduction.

What Is Power Query? Introduction to Data Mash-up Engine of Power BI - RADACAD

 

 

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Hi @v-stephen-msft,

 

Thanks for this info!

 

Actually, my question is specifically about how Power Query works when used from ADF, not how it works in general.

 

I understand that "normal" Power Query (e.g. the PQ that ships with Power BI and Excel) uses the PQ mashup engine to evaluate M expressions and produce the requested output. 

 

However, the quote from the above-mentioned article stating that ADF's PQ activity "translates your M script into a data flow script" makes me think that maybe the mashup engine is not involved at all when PQ is run from ADF—so, when using PQ from ADF, instead of my M expression being run by a mashup engine, it is never touched by a mashup engine but instead the M expression is simply translated into an ADF data flow (Spark) script and run by Spark.

 

Is that correct?

 

/CC: @Ehren 

so, when using PQ from ADF, instead of my M expression being run by a mashup engine, it is never touched by a mashup engine but instead the M expression is simply translated into an ADF data flow (Spark) script and run by Spark

 

That's correct.

Thank you!

Helpful resources

Announcements
Fabcon_Europe_Social_Bogo

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Top Solution Authors
Top Kudoed Authors