dataflow gen2 applied steps - red mark

arkiboys2 — Sat, 02 Nov 2024 09:14:01 GMT

hi,

although I see all results I am unable to find out why there is a red mark here

any suggestions how to tackle this? and should I ignore this since there is result coming back?

in th eexpression below I am trying to remove ":" if it is present at the beginning or end

thank you

Re: dataflow gen2 applied steps - red mark

frithjof_v — Sat, 02 Nov 2024 21:12:45 GMT

The red icon you're seeing is a common indicator, indicating that the step won't fold into a source database query. Instead, the step will be processed in the Power Query M engine's memory.

The red icon does not indicate that anything will fail. So you don't need to worry about that.

However, this icon indicates that this part of the query will not be able to fold. This is about query execution and performance. It is a best practice to keep it green as long as possible (keep as many applied steps as possible green). If possible, consider moving the "Added custom" step to the bottom of your query.

See some useful details here: https://learn.microsoft.com/en-us/power-query/step-folding-indicators

It's a good practice to arrange the applied steps so the M query begins with all the steps that fold (the green steps).

Typically, we use Choose columns (use Choose Columns or Remove Other Columns instead of Remove Columns, as a best practice) and Filter as our first applied steps. Several other M operations are also able to fold.

The ability for M to perform query folding depends on which source system you are using. Typically, SQL-based source systems and relational databases will have good support for query folding of several M operations. For some other sources, M doesn't support query folding or has limited support for it.

A good practice is also to split the queries into staging and transformation. For example, you could have one dataflow purely for ingestion (consider using Choose columns and Filters to reduce the amount of data ingested), and another dataflow/stored procedure/notebook/etc. for transformations.

As with everything, there is an it depends element here as well. But these are generally considered good practices.

Bottom line: the indicator is just there for query execution (query folding) information.

For more information, google "Query folding" or "don't break the fold".

Re: dataflow gen2 applied steps - red mark

arkiboys2 — Sun, 03 Nov 2024 09:45:26 GMT

thank you

topic dataflow gen2 applied steps - red mark in Data Engineering

dataflow gen2 applied steps - red mark

Re: dataflow gen2 applied steps - red mark

Re: dataflow gen2 applied steps - red mark