topic Re: saturation occurred at DataFlowGen2 in Dataflow

saturation occurred at DataFlowGen2

DALI007 — Fri, 23 Feb 2024 09:19:51 GMT

Hello ,

I work with a DataflowGen2 to process extremely large tables. However, after 7 hours of data recovery, a saturation occurred at DataFlowGen2. To resolve this issue, I would like to know if it is possible to schedule a run time out or limit the volume of recovered data.
Could you help me set this up, please?

Thank you

Re: saturation occurred at DataFlowGen2

Anonymous — Fri, 23 Feb 2024 11:05:27 GMT

Hi DALI007

Thanks for using Fabric Community.

Can you please provide more details about the ask? What do you mean by saturation here?

There is no way to set a timeout limit, you can apply a row limit. You can do this by taking x number of rows from the source.

Re: saturation occurred at DataFlowGen2

DALI007 — Fri, 23 Feb 2024 11:25:10 GMT

Hello Anonymous ,
Thank you for your quick feedback
thank you for your quick feedback
After 7hour data recovery I got an error message. can you tellme what parametrage to do to make a line number limit please ?
Thank you

Re: saturation occurred at DataFlowGen2

Anonymous — Fri, 23 Feb 2024 11:41:12 GMT

Hi DALI007

Please refer to this link: https://learn.microsoft.com/en-us/power-query/filter-row-position

This might help you. Please let me know if you have any further questions.

Re: saturation occurred at DataFlowGen2

Anonymous — Mon, 26 Feb 2024 11:04:40 GMT

Hi DALI007
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: saturation occurred at DataFlowGen2

DALI007 — Mon, 26 Feb 2024 14:15:10 GMT

Hello Anonymous,

Thank you for your feedback. Unfortunately, the problem persists. I would like to know if it is possible to configure the DataFlowGen2 to retrieve only a limited number of data from all sources. Or, I wanted to know if it is possible to apply a specific setting to certain users only.

Thank you

Re: saturation occurred at DataFlowGen2

miguel — Mon, 26 Feb 2024 16:16:03 GMT

You can set filters for your queries as needed and limit things that way.

These two articles might help you with how to filter data in Power Query:

Filter a table by row position - Power Query | Microsoft Learn

Filter by values in a column - Power Query | Microsoft Learn

You have full control over what data should be queried to your data source and, if the data source supports query folding, then that should limit what data is being requested to the data source as well.

You can read the article below to learn more about query evaluation and query folding in Power Query:

Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn

Re: saturation occurred at DataFlowGen2

DALI007 — Mon, 26 Feb 2024 16:58:38 GMT

Hello miguel,

Thank you for your feedback. If I understood correctly, you mentioned the use of Power Query on a specific table in your message. However, I would like to know if it is possible to apply this setting automatically to all tables within Data Flow Gen2. In other words, I want this parameter to be applied automatically to each recovery table in Data Flow Gen2.

So I wanted to know, how to make this setting can be applied to all Data Flow Gen2 of the same workspace.

Thank you very much

Re: saturation occurred at DataFlowGen2

miguel — Mon, 26 Feb 2024 17:15:23 GMT

Not a feature that exists today, but you can definitely suggest this as a new feature by posting your idea in the Fabric Ideas portal (https://aka.ms/FabricIdeas).

At the moment if you wish to set a particular limit or want to take advantage of query folding in any particular way, you will have to do it at the query level inside of the Power Query editor. Don't forget that it also depends on how things will get evaluated, so its important to understand what queries might benefit from such filter and what others might not get any sort of benefit from it due to how query folding wouldn't impact it or simply how query folding isn't supported for a particular data source.

Re: saturation occurred at DataFlowGen2

DALI007 — Tue, 27 Feb 2024 09:33:45 GMT

Hello miguel,
I added this idea on fabric community.
Thanks for your help.

Re: saturation occurred at DataFlowGen2

DALI007 — Tue, 27 Feb 2024 09:34:56 GMT

Hello miguel
I added this idea on fabric community.
Thanks for your help.

Re: saturation occurred at DataFlowGen2

DALI007 — Wed, 28 Feb 2024 00:47:00 GMT

Hello miguel ,
The article you gave me talks about power query. and more precisely it talks about a parametrage to make level of each table. I wanted to know if there is a possibility to limit the number of fetch lines at the PowerBI Service source connection please.
Thank you for your help

Re: saturation occurred at DataFlowGen2

miguel — Wed, 28 Feb 2024 01:58:51 GMT

Power Query is the tool used by Dataflows and Power BI to get the data. The article showcases how you can leverage Power Query to implement your own filters and limits when connecting to a source.

Something like "Top 1000 rows" would yield the top 1000 rows of the result of your query. It also depends if your query can fully fold, but conceptually that's the sort of logic that you can implement at each query in your Dataflow, Power Query or Power BI Semantic Model.