new dataflow connector not brings data in preview mode in Power Query when I connect to dataflow

[Anonymous]

We have a dataflow source that is a bit large (9 million rows) I was so excited with the new dataflow connector as we can take the advantage of query folding feature. But when I try to connect the dataflow source with the new connector, it says "WAITING FOR POWERPLATFORMDATAFLOWS" in Power Query Level, and after half an hour it says "Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647" . I can't apply any steps at the Power Query level, because data is not coming to the power query screen. I didn't see any issue with the small tables but I can't use the new connector with the large tables.

[Anonymous]

I am using the new Power Platform Dataflow connector announced in August 2021 release.  I am using the connector in a dataflow to reference another dataflow.  All dataflows are in Premium Gen2 with Enhance Computed on.

I can create a simple dataflow with a small table and it works fine.  I then try to add a second table with 20 million rows.  That times out after 10 minutes during validation (see image).  I can still save the dataflow without validation and then I am able to refresh it successfully.

However, I am unable to do any additional dataflow transformations because the large table is not validated.  If I let the dataflow creator window timeout and then use the advanced editor to add a merge trahsformation, I can save the dataflow (see below digram).  But when I try to refresh the dataflow, it fails with following error.

Error: DataSource.MissingClientLibrary: ADO.NET: Unable to find the requested .Net Framework Data Provider.  It may not be installed. DataSourceKind = PowerPlatformDataflows DataSourcePath = PowerPlatformDataflows. RootActivityId = df7c1fa9-5510-4383-96f4-9c5fd2b9f1d6.Param1 = DataSource.MissingClientLibrary: ADO.NET: Unable to find the requested .Net Framework Data Provider.  It may not be installed. DataSourceKind = PowerPlatformDataflows DataSourcePath = PowerPlatformDataflows Request ID: ccb5aa98-c69e-5e5a-077f-615dddfe0f65.

I suspect all the above behaviors are related but cannot figure out what is wrong or how to fix.

[lbendlin]

Try increasing the memory limits on your PC

[Anonymous]

Hi @lbendlin ,

Thanks for your response. My computer has 32 GB Ram. I tried multiple combinations like 1/28672, 2/14336, 4/7168 but I still have same error message

[lbendlin]

Maybe there is a 2GB buffer issue at your data source?

[Anonymous]

I think the dataflow source is around 3 GB. Is there any limitation for the size of dataflow if so is there any documentation for this? (I am not aware if I can increase the buffer size, I am working in the business unit but I can contact Digital, maybe they can change settings) Because the old connector is working well but I just want to reduce the refresh times in the development stage.