How to copy large datasets with dataflows when dynamic datasources are not supported?

I disagree.

SSIS is several generation older version of Fabric and Powershell is not any ETL tool.

I tried to submit this as an idea https://ideas.fabric.microsoft.com/ideas/create-new-idea/:

Hello,

My problem is that I have too much data for dataflow to handle.

And my other problem is that you don't let me program a solution that let's me to use dataflows despite of that.

I want to read XML files from a REST API.

But there is about a million of files that I need to read.

I can divide the list of file to read with a common date parameter: I read files for a single day only, store them to (file or) database and then read again the previous day starting from today backwards.

So I programmed the dataflow so that I have a parameter which defines which days content I fetch.

So far everything works well.

But next task is either:

• Make a loop inside dataflow in where the memory will be reused, or
• Make a loo from pipeline --> notebook --> dataflow

I don't know how to make the first one, so I tried the second one, but found out that you are not letting me to do this.

When I publish my dataflow, I get error: "dynamic datasources are not supported" (thanks for giving the error at this point already and not on execution)

So my suggestion is one of the following:

• Remove this security setup. There are valid reasons why we need to query dynamic data sources. The fact that dataflow works fine until publishing indicates that this is probably not an architecture limitation, but security decision, which while understandable is in reality preventing us using Fabric.
• Create a (complex) mechanism to allow notebooks to call dataflows bypassing this rule.
• Do both of the above. Do the first one ASAP as short term solution and second one when it fits in the development queue.

More information on my case is here:

https://community.fabric.microsoft.com/t5/Power-Query/How-to-copy-large-datasets-with-dataflows-when-dynamic/m-p/3862112#M127331

But I always get an error The request is blocked. and Service unavailable.