topic Re: Looping to copy tables using a dataflow gen2 in Dataflow

Looping to copy tables using a dataflow gen2

pmscorca — Sun, 23 Jun 2024 17:31:36 GMT

Hi,

I'd like to know if it is possible to implement a loop using a dataflow in order to copy some tables from a lakehouse to another lakehouse or warehouse.

Any suggests to me, please? Thanks

Re: Looping to copy tables using a dataflow gen2

frithjof_v — Sun, 23 Jun 2024 19:19:21 GMT

I never tried, but I don't think looping in order to create multiple destination tables is supported in Dataflow.

What is the reason you need to loop and create copy tables?

Why not use e.g. shortcuts or cross-database queries instead?

I would guess Notebook for looping.

Maybe it's also possible in Data pipeline?

Re: Looping to copy tables using a dataflow gen2

NandanHegde — Mon, 24 Jun 2024 02:35:20 GMT

Unfortunately, based on my understanding as of now dataflows cannot be parameterized for the source and sink names. So you cannot have a meta data driven framework and thus there is no sense with looping the dataflows

Re: Looping to copy tables using a dataflow gen2

v-cboorla-msft — Thu, 27 Jun 2024 12:32:14 GMT

Hi pmscorca

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thank you.