topic Re: Dynamic destinations in data flows? in Dataflow

Dynamic destinations in data flows?

rowrowrow2 — Tue, 30 Apr 2024 15:10:08 GMT

I want to create a process to clean up a table in a data flow (ie trim whitespace, convert dates, etc). I have lots and lots of tables, and I want to apply the same process to. I was interested in using a data flow to set up that process and then have a pipeline run dataflows for each table on my list. The problem is that it seems like each data flow has to have a fixed destination table.

Am I missing something? Is there not a way to dynamically set the dataflow destination/output so the same dataflow could be used on many different tables?

Re: Dynamic destinations in data flows?

miguel — Tue, 30 Apr 2024 15:15:48 GMT

That is correct. This is the current behavior today.

You can request this as a new feature as an idea in the Fabric Ideas portal:

https://aka.ms/FabricIdeas

A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query

Re: Dynamic destinations in data flows?

rowrowrow2 — Tue, 30 Apr 2024 15:24:30 GMT

Done.

Here's a link to the feature request:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=23a1e79c-0507-ef11-a73c-6045bdb80219

Re: Dynamic destinations in data flows?

rowrowrow2 — Tue, 30 Apr 2024 15:27:14 GMT

As of now, I am using the notebook approach. It works fine, but due to the slow notebook startup time, it has to be done in a batch and can't quickly do a few one off tables since it takes 30-60 seconds just to have a notebook startup. There's also the issue of notebooks failing to start, but hopefully that's taken care of with the new notebook queueing that was announced on April 18.