Dynamic destinations in data flows?

[rowrowrow2]

I want to create a process to clean up a table in a data flow (ie trim whitespace, convert dates, etc). I have lots and lots of tables, and I want to apply the same process to. I was interested in using a data flow to set up that process and then have a pipeline run dataflows for each table on my list. The problem is that it seems like each data flow has to have a fixed destination table.

Am I missing something? Is there not a way to dynamically set the dataflow destination/output so the same dataflow could be used on many different tables?

[miguel · Accepted solution]

That is correct. This is the current behavior today. 

You can request this as a new feature as an idea in the Fabric Ideas portal: 

https://aka.ms/FabricIdeas 

A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query

[rowrowrow2]

As of now, I am using the notebook approach. It works fine, but due to the slow notebook startup time, it has to be done in a batch and can't quickly do a few one off tables since it takes 30-60 seconds just to have a notebook startup. There's also the issue of notebooks failing to start, but hopefully that's taken care of with the new notebook queueing that was announced on April 18.

[rowrowrow2]

Done.

Here's a link to the feature request:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=23a1e79c-0507-ef11-a73c-6045bdb80219