Forum Discussion
Dynamic destinations in data flows?
I want to create a process to clean up a table in a data flow (ie trim whitespace, convert dates, etc). I have lots and lots of tables, and I want to apply the same process to. I was interested in using a data flow to set up that process and then have a pipeline run dataflows for each table on my list. The problem is that it seems like each data flow has to have a fixed destination table.
Am I missing something? Is there not a way to dynamically set the dataflow destination/output so the same dataflow could be used on many different tables?
That is correct. This is the current behavior today.
You can request this as a new feature as an idea in the Fabric Ideas portal:
A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query
3 Replies
That is correct. This is the current behavior today.
You can request this as a new feature as an idea in the Fabric Ideas portal:
A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query
Done.
Here's a link to the feature request:
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=23a1e79c-0507-ef11-a73c-6045bdb80219
As of now, I am using the notebook approach. It works fine, but due to the slow notebook startup time, it has to be done in a batch and can't quickly do a few one off tables since it takes 30-60 seconds just to have a notebook startup. There's also the issue of notebooks failing to start, but hopefully that's taken care of with the new notebook queueing that was announced on April 18.
