October 28 & 29: Experts share their secrets on how to pass the Fabric Analytics Engineer certification exam—live. Learn more
I want to create a process to clean up a table in a data flow (ie trim whitespace, convert dates, etc). I have lots and lots of tables, and I want to apply the same process to. I was interested in using a data flow to set up that process and then have a pipeline run dataflows for each table on my list. The problem is that it seems like each data flow has to have a fixed destination table.
Am I missing something? Is there not a way to dynamically set the dataflow destination/output so the same dataflow could be used on many different tables?
Solved! Go to Solution.
That is correct. This is the current behavior today.
You can request this as a new feature as an idea in the Fabric Ideas portal:
A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query
That is correct. This is the current behavior today.
You can request this as a new feature as an idea in the Fabric Ideas portal:
A different approach would be to use a script in a notebook that does the iterative work on all the tables that you require. For Dataflow Gen2, today, you have to setup the destination for every query
As of now, I am using the notebook approach. It works fine, but due to the slow notebook startup time, it has to be done in a batch and can't quickly do a few one off tables since it takes 30-60 seconds just to have a notebook startup. There's also the issue of notebooks failing to start, but hopefully that's taken care of with the new notebook queueing that was announced on April 18.
Done.
Here's a link to the feature request:
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=23a1e79c-0507-ef11-a73c-6045bdb80219