Forum Discussion
Dataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
Could you share a screenshot of what your "recent runs" dialog for the Dataflow looks like? specifically the section thad reads the parameter value that was passed.
Here's an example on how that section might look like:
What we're after is the actual value that was received by the Dataflow to run.
The other notion is how the table was created. If you try to create a new table in the Lakehouse and set it to append, does it actually repro the behavior?
can you guide me to where can I extract the "recent runs" of Dataflow? I only managed to get the below which may not be what you are looking for.