Forum Discussion
Dataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
I'm not sure I follow.
What I'm suggesting is that you click the setting that you have for your data destination at this moment to try a new data destination definition that points to a new table.
It might look like this:
And in the next section it would look like the following:
Once those changes are set, you can save your Dataflow again and try to the pipeline again to see what results you might be receiving.
You can later try to point your data destination logic to an existing table and make sure that the update method is set to be "Append" and then save your Dataflow.
If you're still seeing odd results where only the last date is being loaded, please raise a support ticket through:
Microsoft Fabric service status
Do feel free to share that ticket privately and I can pass it over to the engineering team.
that was exactly the steps I performed.
when I click that Microsoft Fabric service status , and I click Submit a ticket button, it opens up my Power BI / Fabric page, which I cannot submit any ticket. Probably my account does not have enough rights to submit a ticket.
sorry, just another question, if I want to have that Data Pipeline to be scheduled, is it possible? This is because every time when I run it, I need to press this OK button here:
So in the scheduled mode, does it needs that OK button to be pressed? or it is automatically pressed?