Forum Discussion
Dataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
A few things to check:
- Go into the 'Recent runs' of the Dataflow and check those runs. In the parameters section of the Dataflow, check that the value passed is the value expected by your parameter. You can copy the value from the recent runs dialog and then paste it once you have your Dataflow open and the parameter just to see how things would run.
- Check that you're getting the correct response from the API
- Try to save the Dataflow again and redo the operation. You can also check with the REST API (Items - Get Dataflow Definition - REST API (Dataflow) | Microsoft Learn) the contents of your mashup.pq file to see if the logic for your data destination query and its settings are correct (as outlined here Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn)
From the screenshot we're missing what dates were passed and we don't also have sufficient information as to how the API for https://api.data.gov.sg/v1/environment/rainfall might work when you pass a date that is perhaps in the future- if you pass a date from December 2025, will it give you an empty result and perhaps thats why you're getting it? are the results of the notebook creating values that are not in the range that is expected?