Forum Discussion
Dataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
A few things to check:
- Go into the 'Recent runs' of the Dataflow and check those runs. In the parameters section of the Dataflow, check that the value passed is the value expected by your parameter. You can copy the value from the recent runs dialog and then paste it once you have your Dataflow open and the parameter just to see how things would run.
- Check that you're getting the correct response from the API
- Try to save the Dataflow again and redo the operation. You can also check with the REST API (Items - Get Dataflow Definition - REST API (Dataflow) | Microsoft Learn) the contents of your mashup.pq file to see if the logic for your data destination query and its settings are correct (as outlined here Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn)
From the screenshot we're missing what dates were passed and we don't also have sufficient information as to how the API for https://api.data.gov.sg/v1/environment/rainfall might work when you pass a date that is perhaps in the future- if you pass a date from December 2025, will it give you an empty result and perhaps thats why you're getting it? are the results of the notebook creating values that are not in the range that is expected?
"Activity name","Activity status","Error","Run start","Duration","Input","Output"
"Dataflow1","Succeeded","","8/10/2025, 8:52:20 PM","51s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-10"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""823f9c67-3347-4cb3-b596-23e05b0f1365"",
""rootActivityId"": ""8c0a284e-ea3c-468e-bc42-bb0fb68930fc"",
""executionDuration"": 47
}"
"Dataflow1","Succeeded","","8/10/2025, 8:51:30 PM","49s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-09"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""082dd81b-610f-4e50-8e65-680b4e467ea7"",
""rootActivityId"": ""507ca1f6-cd0e-4c25-bcd1-588518070800"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:50:39 PM","50s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-08"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""596daeee-1374-43ea-bcf5-45f400a790b4"",
""rootActivityId"": ""9ae9c9f3-e2d3-4269-91f6-357bcb7ad3f8"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:49:49 PM","50s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-07"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""0e1d813c-46bf-4d38-af50-b63b4ef7ca6a"",
""rootActivityId"": ""0c04ec58-8a0d-493b-aeff-2f1b3d01037f"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:48:59 PM","49s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-06"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""8c3ba27f-df34-4e9e-9b5d-c0e6615aba36"",
""rootActivityId"": ""90ea7971-848a-418b-b3a9-c6f8ab695c3a"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:47:58 PM","1m 0s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-05"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""aaf15235-a3b1-4951-bb93-004bfa7bd3c1"",
""rootActivityId"": ""10d3c16b-3743-467a-8b44-3ab0adb23079"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:47:08 PM","49s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-04"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""445b66dd-717a-4bf4-bc56-5dea27e885d2"",
""rootActivityId"": ""8ba4a3ca-53e9-49c2-ad56-1c8b8c0f436b"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:46:07 PM","59s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-03"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""7873aa15-8f63-40c0-8ab8-32bd3d045efe"",
""rootActivityId"": ""15f5ed25-a11f-4499-af23-a6b95cc158a4"",
""executionDuration"": 46
}"
"Dataflow1","Succeeded","","8/10/2025, 8:45:16 PM","50s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-02"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""7f3c4cda-f420-4487-9043-4a8ba6f2fc0b"",
""rootActivityId"": ""c5028936-ece5-4527-a195-96b07d26ef03"",
""executionDuration"": 47
}"
"Dataflow1","Succeeded","","8/10/2025, 8:44:26 PM","49s","{
""dataflowId"": ""db6968e7-d4db-46e4-8406-fd7857e5e5d8"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""notifyOption"": ""NoNotification"",
""dataflowType"": ""DataflowFabric"",
""parameters"": {
""inputDate"": {
""value"": ""2025-05-01"",
""type"": ""string""
}
}
}","{
""jobInstanceId"": ""b1bcb012-d158-4112-8b84-e7296db2a2ad"",
""rootActivityId"": ""395f3050-00d7-48bf-bb46-a4c70597f74f"",
""executionDuration"": 46
}"
"ForEach1","Succeeded","","8/10/2025, 8:44:26 PM","8m 48s","{
""ItemsCount"": ""10""
}","{}"
"Notebook_Dates","Succeeded","","8/10/2025, 8:43:57 PM","28s","{
""notebookId"": ""7dc6e468-bef6-4e11-bcf3-12b5805e50ab"",
""workspaceId"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""parameters"": {
""StartDate"": {
""value"": ""2025-05-01"",
""type"": ""string""
},
""EndDate"": {
""value"": ""2025-05-10"",
""type"": ""string""
}
}
}","{
""status"": ""Succeeded"",
""result"": {
""runId"": ""5b0acefd-7443-46fb-925c-11387b066a04"",
""runStatus"": ""Succeeded"",
""sessionId"": ""447ebac8-73b0-440a-a1fe-b77e69a5658a"",
""sparkPool"": ""ca4821de-7641-44ea-9a6d-4311320a74c4"",
""error"": null,
""lastCheckedOn"": ""2025-08-10T12:44:21.3733333Z"",
""metadata"": {
""isForPipeline"": null,
""runStartTime"": ""2025-08-10T12:44:06.63098Z"",
""runEndTime"": ""2025-08-10T12:44:21.3737615Z""
},
""highConcurrencyModeStatus"": null,
""exitValue"": ""[{\""date\"": \""2025-05-01\""}, {\""date\"": \""2025-05-02\""}, {\""date\"": \""2025-05-03\""}, {\""date\"": \""2025-05-04\""}, {\""date\"": \""2025-05-05\""}, {\""date\"": \""2025-05-06\""}, {\""date\"": \""2025-05-07\""}, {\""date\"": \""2025-05-08\""}, {\""date\"": \""2025-05-09\""}, {\""date\"": \""2025-05-10\""}]""
},
""message"": ""Notebook execution is in Succeeded state, runId: 5b0acefd-7443-46fb-925c-11387b066a04"",
""SparkMonitoringURL"": ""workloads/de-ds/sparkmonitor/7dc6e468-bef6-4e11-bcf3-12b5805e50ab/447ebac8-73b0-440a-a1fe-b77e69a5658a"",
""sessionSource"": """",
""sessionTag"": """",
""executionDuration"": 18
}"I inserted the output as seen below, and those dates that I passed to the Dataflow is from 2025-05-01 to 2025-05-10, which are all valid for the API.
Could you share a screenshot of what your "recent runs" dialog for the Dataflow looks like? specifically the section thad reads the parameter value that was passed.
Here's an example on how that section might look like:
What we're after is the actual value that was received by the Dataflow to run.
The other notion is how the table was created. If you try to create a new table in the Lakehouse and set it to append, does it actually repro the behavior?
can you guide me to where can I extract the "recent runs" of Dataflow? I only managed to get the below which may not be what you are looking for.