I have this data pipeline which contains the Dataflow Gen2 and gets its inputDate for its API data:
The Dataflow uses Update method as Append:
That is, I want the SG_Rainfall data to be appended from StartDate (2025-05-01) to EndDate (2025-05-10).
However, when I query its data, I only has the last day data, that is only on 2025-05-10. Why?
The whole of the pipeline was executed successfully as below:
Dataflow refresh activity inside of your pipeline -- can you guide the steps to get this?
I'm not sure I follow.
What I'm suggesting is that you click the setting that you have for your data destination at this moment to try a new data destination definition that points to a new table.
It might look like this:
And in the next section it would look like the following:
Once those changes are set, you can save your Dataflow again and try to the pipeline again to see what results you might be receiving.
You can later try to point your data destination logic to an existing table and make sure that the update method is set to be "Append" and then save your Dataflow.
If you're still seeing odd results where only the last date is being loaded, please raise a support ticket through:
Microsoft Fabric service status
Do feel free to share that ticket privately and I can pass it over to the engineering team.
sorry, just another question, if I want to have that Data Pipeline to be scheduled, is it possible? This is because every time when I run it, I need to press this OK button here:
So in the scheduled mode, does it needs that OK button to be pressed? or it is automatically pressed?
You can check the Dataflow "recent runs" by going into the workspace list where your Dataflow is located, click the three dots (...) and select the option that reads "Recent runs". Then you can pick the run that you wish to share to navigate to it and then take a screenshot of it.
I tried to create a small repro of this scenario. My Mashup script ended up looking like this:
[StagingDefinition = [Kind = "FastCopy"]]
section Section1;
[DataDestinations = {[Definition = [Kind = "Reference", QueryName = "Query_DataDestination", IsNewTarget = true], Settings = [Kind = "Manual", AllowCreation = true, ColumnSettings = [Mappings = {[SourceColumnName = "timestamp", DestinationColumnName = "timestamp"]}], DynamicSchema = false, UpdateMethod = [Kind = "Append"], TypeSettings = [Kind = "Table"]]]}]
shared Query = let
Source = Web.Contents("https://api.data.gov.sg/v1/environment/rainfall", [Query= [date=myDate]]),
Custom = Json.Document(Source),
Navigation = Table.FromRecords(Custom[items]),
#"Removed other columns" = Table.SelectColumns(Navigation, {"timestamp"}),
#"Removed duplicates" = Table.Distinct(#"Removed other columns", {"timestamp"}),
#"Changed column type" = Table.TransformColumnTypes(#"Removed duplicates", {{"timestamp", type datetimezone}}),
#"Changed column type 1" = Table.TransformColumnTypes(#"Changed column type", {{"timestamp", type date}}),
#"Removed duplicates 1" = Table.Distinct(#"Changed column type 1", {"timestamp"}),
#"Sorted rows" = Table.Sort(#"Removed duplicates 1", {{"timestamp", Order.Descending}}),
#"Kept top rows" = Table.FirstN(#"Sorted rows", 1)
in
#"Kept top rows";
shared Query_DataDestination = let
Pattern = Lakehouse.Contents([CreateNavigationProperties = false, EnableFolding = false]),
Navigation_1 = Pattern{[workspaceId = "REDACTED"]}[Data],
Navigation_2 = Navigation_1{[lakehouseId = "REDACTED"]}[Data],
TableNavigation = Navigation_2{[Id = "rainfall", ItemKind = "Table"]}?[Data]?
in
TableNavigation;
shared myDate = "2025-01-01" meta [IsParameterQuery = true, IsParameterQueryRequired = true, Type = type text];
When I run it through the pipeline or even through the REST API to pass the values for evaluation, it does append the data. For example, I passed these values:
- 2025-01-01
- 2025-06-01
- 2025-08-04
Here's my data loaded into my Lakehouse table looks like:
And here's a screenshot of one of the evaluations for the Dataflow through the "recent runs" dialog:
Would you mind sharing a screenshot of how your Dataflow refresh activity looks like inside the data pipeline and the script that you're using to pass the parameter values to it? There's a possibility that perhaps you've set your parameter to be optional at your Dataflow level and you're not passing a value to it through the Dataflow activity in a Data pipeline. That'll mean that every execution just runs using the default value for that parameter which would explain with just uses the same value over and over again.
Could you share a screenshot of what your "recent runs" dialog for the Dataflow looks like? specifically the section thad reads the parameter value that was passed.
Here's an example on how that section might look like:
What we're after is the actual value that was received by the Dataflow to run.
The other notion is how the table was created. If you try to create a new table in the Lakehouse and set it to append, does it actually repro the behavior?
can you guide me to where can I extract the "recent runs" of Dataflow? I only managed to get the below which may not be what you are looking for.
Helpful resources
Fabric Monthly Update - July 2025
Check out the July 2025 Fabric update to learn about new features.
Fabric Community Update - August 2025
Find out what's new and trending in the Fabric community.
