Forum Discussion
Post PatronDataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
Community AdminYou can check the Dataflow "recent runs" by going into the workspace list where your Dataflow is located, click the three dots (...) and select the option that reads "Recent runs". Then you can pick the run that you wish to share to navigate to it and then take a screenshot of it.
I tried to create a small repro of this scenario. My Mashup script ended up looking like this:
[StagingDefinition = [Kind = "FastCopy"]]
section Section1;
[DataDestinations = {[Definition = [Kind = "Reference", QueryName = "Query_DataDestination", IsNewTarget = true], Settings = [Kind = "Manual", AllowCreation = true, ColumnSettings = [Mappings = {[SourceColumnName = "timestamp", DestinationColumnName = "timestamp"]}], DynamicSchema = false, UpdateMethod = [Kind = "Append"], TypeSettings = [Kind = "Table"]]]}]
shared Query = let
Source = Web.Contents("https://api.data.gov.sg/v1/environment/rainfall", [Query= [date=myDate]]),
Custom = Json.Document(Source),
Navigation = Table.FromRecords(Custom[items]),
#"Removed other columns" = Table.SelectColumns(Navigation, {"timestamp"}),
#"Removed duplicates" = Table.Distinct(#"Removed other columns", {"timestamp"}),
#"Changed column type" = Table.TransformColumnTypes(#"Removed duplicates", {{"timestamp", type datetimezone}}),
#"Changed column type 1" = Table.TransformColumnTypes(#"Changed column type", {{"timestamp", type date}}),
#"Removed duplicates 1" = Table.Distinct(#"Changed column type 1", {"timestamp"}),
#"Sorted rows" = Table.Sort(#"Removed duplicates 1", {{"timestamp", Order.Descending}}),
#"Kept top rows" = Table.FirstN(#"Sorted rows", 1)
in
#"Kept top rows";
shared Query_DataDestination = let
Pattern = Lakehouse.Contents([CreateNavigationProperties = false, EnableFolding = false]),
Navigation_1 = Pattern{[workspaceId = "REDACTED"]}[Data],
Navigation_2 = Navigation_1{[lakehouseId = "REDACTED"]}[Data],
TableNavigation = Navigation_2{[Id = "rainfall", ItemKind = "Table"]}?[Data]?
in
TableNavigation;
shared myDate = "2025-01-01" meta [IsParameterQuery = true, IsParameterQueryRequired = true, Type = type text];
When I run it through the pipeline or even through the REST API to pass the values for evaluation, it does append the data. For example, I passed these values:
- 2025-01-01
- 2025-06-01
- 2025-08-04
Here's my data loaded into my Lakehouse table looks like:
And here's a screenshot of one of the evaluations for the Dataflow through the "recent runs" dialog:
Would you mind sharing a screenshot of how your Dataflow refresh activity looks like inside the data pipeline and the script that you're using to pass the parameter values to it? There's a possibility that perhaps you've set your parameter to be optional at your Dataflow level and you're not passing a value to it through the Dataflow activity in a Data pipeline. That'll mean that every execution just runs using the default value for that parameter which would explain with just uses the same value over and over again.
Post Patron