Forum Discussion
Post PatronDataflow - Append
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
me and tan_thiamhuat had an offline sync.
The issue was related to how the data destination settings were set. After manually changing the creation of a new table to go into the manual settings (and avoid the automatic settings) things worked as expected.
Marking this thread as resolved.
It was great and fantastic that Miguel help me solve this issue within 15mins when I took more than 2 days to look into what is wrong 😁. Thanks Miguel for your proactive effort to do it offline to get this solved. Appreciate it a lot.
Community AdminWould you mind sharing the screenshot of your Dataflow refresh activity inside of your pipeline?
also, to rule out a metadata sync issue, would you mind querying the data directly inside the Lakehouse and not use it's SQL Endpoint ?
Post PatronDataflow refresh activity inside of your pipeline -- can you guide the steps to get this?
- miguel
I just saw the screenshots that you shared around the "Recent runs" of your Dataflow, so we can rule out that a value is not being passed to the Dataflow.
The best thing that you can do is to modify your data destination settings and point the solution to a new table.
If you can repro the behavior even when changing the data destination settings that create a new table, please share that repro steps with us. You can also use the simplified mashup code that I shared to compare it with your own. Anything that would help us reproduce this behavior on our environments will help us pinpoint what could be happening.
- tan_thiamhuat
when I create a new table, there is no Append feature? meaning that I cannot have a range of dates. So, I need to create a new table for one date of data first? then run the same data pipeline with a range of dates for the append to the table created?
- miguel
I'm not sure I follow.
What I'm suggesting is that you click the setting that you have for your data destination at this moment to try a new data destination definition that points to a new table.
It might look like this:
And in the next section it would look like the following:
Once those changes are set, you can save your Dataflow again and try to the pipeline again to see what results you might be receiving.
You can later try to point your data destination logic to an existing table and make sure that the update method is set to be "Append" and then save your Dataflow.
If you're still seeing odd results where only the last date is being loaded, please raise a support ticket through:
Microsoft Fabric service status
Do feel free to share that ticket privately and I can pass it over to the engineering team.