Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount.
Register nowThe Power BI DataViz World Championships are on! With four chances to enter, you could win a spot in the LIVE Grand Finale in Las Vegas. Show off your skills.
Hi All,
Wondering if this is a bug or if this is by design.
I have a DataFlow Gen2 operation that retrieves ticket data from our ITSM that has been modified in the last 7 days. In the Query within the DataFlow I have set the Ticket ID column as a Primary Key, with the intent that this prevents duplicate records from being added during append operations. Instead, what I am finding is that if an entry with the same ticket id already exists, a new one is appended.
There don't apppear to be any specific indication in the LakeHouse's table definition that there is a primary key set.
Is this a feature that has not yet been implemented for LakeHouse?
The data is being retrieved from a REST API via an on-prem data gateway (3000.186.16).
Any suggestions or ideas?
Regards,
JohnB
Hi @lbendlin ,
Thanks for responding.
Due to the nature of the data, I cannot use an Incremental refresh in the way documented since a "ticket" in the ITSM may last accross several refresh cycles, and change state during that time. For reporting purposes I need to make sure that I have the latest state of the ticket, rather than freezing it on initial import, if that makes sense.
From reading several articles (not specific to DataFlow Gen2) for PowerBI/Fabric, it suggested at setting a Primary Key would result in an UPSERT if a matching entry was found. Is this not the case for Lakehouse/DataFlow Gen2?
For now I am attempting to deduplicate the data via other means, however, if this process can be simplified, that would be much preferred.
Thanks.
JohnB
Did you set up incremental refresh for that dataflow?
I am having a similar problem, was there a resoution to this? Thank You
User | Count |
---|---|
44 | |
26 | |
22 | |
20 | |
17 |
User | Count |
---|---|
52 | |
45 | |
24 | |
24 | |
21 |