Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
bcdobbs
Super User
Super User

Dataflows Gen2 - Delta Table Overwrite Issue

Hi,

I have really odd issue...

 

I have an gen2 dataflow that copies a csv file from an on prem location into a delta table. It's set to overwrite and in general apepars to work fine. (Would prefer to be able to do that direct in data factory but it can't read on prem yet).

 

If I query it with the sql endpoint, it gives the expected data that matches the source csv:

bcdobbs_0-1702472222044.png

 

If I read it into a spark dataframe, though I get old versions of the same records displaying:

bcdobbs_1-1702472514356.png

The problem is I'm then using that dataframe to create what should be a unique dimenion.

 

I've tried specificying the latest timetravel version explicitly and it still does the same.

 

Only thing I can think is there is some oddity in the delta meta data being written by dataflows.



Ben Dobbs

LinkedIn | Twitter | Blog

Did I answer your question? Mark my post as a solution! This will help others on the forum!
Appreciate your Kudos!!
2 REPLIES 2
bcdobbs
Super User
Super User

Thanks @Joshrodgers123 at least that makes me feel like I'm not doing something silly!



Ben Dobbs

LinkedIn | Twitter | Blog

Did I answer your question? Mark my post as a solution! This will help others on the forum!
Appreciate your Kudos!!
Joshrodgers123
Advocate IV
Advocate IV

I have the same issue, but reversed. I have a dataflow writing to a lakehouse. If I read the data in a notebook, I can see the correct and latest data. If I read the data through the SQL Endpoint (or even the table preview in the lakehouse), it shows an older version of the delta table. 

 

I opened a support ticket (2312050040012594) and they are investigating. They said it was a sync issue. 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.