Forum Discussion

M98's avatar
M98
New Member
10 months ago
Solved

Dataflow Gen2 Incremental Load

I’m performing an incremental load in a Dataflow Gen2 using the LastModifiedDate column from the source table. This column is stored in Unix epoch format, so I convert it to a full datetime using thi...
capacities
workspaces
  • mmmmm_98's avatar
    mmmmm_98
    9 months ago

    The process has already been folded, but I suspect the SQL crash could also be due to concurrent access from other applications connected to the same SQL server. The additional load might have contributed to the failure.

v-ssriganesh's avatar
v-ssriganesh
Community Support
10 months ago

Hi M98,
Thank you for reaching out to the Microsoft fabric community forum.

The SQL Server crash is unlikely to be directly caused by the incremental load missing changes or being misconfigured, though these could contribute to larger data pulls. More likely, the issue stems from broken query folding, where the full dataset is processed instead of just incremental rows due to the datetime conversion happening before filtering. A reference table out of sync could amplify this by reloading excessive data, but the crash points more to the load configuration. Ensuring folding and proper sync should resolve it.

Best regards,
Ganesh Singamshetty.