Forum Discussion

davidjmpinto's avatar
davidjmpinto
Icon for Helper I rankHelper I
2 days ago

Composite model: Direct Lake auto-update + parallel refresh, no errors but stale data

I have a Fabric pipeline: a copy job writes tables to a lakehouse, then the semantic model is refreshed. At the same time, a notebook writes the Delta tables used by the Direct Lake part of the model (VACUUM, then overwrite).

The model is composite: import tables plus Direct Lake on OneLake tables, so I need the semantic model refresh for the import tables. I run the refresh in parallel with the notebook to reduce pipeline time, and I rely on "Keep your Direct Lake data up to date" to reframe the Direct Lake tables.

On 1 Nov (UTC):
- pipeline refresh: 05:43:41 to 05:48:40, Completed
- notebook write (DESCRIBE HISTORY, overwrite): 05:49:05
- RefreshedTime of the Direct Lake partitions (INFO.PARTITIONS): 05:49:10 and 05:49:51

The refresh history shows no errors, but the report data is not updated. 

Questions:
1. If a Delta commit lands close to a pipeline refresh on a composite model, can the Direct Lake tables end up reframed without the new data and without any error?
2. Is running the refresh in parallel with automatic updates a supported pattern, or should the refresh run after the notebook?

Need Help

2 Replies

  • benibix's avatar
    benibix
    Icon for Helper III rankHelper III
    2 days ago

    Hi davidjmpinto,

    Same setup as your other two threads, so in short (the details and the accepted answer are in Direct Lake auto update with parallel refresh in a composite model):

    • Yes, the Direct Lake tables can end up on an older Delta version without any error. A framing only sees the commits that are visible when it runs, and a successful refresh doesn't check whether newer data landed just after.
    • You can't rely on the order when a pipeline refresh and automatic updates run on the same model at the same time. Run one semantic model refresh after both the copy job and the notebook have succeeded. In a composite model that single refresh loads the import tables and reframes the Direct Lake tables together.
    • If the report should only change when the whole load is done, turn off "Keep your Direct Lake data up to date" so the pipeline is the only thing that reframes.
  • SamInogic's avatar
    SamInogic
    Icon for Super User rankSuper User
    1 day ago

    Hi,

    Yes, this can happen if the Direct Lake tables are reframed before the notebook finishes writing the latest Delta changes. A successful semantic model refresh does not necessarily mean the Direct Lake tables are using the latest data.

    I’d recommend running the notebook first and triggering the semantic model refresh only after the Delta writes have completed. This avoids a timing issue between the two operations.

    Also, check whether the report is using the expected Direct Lake tables and whether fallback to DirectQuery is occurring. If the issue persists, test with a refresh after the notebook completes and compare the results.

    I would not assume this is a confirmed platform limitation without further testing, but sequential execution is the safer approach for this pipeline.

    Hope this Helps!

    Thanks