Forum Discussion

davidjmpinto's avatar
davidjmpinto
Icon for Helper I rankHelper I
2 days ago

Direct Lake: pipeline refresh in parallel with notebook write, data not updated

What happens if the Delta commit lands during the refresh?
My case
Fabric pipeline:

  1. A copy job writes tables to a lakehouse.
  2. Then it refreshes the semantic models (import and Direct Lake on OneLake).
  3. In parallel, a notebook writes the Delta tables used by the Direct Lake model (VACUUM, then an overwrite).

“Keep your Direct Lake data up to date” is enabled on the Direct Lake model.

What I found (refresh operation)

  • Semantic model refresh in the refresh history: started 05:43:41, ended 05:48:40, Completed.
  • Notebook write (DESCRIBE HISTORY, v233, overwrite): 05:49:05.
  • Partition RefreshedTime from DAX Studio (INFO.PARTITIONS()):
    • pbi_fact_is_direct_lake: 05:49:10
    • pbi_fact_bp_direct_lake: 05:49:51
    • import tables: between 05:44 and 05:48

So the model shows a reframe right after the notebook’s commit.

Problem
The refresh history shows no errors and no cancelled refreshes (I did not see the “a new refresh is detected on the same database” error that a community post mentions), but the data in the report is not updated.

Question

  • If the notebook’s Delta commit lands while the pipeline refresh is still running, what happens?
  • Does the automatic update still reframe afterwards?Or can it be skipped or cancelled because a refresh is already running?
  • Is it supported to run the refresh activity and automatic updates on the same model in parallel, or should I remove one of them?
  • If a Delta commit lands close to a pipeline refresh on the same Direct Lake model, can the model end up reframed without the new data and without any error shown?

 

Need Help

1 Reply

  • benibix's avatar
    benibix
    Icon for Helper III rankHelper III
    2 days ago

    Hi davidjmpinto,

    Your thread on the same setup in the Desktop forum has an accepted answer: Direct Lake auto update with parallel refresh in a composite model. For the questions here, the short version:

    • A Direct Lake table only shows the Delta version it was last framed on. A framing that runs before the notebook's commit is visible pins the old version and still reports success, so you get no error.
    • Automatic updates do reframe after a new commit, but you don't control when, and with a refresh running on the same model at the same moment you can't rely on the order.
    • So don't run both on the same model in parallel. Run the semantic model refresh once, after both the copy job and the notebook have succeeded. In a composite model that one refresh reloads the import tables and reframes the Direct Lake tables in the same operation.
    • If the report should only change when the whole load is done, also turn off "Keep your Direct Lake data up to date", so the pipeline's refresh is the only thing that reframes.

    The copy job and the notebook can keep running in parallel. Only the refresh has to wait for both.