Forum Discussion
Error Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2
Hello,
I am encountering the following error when attempting to copy data from the Bronze Lakehouse to the Silver Lakehouse using Dataflow Gen2:
Error:
Extracted_data_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: 02ddb959-4800-4b38-988b-0c58354041a3). Operation on target ca-47ecc963-4615-4f7f-8a72-dacd9d97d62a failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = 02ddb959-4800-4b38-988b-0c58354041a3". Error code: 999999. (Request ID: 1ef47cdf-9734-4ef3-96eb-cd4748ae0adb).
Process Overview:
- Data is retrieved via API calls. After approximately 10 hours, the session times out.
- To address this, I implemented a checkpoint file that stores the IDs of records already copied to the Delta table. This allows the process to resume from where it stopped by reconciling IDs from the API with those in the checkpoint file, skipping processed records and continuing with the remaining ones.
- When the session timed out, I created a Dataflow Gen2 to copy data from Bronze to Silver Lakehouse, which succeeded.
- After reconnecting the session and running the extraction again, it correctly skipped processed IDs and completed the remaining extraction.
- However, when I refreshed and ran the Dataflow Gen2 again, I encountered the error above. Creating a new Dataflow Gen2 resulted in the same error.
Note:
The checkpoint mechanism was implemented for another extraction process, which completed successfully without session timeout. The Dataflow Gen2 is specifically copying data from Bronze to Silver Lakehouse.Implementation Details:
- Writing to Delta Table:# Append to Delta tabledf_batch.write.format("delta").mode("append").saveAsTable(lakehouse_table_name)total_records_written += len(batch_data)print(f" Batch written | Data: {len(processed_data_ids)}/{len(data_data)} | Total records: {total_records_written:,}")
- Saving Checkpoint:# Save checkpointtry:checkpoint = {'processed_data_ids': list(processed_data_ids),'failed_data': failed_sites}os.makedirs(os.path.dirname(checkpoint_file), exist_ok=True)with open(checkpoint_file, 'wb') as f:pickle.dump(checkpoint, f)# Save data metadata (lastData tracker)with open(data_metadata_file, 'wb') as f:pickle.dump(data_last_data_tracker, f)<meta http-equiv="Content-Type" content="text/html; charset=us-ascii" /> a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; }
Could you please advise on what might be causing this issue and suggest a possible solution?
Thank you!
- Saving Checkpoint:
- Writing to Delta Table:
Hi TGG360,
Thanks for the detailed testing — your observations are detailed and very helpful.
The issue seems to be delta snapshot availability timing under heavy commit pressure, specifically with Dataflow Gen2.
TGG360 wrote:I noticed that the Delta logs were still being generated even after the run had completed.
My assumption: DF tries to read the silver table while spark is still finalizing the delta snapshot (writing _delta_log commits). Under heavy load, that finalization lags, so DF sees no snapshot and fails, even though the table might exist. Under maxed-out capacity, this window widens, making the behavior appear inconsistent (After 2 days (as you stated), DF suddenly succeeded.)
As a next step, could you please:
- Add a buffer. Basically, once your first DF run completes, the second instance should run after a delay. Unfortunately, I don't think there is a reliable way to detect _delta_log completion. Hence, add a time-based 40-50 min delay (due to your capacity maxing out, extend if required) between each DF completion and succeeding DF instance's start.
- (just sharing; not needed): As a manual delay check, repeatedly run DESCRIBE DETAIL <table> every 4-5 minutes and monitor lastModified and sizeInBytes. When these stop changing consistently over multiple checks, the snapshot is likely stable.
- If possible, temporarily scale up capacity during ingestion windows, scale down afterwards. This shortens snapshot lag and reduces failure probability.
If the issue persists despite these mitigations, it would be worth raising a Fabric support case, as DF Gen2 should ideally wait for snapshot stabilization or retry more gracefully instead of hard fail. Also, if Pipelines are working reliably for your scenario, prefer them for Bronze-to-Silver movement under high load.
Awaiting your further results 😊!
- Add a buffer. Basically, once your first DF run completes, the second instance should run after a delay. Unfortunately, I don't think there is a reliable way to detect _delta_log completion. Hence, add a time-based 40-50 min delay (due to your capacity maxing out, extend if required) between each DF completion and succeeding DF instance's start.
7 Replies
- stoic-harsh
Super User
Hi TGG360,
Your error says DeltaSnapshotError: Delta table snapshot doesn't exist.
It indicates that Dataflow cannot find a valid Delta snapshot (_delta_log_) for the table. This usually happens when the Delta table is left in an inconsistent state due to interrupted or failed Spark writes. In Microsoft Fabric, OneLake is not a POSIX filesystem, so using os.makedirs() and open() can also silently fail or corrupt Delta metadata. Additionally, concurrent use of Spark and Dataflow Gen2, to the same table, can further break the Delta snapshot.
To confirm, try running (if this fails, the delta table is broken):
%%sql DESCRIBE DETAIL <schema.table_name>To solve, rebuild the snapshot by overwriting the table:
df = spark.table("<schema.table_name>") df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("<schema.table_name>")To prevent snapshot corruption, avoid concurrent Spark, Dataflow writes on the same table. Recommended pattern:
- Only Spark writes Bronze
- Only Dataflow Gen2 writes Silver.
- If Spark must write to Silver, pause Dataflow refresh until Spark completes.
Further, replace file-based checkpoints with Lakehouse-native Delta storage for tracking.
checkpoint = { 'processed_data_ids': list(processed_data_ids), 'failed_data': failed_sites } checkpoint_df.write.mode("overwrite").format("delta").saveAsTable("checkpoint_table")Hope this helps!
Hi stoic-harsh,
Thanks for your input.
I ran describe detail function and I was able to view the Delta table. I noticed that the Delta logs were still being generated even after the run had completed. I retried the initial DF Gen2 process (without making any code changes or rerunning the notebook), and it worked. My guess is that Spark wasn’t able to read the snapshot earlier because the Delta log updates were still in progress.I’ve also implemented the enhancement to switch the checkpoint file from pickle to Lakehouse Delta, and it’s currently being tested. The dataset is about 56 million records pulled through the API, so I don't believe this issue is related to volume.
Here are the testing steps and results so far:There are 3 different API calls, all using the same code structure.
Before changing the checkpoint file (pickle to delta table)
API Call 1 (~400 records): DF Gen2 successfully copied the transformed data from Bronze (Table 1) to Silver.
- API Call 2 (~55 million records): DF Gen2 consistently threw the error I originally posted. After many attempts, two days later after the post (with no code changes and without rerunning the notebook), DF Gen2 suddenly succeeded in copying the transformed data from Bronze (Table 2) to Silver.
- API Call 3 (~22 million records): DF Gen2 successfully copied the transformed data from Bronze (Table 3) to Silver.
After changing the checkpoint file to delta and rerunning the notebooks (UPSERT applied)API Call 1 (~400 records): DF Gen2 successfully copied the transformed data from Bronze (Table 1) to Silver.
API Call 2 (~58 million records): DF Gen2 is again throwing the same error. I created a new DF Gen2 instance, but still no success.
API Call 3 (~24 million records): DF Gen2 is now failing with the same error as well. Creating a new DF Gen2 instance did not resolve the issue.
Additional Notes- Compute capacity has been maxed out during all tests.
- Data Pipeline runs perfectly fine, but DF Gen2 consistently throws the error.
- The data is definitely present, this seems to be an issue specifically with DF Gen2’s ability to read or process it under certain load or Delta snapshot conditions.
- At this point, it’s difficult to pinpoint the exact root cause, as behavior is inconsistent across runs and across the different API loads.
- stoic-harsh
Hi TGG360,
Thanks for the detailed testing — your observations are detailed and very helpful.
The issue seems to be delta snapshot availability timing under heavy commit pressure, specifically with Dataflow Gen2.
TGG360 wrote:I noticed that the Delta logs were still being generated even after the run had completed.
My assumption: DF tries to read the silver table while spark is still finalizing the delta snapshot (writing _delta_log commits). Under heavy load, that finalization lags, so DF sees no snapshot and fails, even though the table might exist. Under maxed-out capacity, this window widens, making the behavior appear inconsistent (After 2 days (as you stated), DF suddenly succeeded.)
As a next step, could you please:
- Add a buffer. Basically, once your first DF run completes, the second instance should run after a delay. Unfortunately, I don't think there is a reliable way to detect _delta_log completion. Hence, add a time-based 40-50 min delay (due to your capacity maxing out, extend if required) between each DF completion and succeeding DF instance's start.
- (just sharing; not needed): As a manual delay check, repeatedly run DESCRIBE DETAIL <table> every 4-5 minutes and monitor lastModified and sizeInBytes. When these stop changing consistently over multiple checks, the snapshot is likely stable.
- If possible, temporarily scale up capacity during ingestion windows, scale down afterwards. This shortens snapshot lag and reduces failure probability.
If the issue persists despite these mitigations, it would be worth raising a Fabric support case, as DF Gen2 should ideally wait for snapshot stabilization or retry more gracefully instead of hard fail. Also, if Pipelines are working reliably for your scenario, prefer them for Bronze-to-Silver movement under high load.
Awaiting your further results 😊!
- Add a buffer. Basically, once your first DF run completes, the second instance should run after a delay. Unfortunately, I don't think there is a reliable way to detect _delta_log completion. Hence, add a time-based 40-50 min delay (due to your capacity maxing out, extend if required) between each DF completion and succeeding DF instance's start.
ssritharHi TGG360 ,
You're encountering a Delta Snapshot Error when refreshing a Dataflow Gen2 that reads from a Bronze Lakehouse table and writes to Silver. This happened after your custom pipeline wrote data using .saveAsTable() and resumed from a checkpoint.
Primary cause:
Dataflow Gen2 cannot find a valid Delta snapshot for the source or target table, likely due to:- Custom write using .saveAsTable() that doesn't create the Delta metadata in a way Dataflow Gen2 expects.
- The Delta table's _delta_log folder is missing, corrupted, or not accessible to Dataflow Gen2.
- Table registration mismatch — Dataflow expects the table to be registered in the Fabric Lakehouse catalog, and it may not be.
Fixes:
- Switch from saveAsTable() to save() with Delta format and explicit path.
- Ensure _delta_log folder is present and intact.
- Ensure consistent schema + completed writes.
- Recreate the Dataflow Gen2 Connection (if corrupted)
Why It Worked Once, Then Failed
- First run: Dataflow created its own Delta table with a valid snapshot → worked
- Later, your resumed process wrote with .saveAsTable() or custom logic → snapshot mismatch
- Now the table is out of sync with what Dataflow expects
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly. If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Hi TGG360 ,
Thank you stoic-harsh for the response provided!
Has your issue been resolved? If the response provided by the community member addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.
Thank you.Hi TGG360 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.
- deborshi_nag
Hi TGG360
If the error message points to a corruption of the /_delta_log, then it is best to rebuild it and see if that works.
You can run the following Spark code that will rebuild the folder - replace the location of the path in the code below.
from delta.tables import DeltaTable # Use a path relative to your Lakehouse root DeltaTable.convertToDelta(spark, "parquet.`Files/raw/sales/`")Hope this helps - please appreciate leaving a Kudos or accepting as a Solution!
