User Profile
TGG360
Frequent Visitor
Joined 6 months ago
User Widgets
Contributions
Re: Error Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2
The issue with the DF Gen2 integration has been resolved. The data source configuration should use the SQL connection string rather than the Lakehouse connector. After updating the data source to use the SQL connection string, all processes tested successfully.2.8KViews2likes0CommentsRe: Error Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2
Hi stoic-harsh, Thanks for your input. I ran describe detail function and I was able to view the Delta table. I noticed that the Delta logs were still being generated even after the run had completed. I retried the initial DF Gen2 process (without making any code changes or rerunning the notebook), and it worked. My guess is that Spark wasn’t able to read the snapshot earlier because the Delta log updates were still in progress. I’ve also implemented the enhancement to switch the checkpoint file from pickle to Lakehouse Delta, and it’s currently being tested. The dataset is about 56 million records pulled through the API, so I don't believe this issue is related to volume. Here are the testing steps and results so far: There are 3 different API calls, all using the same code structure. Before changing the checkpoint file (pickle to delta table) API Call 1 (~400 records): DF Gen2 successfully copied the transformed data from Bronze (Table 1) to Silver. API Call 2 (~55 million records): DF Gen2 consistently threw the error I originally posted. After many attempts, two days later after the post (with no code changes and without rerunning the notebook), DF Gen2 suddenly succeeded in copying the transformed data from Bronze (Table 2) to Silver. API Call 3 (~22 million records): DF Gen2 successfully copied the transformed data from Bronze (Table 3) to Silver. After changing the checkpoint file to delta and rerunning the notebooks (UPSERT applied) API Call 1 (~400 records): DF Gen2 successfully copied the transformed data from Bronze (Table 1) to Silver. API Call 2 (~58 million records): DF Gen2 is again throwing the same error. I created a new DF Gen2 instance, but still no success. API Call 3 (~24 million records): DF Gen2 is now failing with the same error as well. Creating a new DF Gen2 instance did not resolve the issue. Additional Notes Compute capacity has been maxed out during all tests. Data Pipeline runs perfectly fine, but DF Gen2 consistently throws the error. The data is definitely present, this seems to be an issue specifically with DF Gen2’s ability to read or process it under certain load or Delta snapshot conditions. At this point, it’s difficult to pinpoint the exact root cause, as behavior is inconsistent across runs and across the different API loads.928Views0likes1CommentError Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2
Hello, I am encountering the following error when attempting to copy data from the Bronze Lakehouse to the Silver Lakehouse using Dataflow Gen2: Error: Extracted_data_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: 02ddb959-4800-4b38-988b-0c58354041a3). Operation on target ca-47ecc963-4615-4f7f-8a72-dacd9d97d62a failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = 02ddb959-4800-4b38-988b-0c58354041a3". Error code: 999999. (Request ID: 1ef47cdf-9734-4ef3-96eb-cd4748ae0adb). Process Overview: Data is retrieved via API calls. After approximately 10 hours, the session times out. To address this, I implemented a checkpoint file that stores the IDs of records already copied to the Delta table. This allows the process to resume from where it stopped by reconciling IDs from the API with those in the checkpoint file, skipping processed records and continuing with the remaining ones. When the session timed out, I created a Dataflow Gen2 to copy data from Bronze to Silver Lakehouse, which succeeded. After reconnecting the session and running the extraction again, it correctly skipped processed IDs and completed the remaining extraction. However, when I refreshed and ran the Dataflow Gen2 again, I encountered the error above. Creating a new Dataflow Gen2 resulted in the same error. Note: The checkpoint mechanism was implemented for another extraction process, which completed successfully without session timeout. The Dataflow Gen2 is specifically copying data from Bronze to Silver Lakehouse. Implementation Details: Writing to Delta Table: # Append to Delta table df_batch.write.format("delta").mode("append").saveAsTable(lakehouse_table_name) total_records_written += len(batch_data) print(f" Batch written | Data: {len(processed_data_ids)}/{len(data_data)} | Total records: {total_records_written:,}") Saving Checkpoint: # Save checkpoint try: checkpoint = { 'processed_data_ids': list(processed_data_ids), 'failed_data': failed_sites } os.makedirs(os.path.dirname(checkpoint_file), exist_ok=True) with open(checkpoint_file, 'wb') as f: pickle.dump(checkpoint, f) # Save data metadata (lastData tracker) with open(data_metadata_file, 'wb') as f: pickle.dump(data_last_data_tracker, f) Could you please advise on what might be causing this issue and suggest a possible solution? Thank you!Solved3.3KViews1like7CommentsError Encountered When Copying Data from Bronze to Silver Lakehouse Using Dataflow Gen2
Hello, I am encountering the following error when attempting to copy data from the Bronze Lakehouse to the Silver Lakehouse using Dataflow Gen2: Error: Extracted_data_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: 02ddb959-4800-4b38-988b-0c58354041a3). Operation on target ca-47ecc963-4615-4f7f-8a72-dacd9d97d62a failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = 02ddb959-4800-4b38-988b-0c58354041a3". Error code: 999999. (Request ID: 1ef47cdf-9734-4ef3-96eb-cd4748ae0adb). Process Overview: Data is retrieved via API calls. After approximately 10 hours, the session times out. To address this, I implemented a checkpoint file that stores the IDs of records already copied to the Delta table. This allows the process to resume from where it stopped by reconciling IDs from the API with those in the checkpoint file, skipping processed records and continuing with the remaining ones. When the session timed out, I created a Dataflow Gen2 to copy data from Bronze to Silver Lakehouse, which succeeded. After reconnecting the session and running the extraction again, it correctly skipped processed IDs and completed the remaining extraction. However, when I refreshed and ran the Dataflow Gen2 again, I encountered the error above. Creating a new Dataflow Gen2 resulted in the same error. Note: The checkpoint mechanism was implemented for another extraction process, which completed successfully without session timeout. The Dataflow Gen2 is specifically copying data from Bronze to Silver Lakehouse. Implementation Details: Writing to Delta Table: # Append to Delta table df_batch.write.format("delta").mode("append").saveAsTable(lakehouse_table_name) total_records_written += len(batch_data) print(f" Batch written | Data: {len(processed_data_ids)}/{len(data_data)} | Total records: {total_records_written:,}") Saving Checkpoint: # Save checkpoint try: checkpoint = { 'processed_data_ids': list(processed_data_ids), 'failed_data': failed_sites } os.makedirs(os.path.dirname(checkpoint_file), exist_ok=True) with open(checkpoint_file, 'wb') as f: pickle.dump(checkpoint, f) # Save data metadata (lastData tracker) with open(data_metadata_file, 'wb') as f: pickle.dump(data_last_data_tracker, f) <meta http-equiv="Content-Type" content="text/html; charset=us-ascii" /> a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } Could you please advise on what might be causing this issue and suggest a possible solution? Thank you!Solved1.1KViews0likes7Comments
Data Privacy
Microsoft Fabric Community and Privacy
To learn more about how we manage your data, please review the Microsoft Fabric Community Data Privacy guide.