Forum Discussion
Solved
Duplicate rows when Full load notebook runs
Hello Forum, I have a table - named OINV ingested from SAP to Bronze layer lakehouse via a dataflow. I would like to ingest this table from Bronze to Silver lakehouse with the following code. ...
Hi nikhil0511
Apologies for the inconvenience.
Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.
Please go ahead and raise a support ticket to reach our support team:
https://support.fabric.microsoft.com/support
After creating a Support ticket please provide the ticket number as it would help us to track for more information.
Thank you.
Hello V-cboorla-msft,
Thank you for the response.
We are indeed using the clear cache code for one of our tables from Bronze to silver.
# Step 1: Unpersist the DataFrame if it is cached
if 'epages_web_stores_df' in locals() or 'epages_web_stores_df' in globals():
epages_web_stores_df.unpersist()
# Step 2: Set the DataFrame variable to None
epages_web_stores_df = None
Even after using this and use the same code that overwrites the table data, we are still seeing the entire table with duplicate rows. The number of rows simply double. Even in the first run.