Forum Discussion

nikhil0511's avatar
nikhil0511
Icon for Advocate I rankAdvocate I
2 years ago
Solved

Duplicate rows when Full load notebook runs

Hello Forum,    I have a table - named OINV ingested from SAP to Bronze layer lakehouse via a dataflow. I would like to ingest this table from Bronze to Silver lakehouse with the following code.  ...
Need Help
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi nikhil0511 

     

    Apologies for the inconvenience.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi nikhil0511 

 

Thanks for using Microsoft Fabric Community.

It seems like you are experiencing an issue where the number of rows in your table multiplies after the second run of your code. Here are few possible reasons that might help you.

Data Duplication:

  • If your data source (in this case, the ‘OINV’ table in the Bronze layer) is being updated between your runs, and the new data includes rows that were already present in the previous data, this could lead to duplication when you overwrite the Silver layer table.
  • Ensure that the ‘OINV’ table in the Bronze layer isn’t being updated with duplicate rows between your runs.

Caching Issues:

  • Spark, the processing engine you are using, caches data for performance optimization. Sometimes, this can lead to unexpected results if the cache isn’t invalidated properly.
  • Before running your code, you can clear the Spark cache to ensure that you are working with the most recent data. You can do this by calling spark.catalog.clearCache()

If the issue still persists, please do let us know. Glad to help.

 

I hope this information helps.

 

Thank you.