Forum Discussion

nikhil0511's avatar
nikhil0511
Advocate I
2 years ago
Solved

Duplicate rows when Full load notebook runs

Hello Forum,    I have a table - named OINV ingested from SAP to Bronze layer lakehouse via a dataflow. I would like to ingest this table from Bronze to Silver lakehouse with the following code.  ...
Need Help
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi nikhil0511 

     

    Apologies for the inconvenience.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

nikhil0511's avatar
nikhil0511
Advocate I
2 years ago

Hello V-cboorla-msft,

 

Today I have used the following code to run and see if there is a chance the number of rows doesn't double, after your advice.

spark.catalog.clearCache()
 
%%sql
DROP TABLE IF EXISTS OINV
 
#Load initially full OINV from Bronze to Silver
from pyspark.sql.types import *

source_table_name = 'OINV'
target_table_name = 'OINV_new'

OINV_df = spark.read.parquet('Files/' + source_table_name)
OINV_df\
    .write\
    .mode('overwrite')\
    .format('delta')\
    .save('Tables/' + target_table_name)
 
df = spark.sql("SELECT * FROM DE_LH_200_SILVER_MarketingDocuments.OINV")
df.count()
I have deleted the Tables in Bronze lakehouse that was created by the dataflow. Re-run the dataflow and executed the above Notebook cells. Initially I have 657966 rows. After refreshing the dataflow second time and executing the notebook, the result is 1315948.
 
Seems the caching step and table deletion step is not useful. 
Please note I am connecting the tables from the Brone layer as shortcut in Files/OINV.

 

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi nikhil0511 

     

    Apologies for the inconvenience.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi nikhil0511 

       

      We haven’t heard from you on the last response and was just checking back to see if you've had a chance to submit a support ticket. If you have, a reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

       

      Thank you.

    • nikhil0511's avatar
      nikhil0511
      Advocate I
      2 years ago

      I have created the ticket withsupport as suggested. 

      2406170050002527