Forum Discussion

westf's avatar
westf
Regular Visitor
1 year ago

Problem loading Lakehouse file/table into warehouse

Hello,  I'm facing issues loading data from Fabric Lakehouse Table/File to Fabric Warehouse. Below is my notebook code. Im using pyspark       lakehouse_table_path = f"{lakehouse_path}/Tables/{...
data pipeline
data warehouse
Mirrored Azure SQL Database
warehouse
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi westf 

 

It seems df.to_sql() is a Pandas dataframe function. I haven't tested it yet. 

 

BTW, do you consider using copy data activity in data pipeline to copy data from lakehouse into warehouse? You can set up the column mapping within the activity. This will be easier. 

How to copy data using copy activity - Microsoft Fabric | Microsoft Learn

 

Best Regards,
Jing
Community Support Team

  • westf's avatar
    westf
    Regular Visitor
    1 year ago

    Thanks for the reply. i can't use copy activity becuase im writing dynamic pipeline using notebook. All i need is a way to push Lakhouse table data into Warehouse.

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Hi westf 

       

      You may check the data type of the datetime column in the Pandas dataframe. In my testing, I got the following datetime64. It may be not compatible with the datetime2 type in the destination warehouse. I guess this might be the cause of the error. Not sure if this data type could be converted to meet the target. You may research this further. 

      Best Regards,
      Jing