Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Error Writing DataFrame to Warehouse via synapsesql – Worked Previously

For the past week, I've been encountering the following error when trying to read from the Lakehouse and write the DataFrame to the Warehouse using synapsesql. Prior to that, everything was working p...
data warehouse
warehouse
BhaveshPatel's avatar
BhaveshPatel
Icon for Super User rankSuper User
1 year ago

Hi Anonymous 

 

This is not assisting us in the first place. What does it mean by silver warehouse? At first, was there a bronze table working before? Can you share some screenshots? 

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    You can eaisly reprodure the error, try with notebook to read data from MS Fabric Lakehouse and save it to MS Fabric Warehouse

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Thanks "BahveshPatel" for your reponse. Sorry, some data and screenshot are sensitives so i can't share that.

    But i can explain again the issue :

    We work with medallion architecture, we use spark notebook to read the data from "bronze lakehouse", and then transforme it and finally we wrote the transformed dataframe in "Fabric Warehouse". This warehouse we name it "Silver warehouse".

    In the notebook, we use "synapsesql" methode to write the dataframe to the warehouse (the source code shared with you). This methode run without any issue until 5 days ago. indicate that we can not list parquet file, and seems to SAS Key to change, but nothing changed in our environnement Fabric.


    Is that clear enough for you ? is that help you to figure out the issue ?

     

    Béchir

    • BhaveshPatel's avatar
      BhaveshPatel
      Icon for Super User rankSuper User
      1 year ago

      Hi Anonymous 

       

      Due to change into synapsesql, means that synapse sql should not be used going forward ( depreciate ) we need to use Databricks Delta Lake Open Source table in broze+ silver warehouse. synapse sql is using data lake whereas databricks delta lake is using _delta_log( Transaction log ) with parquet files. You should not use spark notebook for everything.

       

      Python --> Apache Spark ( Data Lake ) --> Databricks Delta Lake ( Pyspark initially and subsequent use Spark SQL ) 

       

      Also, Lakehouse = Fabric Warehouse = Power BI / SSAS Tabular Semantic Model 

      • Anonymous's avatar
        Anonymous
        Not applicable
        1 year ago

        Thank you, Bhavesh Patel, for your response.

        However, your answer seems to be outside the scope of our current setup. We are working entirely within a Microsoft Fabric environment.

        The objective is to write data from a Microsoft Fabric Lakehouse to a Microsoft Fabric Warehouse using a Fabric Notebook. Databricks is not our scope.

        The synapsesql function was working correctly, and according to Microsoft, it has not been deprecated.