Forum Discussion

IMett's avatar
IMett
Icon for Helper III rankHelper III
2 years ago

Using Data from Lakehouse in any way fails - what am I doing wrong?

Hi everyone, I am quite frustrated and need some help. I've created a Lakehouse in Fabric and created some Delta tables there using a PySpark Notebook. Unfortunately, now I am stuck and don't kno...
data engineering
lakehouse
AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Hi IMett what's the method in which you're creating the Dekta tables? Can you post your process (code etc).  

Have you tried creating a new Lakehouse and repeating the process?

 

It would also be worth raising a support ticket with Microsoft so they can investigate too.

  • IMett's avatar
    IMett
    Icon for Helper III rankHelper III
    2 years ago

    Hi Andy,

    Thank you for your reply.

    I create the data from scratch in python. The data is a simulation experiment where data rows are created, no data is ingested from other sources.

    The relevant parts of the codes are those:

    ergebnisse = pd.DataFrame({
        'ExperimentID': [0],
        'WurfID': [0],
        'farbe': ['test'],
        'buchstabe': ['X'],
        'Typ': ['nix'],
        'AnzahlMarker': [-1],
        'Gescheitert': [False],
        'AnzahlSchiffe': [0]
    })
    def sim_0schwerter(anzahl_experimente, tableName, ergebnisse = ergebnisse):
        results = []
     
    ... some logic to create the results in a for loop...

     

        ergebnisse = pd.DataFrame(results, columns=ergebnisse.columns)
        erg_sparkdf=spark.createDataFrame(ergebnisse)
               
        table_path = "Tables/" + tableName
        erg_sparkdf.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(table_path)
     
    sim_0schwerter(anzExpr, "NoSwordsBig",ergebnisse=ergebnisse)
     
    I have tried to copy everything to a different (new) Lakehouse, same result.
    Is there something wrong with the last statement, where the delta file is created?
    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi IMett ,

      Just want to check are you still facing the issue?

      If yes, the best course of action is to open a support ticket and have our support team take a closer look at it.

       

      Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

       

      After creating a Support ticket please provide the ticket number as it would help us to track for more information.

       

      Hope this helps. Please let us know if you have any other queries.

      • IMett's avatar
        IMett
        Icon for Helper III rankHelper III
        2 years ago

        Hi Anonymous 
        I opened a support ticket under ticket # 2405290050004525