Forum Discussion

pavarayaaa's avatar
pavarayaaa
Regular Visitor
2 years ago
Solved

Error when writing data into schema enabled lakehouse using notebooks

I was try using PySpark and SparkSQL to write/create data/table into my lakehouse and it gave me an error. I was try using both .save and .saveAsTable options, but it didn't work. When I was using .save, it gave me a bad request error and .saveAsTable gave me a forbidden error. This was when I created the lakehouse with schema enabled. 

 

But, when I created the lakehouse without enabling schema it didn't gave me any error, just run successfully and created the table.

 

What is the issue exactly? Is it with permission or? Please help me out here.

 

TIA.

Lakehouse
  • Anonymous's avatar
    Anonymous
    1 year ago

    Resolution: The issue is caused by the use of internal APIs. In some cases, the internal APIs are returning a 403 when routed to a datacenter where the user is not present. The fix is to move to the public APIs. The code is already in there, but we need to provide the switch from the workload code.

     

    %%pyspark

    !echo "spark.trident.pbiApiVersion=v1">>/home/trusted-service-user/.trident-context


    The fix would be deployed within 3 to 4 weeks in all production regions.
     

7 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Resolution: The issue is caused by the use of internal APIs. In some cases, the internal APIs are returning a 403 when routed to a datacenter where the user is not present. The fix is to move to the public APIs. The code is already in there, but we need to provide the switch from the workload code.

     

    %%pyspark

    !echo "spark.trident.pbiApiVersion=v1">>/home/trusted-service-user/.trident-context


    The fix would be deployed within 3 to 4 weeks in all production regions.
     

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi pavarayaaa ,

     

    Thanks for the reply from pawelpo .

     

    When schema validation is enabled, the system may apply more stringent checks to the data being written. Ensure that the data conforms to the schema.

     

    I created a lakehouse with preview schema enabled and did a lot of testing and found that I get an error when using the following statement “saveAsTable”:

    AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

     

    Using this statement works fine: 

    df = spark.read.format(“csv”).option(“header”, “false”).load(“Files/2019.csv”)

# Create a new table
df.write.format(“delta”).saveAsTable(“test2”)

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

    • arlindTrystar's avatar
      arlindTrystar
      Advocate I
      1 year ago

      This does not address the problem. Because the user is getting bad request error and forbidden error, not AnalysisException error.

       

      • Anonymous's avatar
        Anonymous
        Not applicable
        1 year ago

        Hi pavarayaaa ,

         

        Thanks for the reply from arlindTrystar .

         

        If none of the workaround we provided works, perhaps creating lakehouse without schema enabled is your last option.

         

        Best Regards,
        Yang
        Community Support Team

         

        If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
        If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi pavarayaaa ,

     

    Thanks for the reply from v-ayaanali / arlindTrystar.

     

    Is my follow-up just to ask if the problem has been solved?

     

    I and some users have provided workaround, if it works for you could you accept that answer as a solution to help other members find it faster?

     

    Thank you very much for your cooperation!

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!