Forum Discussion

pavarayaaa's avatar
pavarayaaa
Regular Visitor
2 years ago
Solved

Error when writing data into schema enabled lakehouse using notebooks

I was try using PySpark and SparkSQL to write/create data/table into my lakehouse and it gave me an error. I was try using both .save and .saveAsTable options, but it didn't work. When I was using .s...
Lakehouse
  • Anonymous's avatar
    Anonymous
    1 year ago

    Resolution: The issue is caused by the use of internal APIs. In some cases, the internal APIs are returning a 403 when routed to a datacenter where the user is not present. The fix is to move to the public APIs. The code is already in there, but we need to provide the switch from the workload code.

     

    %%pyspark

    !echo "spark.trident.pbiApiVersion=v1">>/home/trusted-service-user/.trident-context


    The fix would be deployed within 3 to 4 weeks in all production regions.
     

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi pavarayaaa ,

 

Thanks for the reply from pawelpo .

 

When schema validation is enabled, the system may apply more stringent checks to the data being written. Ensure that the data conforms to the schema.

 

I created a lakehouse with preview schema enabled and did a lot of testing and found that I get an error when using the following statement “saveAsTable”:

AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

 

Using this statement works fine: 

df = spark.read.format(“csv”).option(“header”, “false”).load(“Files/2019.csv”)

# Create a new table
df.write.format(“delta”).saveAsTable(“test2”)

 

If you have any other questions please feel free to contact me.

 

Best Regards,
Yang
Community Support Team

 

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

  • arlindTrystar's avatar
    arlindTrystar
    Advocate I
    1 year ago

    This does not address the problem. Because the user is getting bad request error and forbidden error, not AnalysisException error.

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Hi pavarayaaa ,

       

      Thanks for the reply from arlindTrystar .

       

      If none of the workaround we provided works, perhaps creating lakehouse without schema enabled is your last option.

       

      Best Regards,
      Yang
      Community Support Team

       

      If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
      If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!