topic Re: Error when writing data into schema enabled lakehouse using notebooks in Data Engineering

Error when writing data into schema enabled lakehouse using notebooks

pavarayaaa — Wed, 21 Aug 2024 11:16:37 GMT

I was try using PySpark and SparkSQL to write/create data/table into my lakehouse and it gave me an error. I was try using both .save and .saveAsTable options, but it didn't work. When I was using .save, it gave me a bad request error and .saveAsTable gave me a forbidden error. This was when I created the lakehouse with schema enabled.

But, when I created the lakehouse without enabling schema it didn't gave me any error, just run successfully and created the table.

What is the issue exactly? Is it with permission or? Please help me out here.

TIA.

Re: Error when writing data into schema enabled lakehouse using notebooks

pawelpo — Thu, 22 Aug 2024 06:30:51 GMT

I can reproduce your errors. Looks like a bug to me, as Microsoft described in Fabric documentation it should be possible to create new tables using saveAsTable (Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn). Consider reporting it as a bug to Microsoft.

Also, keep in mind that lakehouse schemas are currently in public preview having tons of limitations (Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn).

HTH,

Pawel

Re: Error when writing data into schema enabled lakehouse using notebooks

Anonymous — Thu, 22 Aug 2024 07:21:02 GMT

Hi @pavarayaaa ,

Thanks for the reply from pawelpo .

When schema validation is enabled, the system may apply more stringent checks to the data being written. Ensure that the data conforms to the schema.

I created a lakehouse with preview schema enabled and did a lot of testing and found that I get an error when using the following statement “saveAsTable”:

AnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

Using this statement works fine:

df = spark.read.format(“csv”).option(“header”, “false”).load(“Files/2019.csv”) # Create a new table df.write.format(“delta”).saveAsTable(“test2”)

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Error when writing data into schema enabled lakehouse using notebooks

arlindTrystar — Fri, 23 Aug 2024 12:48:13 GMT

This is a bug, they have not resolved it yet.
Check this: https://community.fabric.microsoft.com/t5/Data-Engineering/An-error-occurred-while-calling-o321-sql/m-p/4091311

It is similar to your problem. We had to redo our work, re-create lakehouses without schema enabled.

Re: Error when writing data into schema enabled lakehouse using notebooks

Anonymous — Fri, 23 Aug 2024 22:52:25 GMT

Resolution: The issue is caused by the use of internal APIs. In some cases, the internal APIs are returning a 403 when routed to a datacenter where the user is not present. The fix is to move to the public APIs. The code is already in there, but we need to provide the switch from the workload code.

%%pyspark

!echo "spark.trident.pbiApiVersion=v1">>/home/trusted-service-user/.trident-context

The fix would be deployed within 3 to 4 weeks in all production regions.

Re: Error when writing data into schema enabled lakehouse using notebooks

Anonymous — Mon, 26 Aug 2024 01:14:43 GMT

Hi @pavarayaaa ,

Thanks for the reply from v-ayaanali / arlindTrystar.

Is my follow-up just to ask if the problem has been solved?

I and some users have provided workaround, if it works for you could you accept that answer as a solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: Error when writing data into schema enabled lakehouse using notebooks

arlindTrystar — Mon, 26 Aug 2024 07:20:18 GMT

This does not address the problem. Because the user is getting bad request error and forbidden error, not AnalysisException error.

Re: Error when writing data into schema enabled lakehouse using notebooks

Anonymous — Tue, 27 Aug 2024 01:18:08 GMT

Hi @pavarayaaa ,

Thanks for the reply from arlindTrystar .

If none of the workaround we provided works, perhaps creating lakehouse without schema enabled is your last option.

Best Regards,
Yang
Community Support Team