Forum Discussion
Nullable = False in schema not correctly preserved / written using pyspark
- 1 year ago
Hello spencer_sa - I've done some more research on this. Based on my findings, while PySpark does include the ability to specify the schema, I believe that when writing a delta table to a lakehouse in this way, the nullability specifications in the schema are not preserved - and all columns set to nullable so that it is optimized for schema-on-read operations. Since the parameters are available that lead the user to think the schema can be explicitly specified, I think it would be good to get feedback from Microsoft so they can confirm.
Thanks for confirming. I've raised as a 'report a problem'.
- Anonymous1 year agoNot applicable
Hi spencer_sa,
Any responded that you received from dev team? If they shared any root causing about these , did you mind sharing them here? I think they will help for other users who faced the similar scenario.
Regards,
Xiaoxin Sheng