Forum Discussion

anawast's avatar
anawast
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Discrepancy in spark.read.schema in Databricks and Azure Fabric.

Spark.read.json offers 2 functionalities: 1. You can impose a schema on top of spark.read.json using spark.read.schema(schema).json(df). This will ensure that only records that fit the schema will b...
Data Integration
Anonymous's avatar
Anonymous
Not applicable
2 years ago

HI anawast,

I suppose this may be related to the internal processing, they recognize and convert the invalid value to default values. If you do not want this part existed in result, you may need to do filter operation before they load into the data frame.

Regards,

Xiaoxin Sheng