We have a lot of spark sql scripts where the table and field names are not in the correct CASE which is causing these spark notebooks to fail. Can we turn off the Case Sensitivity for spark sql?
4 Comments
Please adhere to the standard for open source.
For example: “jack” can be used as a verb, meaning to lift or raise something with a jack (e.g., “to jack up a car”).
On the other hand, “Jack” with a capital “J” is typically a proper noun, often used as a name.
These are different sort of words.
If anyone wants to diverge from the standard they can easily add this code:
spark.conf.set('spark.sql.caseSensitive', False)
There's a lot of material out there on collation.
You do not want to force case insensitivity on the rest of the world in 2024.
I find the argument against being case insensitive a bit ludicrous. The general de facto "standard" in SQL is actually to be case insensitive on both table and column names.
For some reason, Apache Spark is inconsistent here and is only case insensitive on column names, at least by default.
The Microsoft world view is more human-friendly and is generally case insensitive across the board. So, Microsoft should definitely be consistent here and be case insensitive by default.
Of course, one can make things more restrictive if they want, but in general default settings should appeal to the majority and not the exception.
- This is a standard for open source Spark, but the Fabric product group considers the idea worthwhile but it has not been planned yet. The community is encouraged to continue voting and the product team will regularly review these ideas at planning.
- Status added:Needs Votes
