Turn off Case Sensitivity for Spark SQL

Please adhere to the standard for open source.

For example: “jack” can be used as a verb, meaning to lift or raise something with a jack (e.g., “to jack up a car”).

On the other hand, “Jack” with a capital “J” is typically a proper noun, often used as a name.

These are different sort of words.

If anyone wants to diverge from the standard they can easily add this code:

spark.conf.set('spark.sql.caseSensitive', False)

There's a lot of material out there on collation.

You do not want to force case insensitivity on the rest of the world in 2024.

I find the argument against being case insensitive a bit ludicrous. The general de facto "standard" in SQL is actually to be case insensitive on both table and column names.

For some reason, Apache Spark is inconsistent here and is only case insensitive on column names, at least by default.

The Microsoft world view is more human-friendly and is generally case insensitive across the board. So, Microsoft should definitely be consistent here and be case insensitive by default.

Of course, one can make things more restrictive if they want, but in general default settings should appeal to the majority and not the exception.