We have a lot of spark sql scripts where the table and field names are not in the correct CASE which is causing these spark notebooks to fail. Can we turn off the Case Sensitivity for spark sql?
I find the argument against being case insensitive a bit ludicrous. The general de facto "standard" in SQL is actually to be case insensitive on both table and column names.
For some reason, Apache Spark is inconsistent here and is only case insensitive on column names, at least by default.
The Microsoft world view is more human-friendly and is generally case insensitive across the board. So, Microsoft should definitely be consistent here and be case insensitive by default.
Of course, one can make things more restrictive if they want, but in general default settings should appeal to the majority and not the exception.
