topic Dataflow retains case sensitivity but not anywhere else in Data Engineering

Dataflow retains case sensitivity but not anywhere else

aarongiust — Wed, 31 Jan 2024 16:28:10 GMT

I created a Lakehouse table using a dataflow and it retains the camel casing I used for the table and column names but if I try to do that myself in a notebook using a CREATE TABLE statement, it just makes it all lowercase. Why is that?

Re: Dataflow retains case sensitivity but not anywhere else

govindarajan_d — Thu, 01 Feb 2024 06:47:31 GMT

Hi @aarongiust,

I tried it but it creates Sentence case for me:

Any other specific config that you are passing?

Re: Dataflow retains case sensitivity but not anywhere else

Anonymous — Thu, 01 Feb 2024 11:39:57 GMT

Hi @aarongiust , @govindarajan_d

Spark is not case sensitive by default.

There is a way to handle this issue by adding spark config , using a SparkSession object named spark:

spark.conf.set('spark.sql.caseSensitive', True)

By default it is False.

Hope this is helpful. Please let me know incase of further queries.