Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
12angrymentiger
Resolver I
Resolver I

Spark SQL Query errors - Case Sensitivity issues

All of a sudden as of almost 2 weeks ago our spark sql notebooks have been failing due to case sensitivity in the table and field names.  This has always worked fine in Synapse before, so I'm not sure if Fabric configs were changed or not.

I'm getting errors like Table or view not found: customers, when I'm running a spark sql query on the customers table

j = spark.read.format("delta").load(f"Files/{storage_account}/silver/Customers/")
j.createOrReplaceTempView("Customers")
 
df=spark.sql(""" SELECT * FROM customers """)
 
The table name is lower case customers in my spark sql query and the view name is Camelcasebut this query works perfectly fine in Synapse spark sql.
I'm also getting the same error on different case field names.
 
Not sure what I'm missing here as it has worked fine before in Fabric.
1 ACCEPTED SOLUTION

Hi @12angrymentiger 
Apologies for the issue you have been facing. At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.

A simple workaround to fix this issue is to run the below piece of code before executing the main code.

spark.conf.set('spark.sql.caseSensitive', False)

Hope this helps. Please let me know if you have any further questions. Glad to help.

View solution in original post

6 REPLIES 6
12angrymentiger
Resolver I
Resolver I

@v-nikhilan-msft  I wanted to check with you regarding this issue.

 Please let me know if you have heard of any fixes for this.

 

Thanks

Hi @12angrymentiger 
Thanks for using Fabric Community. Apologies for the issue you have been facing.
Can you please confirm which Spark Version are you using? 
I have confirmed from the internal team that there were no changes done to the case sensitivity. Did you set any spark configurations? If yes can you please give the details about it.
Thanks.

Hi @12angrymentiger ,
I tried to repro the scenario, but did not get any error. I have attached the screenshot.

vnikhilanmsft_1-1700039523778.png

 

 

Please try to check the Spark version which you ar using.

 

 

vnikhilanmsft_2-1700039588419.png

 

Make sure you use the Spark 1.2 version. 
Hope this helps. Please let me know if you have any further queries.

By the way this is still in My Workspace and under the Fabric Trial, but I did change the workspace to use Runtime 1.2 and I'm still getting the error table or view not found.

Hi @12angrymentiger 
Apologies for the issue you have been facing. At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.

A simple workaround to fix this issue is to run the below piece of code before executing the main code.

spark.conf.set('spark.sql.caseSensitive', False)

Hope this helps. Please let me know if you have any further questions. Glad to help.

We figured out when you call another notebook and start using spark sql further down in the main notebook the spark sql caseSensitive is True.

 

We have moved this code to run after the 2nd notebook call to make sure the main notebook  spark sql is caseInsensitive

 

spark.conf.set('spark.sql.caseSensitive', False)

  

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

March 2024  FBC Gallery Image

Fabric Monthly Update - March 2024

Check out the March 2024 Fabric update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.