topic Re: Pyspark Parametrization not working correctly in Data Engineering

Pyspark Parametrization not working correctly

Anonymous — Thu, 23 May 2024 10:48:08 GMT

Hello,

According to the Fabric Runtime 1.2 release notes for Microsoft Fabric Runtime 1.2 in Fabric - Microsoft Fabric | Microsoft Learn, the parameterization of Spark SQL should be supported. However, when I attempt to run a simple SQL query as shown below, I encounter a syntax error.

I get the following syntax error

Could you please help me identify the issue and suggest a solution?

Thank you.

Re: Pyspark Parametrization not working correctly

Anonymous — Fri, 24 May 2024 09:05:11 GMT

Hi Anonymous
Thanks for using Fabric Community.
You can try the below code:

param1 = "abcde"

spark.sql(f"CREATE TABLE {param1}").show()

Hope this helps. Please let me know if you have any further questions.

Re: Pyspark Parametrization not working correctly

Anonymous — Mon, 27 May 2024 06:37:02 GMT

Hey Anonymous ,
Thanks for replying.
Your solution does indeed work , however what I am trying to do is to convert from what you suggested to the format suggested here [SPARK-41271] Parameterized SQL - ASF JIRA (apache.org).
and I don't understand why it is not supported even though it says it is in the fabric runtime relase notes

Re: Pyspark Parametrization not working correctly

Anonymous — Mon, 27 May 2024 10:55:26 GMT

Hi Anonymous
Thanks for bringing this to our notice. The internal team has confirmed this as a bug. I have created an ICM for the same. I will let you know once this gets resolved.

Thanks

Re: Pyspark Parametrization not working correctly

Anonymous — Mon, 27 May 2024 11:27:12 GMT

Hey Anonymous
Thank you for raising this , can you link me the ICM ticket please.

Thanks

Re: Pyspark Parametrization not working correctly

Anonymous — Mon, 27 May 2024 11:36:02 GMT

Hi Anonymous
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.