Reusable function for data transformation - user data functions

Hello tinbaj,
Thank you for your patience and for providing detailed feedback.

We recommend raising a support ticket with Microsoft Fabric support for deeper investigation, as the issue may be specific to your workspace or the UDF’s interaction with your Spark environment. You can explain all the troubleshooting steps you have taken to help them better understand the issue.

You can create a Microsoft support ticket with the help of the link below:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

If this information is helpful, consider marking it as “Accept as solution” and giving a “Kudos” to help others in the community.
Thank you.

Hello tinbaj,

Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.

Thank you for your understanding and assistance.

Hello tinbaj,
We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

Hi v-ssriganesh ,

The ticket I raised with Microsoft did not provide any resolution to this issue. The associate classified this problem as more of a pyspark problem than a UDF issue. Please see below how the conversation ended with the Microsoft associate for this ticket.

As reported, you had a User Data Function (UDF) defined to convert a data from oracle database that is stored in Julian format to a data format. 

• It works when simply passing a julian date as input to this function with implementation like this: 
  data_functions = notebookutils.udf.getFunctions('data_functions')
• When you tried below code, it failed with “Error Message: PySparkTypeError: [NOT_ITERABLE] Column is not iterable”. 
  df_silver = df_silver.withColumn("request_date", data_functions.convert_julian_to_date(df_silver.request_date))
• Registering a SQL function (with below code) threw “PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063”. 
  spark.udf.register("convert_julian_to_date", data_functions.convert_julian_to_date)

As discussed, you created Fabric user data function “convert_julian_to_date”. When you used “notebookutils.udf” to get / invoke the function, it processed successfully. These show that Fabric user data function “convert_julian_to_date” itself is working without issues.

Just to clarify, Fabric User data functions uses “fabric.functions” library to provide the functionality. And like what you did, you can retrieve and invoke the function via “notebookutils.udf”. To my knowledge, Fabric User data functions (“fabric.functions” library) basically enables you to create user data functions in Python, not offering other methods (integrated with 3rd-parties like PySpark) by default.

• https://learn.microsoft.com/en-us/fabric/data-engineering/user-data-functions/python-programming-model
• https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#user-data-function-udf-utilities

Apache Spark DataFrames is 3rd-party and not supported by us – So I couldn’t provide the most accurate information for your other questions. I’d assume that you could call / invoke Fabric User data functions from Apache Spark DataFrames, but you might improperly use those PySpark APIs.

I did some research on “PySparkTypeError: [NOT_ITERABLE] Column is not iterable” – It’d be more about the DataFrame and/or the withColumn() usage.

• Can you perform a quick test by df_silver = df_silver.withColumn("request_date", df_silver.request_date)?
• Would it even work?

For getting “PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063” when using spark.udf.register("convert_julian_to_date", data_functions.convert_julian_to_date),

• My best guess would be that spark.udf.register() requires a Python function, pyspark.sql.functions.udf() or pyspark.sql.functions.pandas_udf(). While data_functions.convert_julian_to_date is from notebookutils.udf that it cannot be recognized or directly registered using this method.  

Unfortunately, all I could do was to check the Fabric user data function, and give my assumptions for those PySpark errors. Hope it would be helpful. To move forward, I’d suggest that:

1. Not sure if PySpark provides support to Users – If yes, you may want to contact PySpark support for the issues you’re facing.
2. For further assistance regarding implementation / design of the whole solution you’re trying to accomplish, you can contact Azure sales or get help from an Azure partner.
1. https://azure.microsoft.com/en-us/contact
2. https://azure.microsoft.com/en-us/partners

Hello tinbaj,
We appreciate your patience and sharing the update on the issue.

From what you've described, it looks like the Fabric User Data Function (UDF) itself is working as expected when used with notebookutils.udf. The issues seem to come up only when trying to use it inside PySpark operations like withColumn() or when attempting to register it with spark.udf.register.

Since PySpark is a third-party tool and isn't fully integrated with Fabric UDFs, this kind of limitation is expected for now. Currently, calling Fabric UDFs directly inside PySpark transformations or registering them as Spark SQL functions isn't supported.

If you still need to apply similar logic to your DataFrame, you might want to rewrite the function using a regular PySpark UDF (pyspark.sql.functions.udf() or pandas_udf) so it works smoothly within the PySpark context.

I totally understand this might not be the solution you were hoping for, but given the current capabilities of Fabric, using PySpark-native methods or checking with PySpark support channels would be the best way forward.

Thank you for your understanding.

Hello tinbaj,

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding.

Hello tinbaj,
Thank you for the update on the issue. Please continue to utilize the Microsoft Fabric Community Forum for further discussions and support.