User Data Functions and Spark

Hi Anonymous,

Thank you for reaching out to Microsoft Fabric Community.

Spark is not supported inside User Data Functions, even though you may see pyspark in the library section. UDF’s run in a restricted python environment that does not include a Spark runtime, which is why you are getting the java gateway error.

• Use a Notebook step or Data Pipeline in the same Translytical Task Flow to refresh the Delta table. For example like below:
  df = spark.read.format("parquet").load("Files/<lakehouse_name>/file_path")
  df.write.format("delta").mode("overwrite").save("Tables/<lakehouse_name>/delta_table")
• If your power bi report is connected to this table via Direct Lake mode, it will reflect the updates automatically no need of manual refresh.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa