topic Re: Adding custom .zip or .egg reference files to PySpark jobs in Data Engineering

Adding custom .zip or .egg reference files to PySpark jobs

gmangiante — Mon, 04 Dec 2023 02:16:36 GMT

Hello-

I'm currently working on a fairly complex Spark ETL job definition that requires a number of additional custom Python files alongside the main file in order to run. I'm currently using the "reference files" feature of Spark jobs in Fabric, but I'm finding it difficult to maintain the job definitions when I have to add 10 ABFSS URLs to the "reference files" area, each one pointing to an individual Python file.

With regular Spark, I would use the --py-files flag on spark-submit to add a .zip or .egg package containing the files, but it seems like I'm only able to add URLs to .py files within the Fabric job definition UI. I know I could create a custom .whl and include it in my Spark environment, but during iterative development and CI/CD, it's much easier to automate creating a .zip file and replacing it via ABFSS than it is to try to figure out how to attach a new .whl to the environment and republish (I'm not even sure the REST APIs exist to do those things).

Am I missing an obvious solution here? Any guidance would be appreciated, and if I'm not missing anything, I guess this is my vote for allowing .zip/.egg files in the "reference files" area of Fabric Spark job definitions. Thanks so much!

Re: Adding custom .zip or .egg reference files to PySpark jobs

Anonymous — Mon, 04 Dec 2023 12:33:52 GMT

Hi @gmangiante ,

Thanks for using Fabric Community,

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Re: Adding custom .zip or .egg reference files to PySpark jobs

Anonymous — Tue, 05 Dec 2023 09:32:47 GMT

Hi @gmangiante ,

Currently this is not supported. Team has taken has feedback on it.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

Re: Adding custom .zip or .egg reference files to PySpark jobs

gmangiante — Tue, 05 Dec 2023 18:20:24 GMT

Appreciate you all taking a look, and will do! Thanks.