Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
gmangiante
Frequent Visitor

Adding custom .zip or .egg reference files to PySpark jobs

Hello-

I'm currently working on a fairly complex Spark ETL job definition that requires a number of additional custom Python files alongside the main file in order to run. I'm currently using the "reference files" feature of Spark jobs in Fabric, but I'm finding it difficult to maintain the job definitions when I have to add 10 ABFSS URLs to the "reference files" area, each one pointing to an individual Python file.

 

With regular Spark, I would use the --py-files flag on spark-submit to add a .zip or .egg package containing the files, but it seems like I'm only able to add URLs to .py files within the Fabric job definition UI. I know I could create a custom .whl and include it in my Spark environment, but during iterative development and CI/CD, it's much easier to automate creating a .zip file and replacing it via ABFSS than it is to try to figure out how to attach a new .whl to the environment and republish (I'm not even sure the REST APIs exist to do those things).

 

Am I missing an obvious solution here? Any guidance would be appreciated, and if I'm not missing anything, I guess this is my vote for allowing .zip/.egg files in the "reference files" area of Fabric Spark job definitions. Thanks so much!

1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi @gmangiante ,

Currently this is not supported. Team has taken has feedback on it.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

View solution in original post

3 REPLIES 3
Anonymous
Not applicable

Hi @gmangiante ,

Currently this is not supported. Team has taken has feedback on it.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

Appreciate you all taking a look, and will do! Thanks.

Anonymous
Not applicable

Hi @gmangiante ,

Thanks for using Fabric Community,

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Helpful resources

Announcements
Fabric July 2025 Monthly Update Carousel

Fabric Monthly Update - July 2025

Check out the July 2025 Fabric update to learn about new features.

July 2025 community update carousel

Fabric Community Update - July 2025

Find out what's new and trending in the Fabric community.