The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hello-
I'm currently working on a fairly complex Spark ETL job definition that requires a number of additional custom Python files alongside the main file in order to run. I'm currently using the "reference files" feature of Spark jobs in Fabric, but I'm finding it difficult to maintain the job definitions when I have to add 10 ABFSS URLs to the "reference files" area, each one pointing to an individual Python file.
With regular Spark, I would use the --py-files flag on spark-submit to add a .zip or .egg package containing the files, but it seems like I'm only able to add URLs to .py files within the Fabric job definition UI. I know I could create a custom .whl and include it in my Spark environment, but during iterative development and CI/CD, it's much easier to automate creating a .zip file and replacing it via ABFSS than it is to try to figure out how to attach a new .whl to the environment and republish (I'm not even sure the REST APIs exist to do those things).
Am I missing an obvious solution here? Any guidance would be appreciated, and if I'm not missing anything, I guess this is my vote for allowing .zip/.egg files in the "reference files" area of Fabric Spark job definitions. Thanks so much!
Solved! Go to Solution.
Hi @gmangiante ,
Currently this is not supported. Team has taken has feedback on it.
Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.
Hope this helps. Please let me know if you have any further queries.
Hi @gmangiante ,
Currently this is not supported. Team has taken has feedback on it.
Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.
Hope this helps. Please let me know if you have any further queries.
Appreciate you all taking a look, and will do! Thanks.
Hi @gmangiante ,
Thanks for using Fabric Community,
We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
User | Count |
---|---|
6 | |
2 | |
2 | |
2 | |
2 |
User | Count |
---|---|
20 | |
18 | |
6 | |
5 | |
4 |