topic Re: Custom package publishing in environment takes forever in Data Engineering

Custom package publishing in environment takes forever

smpa01 — Thu, 26 Sep 2024 19:27:18 GMT

Is publishing a custom package supposed to take forever ? I tried experimenting with creating a custom package and publish it in an environment , it is taking forever (1hr+) and still not done. Folder strucutre

//file1.py def add (x:int,y:int): return x+y //file2.py from .file1 import * def square (x:int,y:int): return add(x,y) ** 2

Re: Custom package publishing in environment takes forever

Anonymous — Fri, 27 Sep 2024 05:45:09 GMT

Hi @smpa01

I don't encounter any failure. I published a custom library which was downloaded from pypi.org and it succeeded in around 20 minutes.

What's your experimental result finally? Did it succeed or fail? Did you receive any error message if it failed?

In addition, what's the version of your service and data region? I guess we are not using the same version as I have an additional "Built-in Libraries" section. My service version is 13.0.24328.49 and my data region is UK South. Maybe I'm using a slightly newer version. Deployments of new versions are not pushed to all regions at once. Maybe you can wait a few days and test it out after the version is updated.

In addition, you may try publishing a publicly released custom library. I've never developed a python library, so I'm not sure if the cause would lie in this custom library.

Let me know if you have any questions.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: Custom package publishing in environment takes forever

Anonymous — Mon, 07 Oct 2024 06:49:04 GMT

Hi @smpa01

We haven’t heard from you on the last response and wanted to check if you are still experiencing this issue. If you need further help on this, please provide more details and we will try to help.

Best Regards,
Jing

Community Support Team

Re: Custom package publishing in environment takes forever

sergeik41 — Sat, 07 Dec 2024 12:41:57 GMT

Why it takes 20 minutes to publish? This should not be exceptable. In Databricks, it takes just a fraction of that.

Re: Custom package publishing in environment takes forever

tmjones2 — Thu, 20 Feb 2025 15:01:56 GMT

I also sit here for what seems like a lifetime whenever i upload a newer version of my library...

Re: Custom package publishing in environment takes forever

BryanRock — Wed, 19 Mar 2025 15:58:55 GMT

I just wrote my first R package, so maybe I'm doing something wrong, but I'm having the same issue when I upload a .tar.gz file and try to publish. I waited 30 minutes. When I refreshed the page, it was as if I never clicked publish. No error message or anything.

Re: Custom package publishing in environment takes forever

smpa01 — Wed, 19 Mar 2025 17:01:37 GMT

I have no experience with R unfortunately.

Python has been prioritized over R in our prod for a long time now. Sorry, I can't weigh in here.

Python packages still take ~20 min.