Forum Discussion

vladenache's avatar
vladenache
New Member
8 hours ago

syncing workspace from GitHub with large files

Hello

The team and I are having a problem with one of our environments. It is used in a specific snowflake ingestion, and has some jar files needed for spark to connect to snowflake. These were already big to begin with, but switching to runtime 2.0 and thus spark 4.1, we need to upgrade the existing jars and now they are too big for Fabric to sync with!

Cannot commit to Git nor Sync from a Git branch containing such a big environment file. Fabric won't allow it. This is not a Git issue, as it allows files up to 2-4 GB (depending on the version), but a Fabric platform issue. It was present a couple of months ago, Microsoft solved it and incresead the size of the files that could be sync-ed by Fabric, but now apparently we overstepped that borderline also.

Anybody can help with this? Copy data activity is not an option for us currently, we need a notebook activity.

 

Thanks

workspaces

2 Replies

  • vladenache's avatar
    vladenache
    New Member
    6 hours ago

    hei benibix​ , thanks for your answer

    so the jar files lie in the custom library section of the environment:

    these are the old ones , which are already pretty big - in total they are almost 80MB.


    upgrading to the new spark version, like i said, will need us to update these jars to newer versions which are over 100 MB in total.



    20 MB extra is all there is , but we are getting errors from fabric when trying to commit to git. I suspect it will also happen when trying to sync from GIt

    I'm not quite sure what is the GIt 25MB file size, since it was clearly exceeded until now and had no issues.

     

  • benibix's avatar
    benibix
    Icon for Helper V rankHelper V
    7 hours ago

    Hi vladenache

    Two limits in the docs could be the one you hit, and they differ a lot. For an environment, each commit is capped at 150 MB, and custom libraries larger than that can't be committed through Git. For Git integration in general, the page lists a 25 MB maximum file size, and with GitHub the combined size of the files in one commit is capped at 50 MB.

    Your jars sit in an environment, so the 150 MB cap should be the one that applies, but I can't tell that from here. Add up the jars in the environment's CustomLibraries folder and compare. If the total is over it, the environment page doesn't describe a way around that, so the jars would have to be added to the environment in Fabric and not through Git.

    What's the exact error text when the commit or sync fails?