I will rephrase the question.
Hi,
PySpark version: 3.3.1.5.2.20240522.3
Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0]
I am getting an error when adding zen-engine in requirements.txt in sparkpool in azure synapse spark pool. The error indicates that there is an issue with building the zen-engine==0.22.0 package during the installation process. The error log points out a failure in preparing metadata for zen-engine from the pyproject.toml
- Following is the error message using requirements.txt:
Collecting zen-engine==0.22.0 (from -r /usr/lib/library-manager/bin/lmjob/sandbox/condaenv.gcir2h2z.requirements.txt (line 35)) Downloading zen_engine-0.22.0.tar.gz (124 kB) 124.8/124.8 kB 19.4 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' - Following is the error while using notebook in sparkpool running in the session: Collecting zen-engine Using cached zen_engine-0.23.0.tar.gz (125 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Checking for Rust toolchain.... Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.
Also, I don't want to to run shell scripts
# Install Rust and Cargo
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
RUN source $HOME/.cargo/env
RUN rustup default
# Add Rust and Cargo to the system's PATH
ENV PATH=$PATH:/root/.cargo/bin
# Verify Rust and Cargo installation
RUN rustc --version
RUN cargo --version
to curl the installer from a public website, then passing the binary to a sub-shell and installing it manually, overriding the session's default PATH and adding the binaries, then manually pip-installing it inside the running session. Would be great to just run of requirements.txt in synapse pyspark manage pools.
