topic Re: Installing webdriver for selenium in Data Engineering

Installing webdriver for selenium

FelipeM — Wed, 20 Nov 2024 06:37:18 GMT

Has anyone successfully installed a WebDriver in the Fabric environment?

I attempted to place the WebDriver binaries in a Data Lake, but it didn’t work. I need to use Selenium to handle authentication for a single-page application, as Beautiful Soup is insufficient for this task.

I already checked this but they solution is outside fabric/bs4 Solved: Notebook Environment, Selenium installation - Microsoft Fabric Community

Re: Installing webdriver for selenium

spencer_sa — Wed, 20 Nov 2024 08:35:25 GMT

Is there any reason %pip install selenium isn't working? This screenshot is from one of my notebooks.

If you are having issues with the pip repositories, we have managed to load .whl files dynamically from a lakehouse by mounting the lakehouse and loading from there with sys. libraries.

Re: Installing webdriver for selenium

Sephir — Thu, 21 Nov 2024 00:03:51 GMT

Selenium is already installed, but a compatible web driver is required to interact with web browser.

@spencer_sa are you able to start the driver and perform any operation? thanks for your time!

Re: Installing webdriver for selenium

Anonymous — Thu, 21 Nov 2024 06:40:20 GMT

Hi @Sephir @FelipeM

This might be a challenge in Fabric. When using Selenium locally, WebDriver can launch a locally installed browser (such as Chrome) to visit websites. However, in Fabric, notebooks run in an Apache Spark-based environment, which is a virtual environment and may not have a specific browser that WebDriver can launch. Therefore, even if WebDriver is available, the goal cannot be achieved due to the lack of web browser.

I could reproduce the same error as you have got: "WebDriverException: Message: Service /home/trusted-service-user/.cache/selenium/chromedriver/linux64/131.0.6778.85/chromedriver unexpectedly exited. Status code was: 127".

According to the error message, it seems it already has a chromedriver which is linux version. When we are using Selenium v4.6 and above we don't need to explicitly download ChromeDriver, GeckoDriver or any browser drivers as such. Selenium Manager would help to get a working environment to run Selenium out of the box. So it has prepared the chromedriver for the Fabric environment. Therefore I guess the cause of failure is the lack of browser in the Fabric environment.

Update:

I read something about headless mode and browserless mode about Azure and Selenium, not sure if this could be applied to Fabric. I haven't tested this yet, to be honest.

c# - Run Selenium Chrome WebDriver on Azure Cloud Service? - Stack Overflow

How to Run Selenium Driver in Azure for Data Collection | by Illiyas K | Medium

Best Regards,
Jing
Community Support Team

Re: Installing webdriver for selenium

spencer_sa — Thu, 21 Nov 2024 12:19:38 GMT

Looks like the webdriver that gets installed when you use webdriver.Chrome(options=options)

is missing some dependencies - specifically libnss3.so, libnssutil3.so, and libnspr4.so - that aren't installed in the native notebook OS of CBL-Mariner/Azure Linux.
Next steps (beyond contacting the Synapse team) might include trying to get those libraries/library files installed. The package manager is tdnf, rather than apt.
[Edit - same dependency issue if you pip install chromedriver-py too]

(results of ldd <chromedriver path> below)