topic Re: Notebook Environment, Selenium installation in Data Engineering

Notebook Environment, Selenium installation

ThomasDealFind — Fri, 16 Feb 2024 11:13:20 GMT

Hi everyone

I am trying to use Selenium python library in a notebook

I created an Environment and add Selenium and all library that seemed requiered to make it work but I can't make it work in my notebook

when using the library, my requierement is to use the function driver, in the library webdriver

For that I need to stecify the .exe driver of the webbrowser of my choice, but I can't uoload file other than :

Someone has try it and made it work ?

Thank you in advance

PS: my need is to scrap dynamic data from a web page, if you have another solution other than Selenium I would take it gladly

Re: Notebook Environment, Selenium installation

v-cboorla-msft — Fri, 16 Feb 2024 12:35:11 GMT

Hi @ThomasDealFind

Thanks for using Microsoft Fabric Community.

As I understand that you want to scrap dynamic data from a web page within the Microsoft Fabric. Certainly! Scraping dynamic data from web pages within the Microsoft Fabric environment can be achieved using various approaches.

One approach is to use Beautiful Soup, a Python library for web scraping. You can create a BeautifulSoup object to parse the HTML content of a web page and extract relevant data.

Consider watching this YouTube video that demonstrates an end-to-end project in Microsoft Fabric involving web scraping and data storage in a lakehouse.

Link : Web Scraping in Microsoft Fabric.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.

Re: Notebook Environment, Selenium installation

ThomasDealFind — Fri, 16 Feb 2024 13:01:50 GMT

Hello, thank you for your response, I already use beautifull soup but I need to retrieve data not accessible by it, in a javascript content. that is why I completed my script with selenium.

Re: Notebook Environment, Selenium installation

HimanshuS-msft — Wed, 21 Feb 2024 18:54:02 GMT

Hello @ThomasDealFind ,
You can package an exe inside a JAR file : https://stackoverflow.com/questions/1383536/including-an-exe-file-to-jar
Thanks
Himanshu

Re: Notebook Environment, Selenium installation

v-cboorla-msft — Tue, 27 Feb 2024 13:05:48 GMT

Hi @ThomasDealFind

Thanks.

Re: Notebook Environment, Selenium installation

v-cboorla-msft — Mon, 04 Mar 2024 04:27:30 GMT

Hi @ThomasDealFind

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

Re: Notebook Environment, Selenium installation

asanchez2 — Wed, 06 Mar 2024 13:36:30 GMT

Hi Thomas,

I have been dealing with selenium installation in a notebook environment without success. In my case, I'm trying to scrap content from a website that involves authentication so I cannot use the provided solution of using Beautifulsoup. So I have to stick to selenium.

In that direction, I managed to provide the chromedriver, and make it executable but its execution failed due to the security constraints of the environment. So I found the following solution:

SOLUTION

- Using a remote selenium service via LambdaTest

# Sample code snippet:

driver = webdriver.Remote(

command_executor=f'https://{lambda_test_username}:{lambda_test_access_key}@hub.lambdatest.com/wd/hub',

options=options

)

I hope this helps everyone whose solution has to be approached with selenium.

Aminael

Re: Notebook Environment, Selenium installation

Anonymous — Mon, 11 Mar 2024 17:19:42 GMT

I have the same issue. My script runs perfectly in jupyter notebook but now when I try to migrate it to Fabric I am getting errors/WebDriverExeptions. I have tried Beautiful Soup but it doesn't have the capabilities that are needed in this instance. If anyone has a solution on how to get selenium/webdriver installed and working in fabric it would be MUCH appreciated!

Re: Notebook Environment, Selenium installation

ThomasDealFind — Thu, 28 Mar 2024 17:01:24 GMT

Hello Aminael,

thank you for your response. It confort me with all the test I did.

My last resort was to try a report server and I am glad that you manage to did it.

I will try it myself anbd hope it work. thank you for the sample code.

It interesting to know that we can't handle authentication with Beautifulsoup. I am still a rookie in this field so at least I won't try this without Selenium.

Regards,

Thomas

Re: Notebook Environment, Selenium installation

Shadow229 — Tue, 14 May 2024 14:28:31 GMT

I'm hoping to do something similar but at present my 'options' varaible is coming from this import:

from selenium.webdriver.chrome.options import Options

which of course, fabric doesnt like.

What are you using for options = options?