Notebook Environment, Selenium installation

[ThomasDealFind]

Hi everyone

I am trying to use Selenium python library in a notebook

I created an Environment and add Selenium and all library that seemed requiered to make it work but I can't make it work in my notebook

when using the library, my requierement is to use the function driver, in the library webdriver

For that I need to stecify the .exe driver of the webbrowser of my choice, but I can't uoload file other than :

Someone has try it and made it work ?

Thank you in advance

PS: my need is to scrap dynamic data from a web page, if you have another solution other than Selenium I would take it gladly

[Anonymous]

I have the same issue. My script runs perfectly in jupyter notebook but now when I try to migrate it to Fabric I am getting errors/WebDriverExeptions. I have tried Beautiful Soup but it doesn't have the capabilities that are needed in this instance. If anyone has a solution on how to get selenium/webdriver installed and working in fabric it would be MUCH appreciated! 

[asanchez2 · Accepted solution]

Hi Thomas,

I have been dealing with selenium installation in a notebook environment without success. In my case, I'm trying to scrap content from a website that involves authentication so I cannot use the provided solution of using Beautifulsoup. So I have to stick to selenium.

In that direction, I managed to provide the chromedriver, and make it executable but its execution failed due to the security constraints of the environment. So I found the following solution:

SOLUTION

- Using a remote selenium service via LambdaTest

 # Sample code snippet:

    driver = webdriver.Remote(

        command_executor=f'https://{lambda_test_username}:{lambda_test_access_key}@hub.lambdatest.com/wd/hub',

        options=options

    )

I hope this helps everyone whose solution has to be approached with selenium. 

Aminael

[Shadow229]

I'm hoping to do something similar but at present my 'options' varaible is coming from this import:

from selenium.webdriver.chrome.options import Options

which of course, fabric doesnt like.

What are you using for options = options?

[ThomasDealFind]

Hello Aminael,

thank you for your response. It confort me with all the test I did.

My last resort was to try a report server and I am glad that you manage to did it.

I will try it myself anbd hope it work. thank you for the sample code.

It interesting to know that we can't handle authentication with Beautifulsoup. I am still a rookie in this field so at least I won't try this without Selenium.

Regards,

Thomas

[HimanshuS-msft]

Hello @ThomasDealFind , 
You can package an exe inside a JAR file : https://stackoverflow.com/questions/1383536/including-an-exe-file-to-jar
Thanks 
Himanshu 

[v-cboorla-msft]

Hi @ThomasDealFind 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[v-cboorla-msft]

Hi @ThomasDealFind 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

[ThomasDealFind]

Hello, thank you for your response, I already use beautifull soup but I need to retrieve data not accessible by it, in a javascript content. that is why I completed my script with selenium.

[v-cboorla-msft]

Hi @ThomasDealFind 

Thanks for using Microsoft Fabric Community.

As I understand that you want to scrap dynamic data from a web page within the Microsoft Fabric. Certainly! Scraping dynamic data from web pages within the Microsoft Fabric environment can be achieved using various approaches.

One approach is to use Beautiful Soup, a Python library for web scraping. You can create a BeautifulSoup object to parse the HTML content of a web page and extract relevant data.

Consider watching this YouTube video that demonstrates an end-to-end project in Microsoft Fabric involving web scraping and data storage in a lakehouse. 

Link : Web Scraping in Microsoft Fabric.

I hope this information helps. Please do let us know if you have any further questions.

Thanks.