Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
ThomasDealFind
Regular Visitor

Notebook Environment, Selenium installation

Hi everyone

 

I am trying to use Selenium python library in a notebook

 

I created an Environment and add Selenium and all library that seemed requiered to make it work but I can't make it work in my notebook

when using the library, my requierement is to use the function driver, in the library webdriver

For that I need to stecify the .exe driver of the webbrowser of my choice, but I can't uoload file other than :

ThomasDealFind_0-1708072225546.png

Someone has try it and made it work ?

 

Thank you in advance

 

PS: my need is to scrap dynamic data from a web page, if you have another solution other than Selenium I would take it gladly

1 ACCEPTED SOLUTION
asanchez2
New Member

Hi Thomas,

 

I have been dealing with selenium installation in a notebook environment without success. In my case, I'm trying to scrap content from a website that involves authentication so I cannot use the provided solution of using Beautifulsoup. So I have to stick to selenium.

 

In that direction, I managed to provide the chromedriver, and make it executable but its execution failed due to the security constraints of the environment. So I found the following solution:

 

SOLUTION

- Using a remote selenium service via LambdaTest

 # Sample code snippet:
    driver = webdriver.Remote(
        command_executor=f'https://{lambda_test_username}:{lambda_test_access_key}@hub.lambdatest.com/wd/hub',
        options=options
    )
 
I hope this helps everyone whose solution has to be approached with selenium. 
Aminael

View solution in original post

9 REPLIES 9
s_arah
Frequent Visitor

I have the same issue. My script runs perfectly in jupyter notebook but now when I try to migrate it to Fabric I am getting errors/WebDriverExeptions. I have tried Beautiful Soup but it doesn't have the capabilities that are needed in this instance. If anyone has a solution on how to get selenium/webdriver installed and working in fabric it would be MUCH appreciated! 

asanchez2
New Member

Hi Thomas,

 

I have been dealing with selenium installation in a notebook environment without success. In my case, I'm trying to scrap content from a website that involves authentication so I cannot use the provided solution of using Beautifulsoup. So I have to stick to selenium.

 

In that direction, I managed to provide the chromedriver, and make it executable but its execution failed due to the security constraints of the environment. So I found the following solution:

 

SOLUTION

- Using a remote selenium service via LambdaTest

 # Sample code snippet:
    driver = webdriver.Remote(
        command_executor=f'https://{lambda_test_username}:{lambda_test_access_key}@hub.lambdatest.com/wd/hub',
        options=options
    )
 
I hope this helps everyone whose solution has to be approached with selenium. 
Aminael

I'm hoping to do something similar but at present my 'options' varaible is coming from this import:

 
from selenium.webdriver.chrome.options import Options
 
which of course, fabric doesnt like.
 
What are you using for options = options?

Hello Aminael,

 

thank you for your response. It confort me with all the test I did.

My last resort was to try a report server and I am glad that you manage to did it.

 

I will try it myself anbd hope it work. thank you for the sample code.

 

It interesting to know that we can't handle authentication with Beautifulsoup. I am still a rookie in this field so at least I won't try this without Selenium.

 

Regards,

Thomas

HimanshuS-msft
Community Support
Community Support

Hello @ThomasDealFind , 
You can package an exe inside a JAR file : https://stackoverflow.com/questions/1383536/including-an-exe-file-to-jar
Thanks 
Himanshu 

 

Hi @ThomasDealFind 

 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.


Thanks.

Hi @ThomasDealFind 


We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.


Thanks.

ThomasDealFind
Regular Visitor

Hello, thank you for your response, I already use beautifull soup but I need to retrieve data not accessible by it, in a javascript content. that is why I completed my script with selenium.

v-cboorla-msft
Community Support
Community Support

Hi @ThomasDealFind 

 

Thanks for using Microsoft Fabric Community.

As I understand that you want to scrap dynamic data from a web page within the Microsoft Fabric. Certainly! Scraping dynamic data from web pages within the Microsoft Fabric environment can be achieved using various approaches.

One approach is to use Beautiful Soup, a Python library for web scraping. You can create a BeautifulSoup object to parse the HTML content of a web page and extract relevant data.

Consider watching this YouTube video that demonstrates an end-to-end project in Microsoft Fabric involving web scraping and data storage in a lakehouse. 

Link : Web Scraping in Microsoft Fabric.

 

I hope this information helps. Please do let us know if you have any further questions.

 

Thanks.

 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.