Web Scraping Integration with Fabric

Hello Anonymous 

Running Selenium in Fabric notebooks has limitations because the environment lacks a browser for WebDriver to interact with directly. While headless mode or remote Selenium services (e.g., LambdaTest) might be workarounds, they require additional configuration and may not fully resolve all issues

Instead of running Selenium directly in Fabric, you can use a remote Selenium service like LambdaTest or BrowserStack. These services allow you to execute browser automation tasks remotely and retrieve the results back into your Fabric environment.

even consider running Selenium scripts on an external environment (e.g., Azure Virtual Machine or local machine) and then transferring the scraped data to Microsoft Fabric via APIs

You explore Microsoft Power Automate for simpler web scraping tasks that don't require extensive coding

After scraping, the extracted data can be sent to Fabric's data warehouse via APIs or Fabric's Data Factory allows you to create pipelines for ingesting data dynamically from various sources, including APIs. This could involve storing raw JSON responses from APIs in a lakehouse before transforming them into structured tables in the warehouse.

https://community.fabric.microsoft.com/t5/Data-Engineering/Installing-webdriver-for-selenium/m-p/4292547

Hope this is helpful.

Thanks