issue while running Fabric notebook in VScode

Hi, riyajshaikh09 

Ensure that the necessary packages are installed. Run the following command in the VSCode terminal.

pip install semantic-link
pip install azure-core
pip install azure-storage-blob

Make sure your import statements are correct and the Python package is available:

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, BooleanType, StringType
from pyspark.sql.functions import col, max, current_date
import sempy.fabric as fabric
import json
from datetime import datetime

from azure.core.credentials import AccessToken
from azure.storage.blob import BlobServiceClient

To resolve the missing Hadoop filesystem class for Py4JJavaError, make sure that the necessary Hadoop and Azure configurations are set up correctly.
Add the following to your Spark configuration:

spark = SparkSession.builder \
    .appName("MergeWithRowHash") \
    .config("fs.azure.account.auth.type.<your_storage_account>.dfs.core.windows.net", "OAuth") \
    .config("fs.azure.account.oauth.provider.type.<your_storage_account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") \
    .config("fs.azure.account.oauth2.client.id.<your_storage_account>.dfs.core.windows.net", "<your-client-id>") \
    .config("fs.azure.account.oauth2.client.secret.<your_storage_account>.dfs.core.windows.net", "<your-client-secret>") \
    .config("fs.azure.account.oauth2.client.endpoint.<your_storage_account>.dfs.core.windows.net", "https://login.microsoftonline.com/<your-tenant-id>/oauth2/token") \
    .getOrCreate()

Ensure that the path to the Azure Data Lake file system is in the correct format:

df = spark.read.json('abfss://<container-name>@<storage-account>.dfs.core.windows.net/<directory-path>/config.json')

Ensure that hadoop-azure and azure-data-lake-storeJARs are available in the Spark class path. You may need to download the JAR files and place them in the Spark installation jars directory or add them to the Spark configuration:

spark = SparkSession.builder \
    .appName("YourAppName") \
    .config("spark.jars", "/path/to/hadoop-azure.jar,/path/to/azure-data-lake-store.jar") \
    .getOrCreate()

In addition, you should check for any proxy settings or network issues that may be preventing access to Azure services.

hackcrr

If this post helps, then please consider Accept it as the solution and kudos to this post to help the other members find it more quickly