Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Error while writing PySpark code to load the csv file data into a dataframe

Hi,   I'm new to PySpark. While writing PySpark code, to load the csv file data into a dataframe, I'm getting following error - Here is my code - Here is my lake house -   Ple...
  • v-veshwara-msft's avatar
    1 year ago

    Hi Anonymous 

    Thanks for using Microsoft Fabric Community.
    The error you are facing is likely due to there is no Lakehouse attached to the Notebook. 
    From the images , in the Explorer pane under Resources there is no Lakehouse added.

    Please add the Lakehouse where the CSV file exists and retry.

    Alternatively, you can include the ABFS path in the code to avoid attaching the Lakehouse.:
    Here is the updated code:

    from pyspark.sql import SparkSession
    spark = SparkSession.builder.appName("CSV Reader").getOrCreate()
    # ABFS Path for the file
    abfs_path = "Your ABFS path here"
    df = spark.read.format("csv") \
        .option("header", "true") \
        .load(abfs_path)
    display(df.limit(10)

    To obtain the ABFS path for the CSV file:
    Navigate to Lakehouse ->  Files -> data -> products.csv, then check the properties of the csv to copy the ABFS path.

    Hope it works.
    If this post helps please accept as solution to help others find it and a kudos would be appreciated.

    Best Regards,
    Vinay