Forum Discussion

ArwaAldoud's avatar
ArwaAldoud
Super User
1 year ago
Solved

Unable to Read Multiple Excel Files Using Wildcard (*) inUnable to Read Mult Lakehouse using PySpark

I’m trying to read multiple Excel (.xlsx) files from a folder in Lakehouse using Notebook PySpark. However, when I use a wildcard (*), I get a FileNotFound error.   Code that fails (attempting to r...
data engineering
lakehouse
notebook
  • ArwaAldoud's avatar
    ArwaAldoud
    1 year ago

    Thank you for your detailed response.

    I followed your steps, but unfortunately, it didn’t work for me. Instead, I specified the exact file name in my scenario, and it worked fine.

    I appreciate your support and guidance

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi ArwaAldoud,

Thank you for reaching out in Microsoft Community Forum.

The wildcard (*) in pd.read_excel() is not supported because Pandas expects an exact file path.

please follow below steps to acheive the error;

1. Use dbutils.fs.ls() to get all Excel files in the folder and read them one by one into Pandas.

2. For improved performance and scalability, it is recommended to use PySpark and leverage Spark’s Excel reader (com.crealytics.spark.excel) to efficiently read multiple files.

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.