topic Re: Data pipeline Excel files in Data Engineering

Data pipeline Excel files

sruthiramana — Wed, 23 Oct 2024 21:30:17 GMT

Hello everyone,

I am trying to use a pipeline (Data Engineering) to load excel files that are stored in blob container and then eventually load as table in the Lakehouse.

1) I have the shortcut for the above files In Lakehouse in file section.

2) I have about 3 excel files with about 180 columns and 60k rows.

The data gets loaded successfully but the column names are loaded as Prop_1,Prop_2 etc.

I lose the column names but have all the data loaded.

Another pipeline -

So, I changed the excel files to csv (also removed the space in the column names just to be sure) and ran the pipeline. Though it fails but it did create the empty table with the same column names as in excel files unlike the other pipeline.

Did anyone work on loading multiple excel or csv using pipeline? any insights on this will be helpful.

Re: Data pipeline Excel files

Anonymous — Thu, 24 Oct 2024 06:01:35 GMT

Hi @sruthiramana ,

I used to import Azure Blob as a source into pipeline and it worked fine.

When loading Excel files, the column names are being replaced with names like Prop_1, Prop_2, etc. This often happens when the pipeline does not correctly interpret the header row of the Excel files. Ensure that the pipeline is configured to recognize the first row as headers.

Please refer to the documentation below for detailed information:

How to configure Excel format in the data pipeline of Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

If the pipeline fails but creates an empty table with the correct column names, it suggests that the pipeline can read the headers but encounters issues with the data rows. Check for any special characters or formatting issues in the CSV files that might be causing the pipeline to fail.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Data pipeline Excel files

sruthiramana — Fri, 25 Oct 2024 02:54:44 GMT

Thank you! @Anonymous