Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
SamWilliams
Regular Visitor

Data pipeline Copy Data issue with CSV file and skip line

I have a set of CSV files that require skipping the first line. I have this configured, and the preview works as expected. However, when running the pipeline, it fails with:

 

row number 3: found more columns than expected column count

 

The CSV layout is:

 

Row 1 > Data description label

Row 2 > Blank

Row 3 > headers

Row 4 > data

 

It appears that when the copy is actuall run, the skip line value is being ignored, and it fails when it gets to the headers, because it was expecting a single column based on Row 1.

7 REPLIES 7
SamWilliams
Regular Visitor

So I had looked at setting the skip to 2, and the preview was incorrect, as it shows row 4 as the headers.

 

However, If I actually run it, it works as expected.

 

So the issue appears to be that the preview auto ignores/skips the blank line, but when running the pipeline, it does not.

 

n0elleli
Community Admin
Community Admin

Hi there!

Do you happen to have First row as header selected in your File format settings? And could you share a screenshot of your source settings and advanced source configurations?

Yes, here is the configuration, error, and source file:

 

SamWilliams_0-1685709178830.png

 

SamWilliams_1-1685709214478.png

 

Note that the preview of the data is correct:

SamWilliams_2-1685709252606.png

This is the error when running the pipeline:

 

Error
 
Troubleshoot activity failures
{ "errorCode": "2200", "message": "ErrorCode=DelimitedTextMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'DataflowTest.csv' with row number 3: found more columns than expected column count 1.,Source=Microsoft.DataTransfer.Common,'", "failureType": "UserError", "target": "Copy data1", "details": [] }
 
Source CSV File (5 rows total: label, blank, headers, data):
 
This is a data file header

Column A,Column B,Column C
Row 4A,Row 4B,Row 4C
Row 5A,Row 5B,Row 5C
 
 

Update on my previous reply: 

 

Just tested your case, uploaded a csv and tried different configurations.

 

It might sound counter intuitive I know, but using Skip line count = 2 breaks the preview data, it wont work as expected. But then when you actually run the pipeline, it works fine!

 

MartinZurita_0-1685719784868.png

MartinZurita_1-1685719837384.png

 

 

Here is how preview data looks

MartinZurita_2-1685719876748.png

 

Thanks @MartinZurita. @SamWilliams can you try whether it works for you? While we will log this bug for inconsitent between data preview and run.

Yes, setting skip to 2 works for the actual run, but not preview.

Sam

Have you tried with skip = 2?

 

 

Helpful resources

Announcements
Sept Fabric Carousel

Fabric Monthly Update - September 2024

Check out the September 2024 Fabric update to learn about new features.

Expanding the Data Factory Forums

New forum boards available in Data Factory

Ask questions in Apache Airflow Job and Mirroring.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

Sept NL Carousel

Fabric Community Update - September 2024

Find out what's new and trending in the Fabric Community.