Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
SamWilliams
Regular Visitor

Data pipeline Copy Data issue with CSV file and skip line

I have a set of CSV files that require skipping the first line. I have this configured, and the preview works as expected. However, when running the pipeline, it fails with:

 

row number 3: found more columns than expected column count

 

The CSV layout is:

 

Row 1 > Data description label

Row 2 > Blank

Row 3 > headers

Row 4 > data

 

It appears that when the copy is actuall run, the skip line value is being ignored, and it fails when it gets to the headers, because it was expecting a single column based on Row 1.

7 REPLIES 7
SamWilliams
Regular Visitor

So I had looked at setting the skip to 2, and the preview was incorrect, as it shows row 4 as the headers.

 

However, If I actually run it, it works as expected.

 

So the issue appears to be that the preview auto ignores/skips the blank line, but when running the pipeline, it does not.

 

n0elleli
Community Admin
Community Admin

Hi there!

Do you happen to have First row as header selected in your File format settings? And could you share a screenshot of your source settings and advanced source configurations?

Yes, here is the configuration, error, and source file:

 

SamWilliams_0-1685709178830.png

 

SamWilliams_1-1685709214478.png

 

Note that the preview of the data is correct:

SamWilliams_2-1685709252606.png

This is the error when running the pipeline:

 

Error
 
Troubleshoot activity failures
{ "errorCode": "2200", "message": "ErrorCode=DelimitedTextMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'DataflowTest.csv' with row number 3: found more columns than expected column count 1.,Source=Microsoft.DataTransfer.Common,'", "failureType": "UserError", "target": "Copy data1", "details": [] }
 
Source CSV File (5 rows total: label, blank, headers, data):
 
This is a data file header

Column A,Column B,Column C
Row 4A,Row 4B,Row 4C
Row 5A,Row 5B,Row 5C
 
 

Update on my previous reply: 

 

Just tested your case, uploaded a csv and tried different configurations.

 

It might sound counter intuitive I know, but using Skip line count = 2 breaks the preview data, it wont work as expected. But then when you actually run the pipeline, it works fine!

 

MartinZurita_0-1685719784868.png

MartinZurita_1-1685719837384.png

 

 

Here is how preview data looks

MartinZurita_2-1685719876748.png

 

Thanks @MartinZurita. @SamWilliams can you try whether it works for you? While we will log this bug for inconsitent between data preview and run.

Yes, setting skip to 2 works for the actual run, but not preview.

Sam

Have you tried with skip = 2?

 

 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

Top Kudoed Authors