topic Re: Data pipeline Copy Data issue with CSV file and skip line in Pipelines

Data pipeline Copy Data issue with CSV file and skip line

SamWilliams — Thu, 01 Jun 2023 18:06:33 GMT

I have a set of CSV files that require skipping the first line. I have this configured, and the preview works as expected. However, when running the pipeline, it fails with:

row number 3: found more columns than expected column count

The CSV layout is:

Row 1 > Data description label

Row 2 > Blank

Row 3 > headers

Row 4 > data

It appears that when the copy is actuall run, the skip line value is being ignored, and it fails when it gets to the headers, because it was expecting a single column based on Row 1.

Re: Data pipeline Copy Data issue with CSV file and skip line

n0elleli — Fri, 02 Jun 2023 03:48:59 GMT

Hi there!

Do you happen to have First row as header selected in your File format settings? And could you share a screenshot of your source settings and advanced source configurations?

Re: Data pipeline Copy Data issue with CSV file and skip line

SamWilliams — Fri, 02 Jun 2023 12:42:47 GMT

Yes, here is the configuration, error, and source file:

Note that the preview of the data is correct:

This is the error when running the pipeline:

Error

Troubleshoot activity failures

{ "errorCode": "2200", "message": "ErrorCode=DelimitedTextMoreColumnsThanDefined,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error found when processing 'Csv/Tsv Format Text' source 'DataflowTest.csv' with row number 3: found more columns than expected column count 1.,Source=Microsoft.DataTransfer.Common,'", "failureType": "UserError", "target": "Copy data1", "details": [] }

Source CSV File (5 rows total: label, blank, headers, data):

This is a data file header

Column A,Column B,Column C

Row 4A,Row 4B,Row 4C

Row 5A,Row 5B,Row 5C

Re: Data pipeline Copy Data issue with CSV file and skip line

Anonymous — Fri, 02 Jun 2023 15:12:35 GMT

Have you tried with skip = 2?

Re: Data pipeline Copy Data issue with CSV file and skip line

SamWilliams — Fri, 02 Jun 2023 15:19:37 GMT

So I had looked at setting the skip to 2, and the preview was incorrect, as it shows row 4 as the headers.

However, If I actually run it, it works as expected.

So the issue appears to be that the preview auto ignores/skips the blank line, but when running the pipeline, it does not.

Re: Data pipeline Copy Data issue with CSV file and skip line

Anonymous — Fri, 02 Jun 2023 15:31:24 GMT

Update on my previous reply:

Just tested your case, uploaded a csv and tried different configurations.

It might sound counter intuitive I know, but using Skip line count = 2 breaks the preview data, it wont work as expected. But then when you actually run the pipeline, it works fine!

Here is how preview data looks

Re: Data pipeline Copy Data issue with CSV file and skip line

GraceGu — Mon, 05 Jun 2023 07:29:04 GMT

Thanks Anonymous. SamWilliams can you try whether it works for you? While we will log this bug for inconsitent between data preview and run.

Re: Data pipeline Copy Data issue with CSV file and skip line

SamWilliams — Mon, 05 Jun 2023 12:47:48 GMT

Yes, setting skip to 2 works for the actual run, but not preview.

Sam