Forum Discussion

arpost's avatar
arpost
Icon for Post Prodigy rankPost Prodigy
2 years ago
Solved

How to handle quotes within a string in Copy Activity?

Greetings, community. I have a CSV file I am trying to load into a Lakehouse using a Copy Activity and am having some trouble related to characters in the file. Here is a fake data sample: "ColumnA"...
Need Help
pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi arpost ,

    As I understand, without having any escape characters we cannot handle this kind of data in data pipeline. This data will be treated as bad data and make it to null. You can continue using Data flow as you were able to find a solution with the help of it.

    Thank you

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi arpost ,

Please try to check this video - Link 

I can also find issue with you data - Double Quotes is missing.


I guess this should be like this -

"ColumnA","ColumnB"
"1","Alpha"
"2","Alphabet
"Soup""
"3","Beta" 

 
Please let me know if this works for you.

arpost's avatar
arpost
Icon for Post Prodigy rankPost Prodigy
2 years ago

Thank you for the reply, Anonymous, and for pointing that out. I fixed that in the examples; must have dropped it somehow. Corrected data is as follows:

 

"ColumnA","ColumnB"
"1","Alpha"
"2","Alphabet
"Soup""
"3","Beta"

The text in orange above is the incoming data that is malformed (no additional escape characters around the quotes). What I'm trying to do is see if Data Factory can overcome this issue with any of its settings.

 

I've been able to solve for this pretty easily with a dataflow, but I want to use a pipeline for this operation.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi arpost ,

    As I understand, without having any escape characters we cannot handle this kind of data in data pipeline. This data will be treated as bad data and make it to null. You can continue using Data flow as you were able to find a solution with the help of it.

    Thank you

    • arpost's avatar
      arpost
      Icon for Post Prodigy rankPost Prodigy
      2 years ago

      Anonymous, bummer. Thanks for the info and for looking into it, though. Dataflow really isn't ideal (if for no other reason than dataflows aren't supported in deployment pipelines), so I'll keep looking.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi arpost ,

        Glad to know that you got some insights over your query. Please continue using Fabric Community on your further queries.