Forum Discussion

amvans_90's avatar
amvans_90
Regular Visitor
1 year ago

Copy data not recognizing existing headers

I'm very new to DF, so I need help figuring out the most effcient way to set up a pipeline.  I have to copy .zip files from an SFTP, store them in Blob storage, and extract .csv files with certain names from the .zip file to process and move into a database.  .zip files are added the SFTP and need to be picked up daily. Each .zip file contains files with different information and different columns (ex. users_todaysdate.csv, demographics_todaysdate.csv, service_rates_todaysdate.csv).

 

I've been able to copy the files from SFTP to Blob and set the trigger to copy them daily, then, using wildcard file path, copying each file with a specific prefix (users, demographics, service_rates) to a new folder, so they're no longer compressed.  However, its not recognizing my headers and keeps adding "PROP_#" as headers to the new file copied into the final folder.  Or it tries to force me to use the "merge files" copy behavior, which still doesn't recognize the headers.  

 

How do I fix this? Is there a better, more effcient way for this pipeline?

Need Help

3 Replies

  • v-lgarikapat's avatar
    v-lgarikapat
    Community Support
    1 year ago

    Hi amvans_90 ,

    Thank you for reaching out to the Microsoft Fabric Community Forum.

    Thanks for sharing the details of your pipeline setup

    Just to make sure I provide the most accurate help, could you please confirm whether you're building this pipeline in Azure Data Factory or in Microsoft Fabric's Data Factory? Both tools offer similar capabilities, but some features and setup steps differ slightly between the two.

     

    Once I know which platform you're using, I’ll be happy to suggest the most efficient way to handle the .zip extraction and ensure the headers are correctly processed in your CSV files.

    Looking forward to your reply.

     

    Best regards,
    Lakshmi Narayana

      • v-lgarikapat's avatar
        v-lgarikapat
        Community Support
        1 year ago

        amvans_90 ,

        Thank you for the confirmation. Since your issue appears to be specific to Azure Data Factory, I would recommend posting it in the Azure Data Factory Community Forum. That forum is more focused on ADF-related topics, and you're likely to receive more targeted guidance from experts there.

         

        Best regards,
        Lakshmi Narayana