I'm very new to DF, so I need help figuring out the most effcient way to set up a pipeline. I have to copy .zip files from an SFTP, store them in Blob storage, and extract .csv files with certain names from the .zip file to process and move into a database. .zip files are added the SFTP and need to be picked up daily. Each .zip file contains files with different information and different columns (ex. users_todaysdate.csv, demographics_todaysdate.csv, service_rates_todaysdate.csv).
I've been able to copy the files from SFTP to Blob and set the trigger to copy them daily, then, using wildcard file path, copying each file with a specific prefix (users, demographics, service_rates) to a new folder, so they're no longer compressed. However, its not recognizing my headers and keeps adding "PROP_#" as headers to the new file copied into the final folder. Or it tries to force me to use the "merge files" copy behavior, which still doesn't recognize the headers.
How do I fix this? Is there a better, more effcient way for this pipeline?
