Forum Discussion
Moving data from S3 to Snowflake
- 1 month ago
There's no self-heal mechanism in the copy activity โ but you've spotted something real. What you're seeing is the staged copy path, and that's where the "healing" impression comes from.
A Lakehouse source (your S3 shortcut) doesn't meet the direct copy criteria for a Snowflake destination, so your copy runs through staging โ which matches the two-step behavior you noticed. The service parses your CSV using the source format settings, rewrites the data into a format the Snowflake COPY command accepts, lands it in Fabric's built-in workspace staging, then runs COPY INTO from there.
The important consequence: on this path, Snowflake never reads your original file. It reads the rewritten version the engine produced. Details: https://learn.microsoft.com/en-us/fabric/data-factory/connector-snowflake-copy-activity
The backslash matters because of that parse step. In the DelimitedText format settings, the default Escape character is backslash and the default Quote character is double quote, so a value like "O'Neal" gets interpreted under those rules before the data is rewritten for staging. Reference: https://learn.microsoft.com/en-us/fabric/data-factory/format-delimited-text
Skip incompatible rows does exactly one thing: it detects rows the engine can't reconcile between source and destination, skips them, and counts them โ plus writes them to a log file if you enable logging under Settings. It never edits a value.
Three checks will tell you what actually happened.
First, look at that value in Snowflake: if it reads O'Neal without the backslash, the escape character was consumed during the parse, which is why the row loaded.
Second, compare rows read vs rows written vs rows skipped in the copy activity output against a COUNT(*) in Snowflake โ read should equal written plus skipped, and if the table holds more than rows written, the extras are leftovers from your earlier failed run (worth a duplicate check, since I don't see a truncate step in your setup).
Third, enable logging and rerun the original file; the log shows exactly which row was skipped and why.
My bet: the value in your table lost its backslash. Let me know which check matches.
๐Parchitect
Solutions Architect ยท Microsoft Fabric Specialist
๐กHelpful? Kudos are appreciated.
โ๏ธSolved? Mark as Solution so others can find it faster.
Hi harrybao0901
I know it might be an obvious question, but I need to ask why you are using. Microsoft Fabric to copy daily between S3 and Snowflake. Shouldn't this be done natively between AWS and Snowflake?
- harrybao09011 month agoFrequent Visitor
Currently, my team uses Fabric as orchestrator, and S3 bucket is managed by a third-party team, files will be uploaded daily. So that's why I use pipeline to copy data into SF.