Forum Discussion

harrybao0901's avatar
harrybao0901
Frequent Visitor
1 month ago
Solved

Moving data from S3 to Snowflake

Hi eveyone, I got into an interesting case when moving data from csv file in S3 bucket into Snowflake table. Since I cannot copy data directly from S3 since it kept getting issue with SSL certificate...
  • Parchitect's avatar
    1 month ago

    There's no self-heal mechanism in the copy activity โ€” but you've spotted something real. What you're seeing is the staged copy path, and that's where the "healing" impression comes from.

     

    A Lakehouse source (your S3 shortcut) doesn't meet the direct copy criteria for a Snowflake destination, so your copy runs through staging โ€” which matches the two-step behavior you noticed. The service parses your CSV using the source format settings, rewrites the data into a format the Snowflake COPY command accepts, lands it in Fabric's built-in workspace staging, then runs COPY INTO from there.

    The important consequence: on this path, Snowflake never reads your original file. It reads the rewritten version the engine produced. Details: https://learn.microsoft.com/en-us/fabric/data-factory/connector-snowflake-copy-activity

     

    The backslash matters because of that parse step. In the DelimitedText format settings, the default Escape character is backslash and the default Quote character is double quote, so a value like "O'Neal" gets interpreted under those rules before the data is rewritten for staging. Reference: https://learn.microsoft.com/en-us/fabric/data-factory/format-delimited-text

     

    Skip incompatible rows does exactly one thing: it detects rows the engine can't reconcile between source and destination, skips them, and counts them โ€” plus writes them to a log file if you enable logging under Settings. It never edits a value.

     

    Three checks will tell you what actually happened.

    First, look at that value in Snowflake: if it reads O'Neal without the backslash, the escape character was consumed during the parse, which is why the row loaded.

    Second, compare rows read vs rows written vs rows skipped in the copy activity output against a COUNT(*) in Snowflake โ€” read should equal written plus skipped, and if the table holds more than rows written, the extras are leftovers from your earlier failed run (worth a duplicate check, since I don't see a truncate step in your setup).

    Third, enable logging and rerun the original file; the log shows exactly which row was skipped and why.

     

    My bet: the value in your table lost its backslash. Let me know which check matches.

     

    ๐Ÿ”Parchitect

    Solutions Architect ยท Microsoft Fabric Specialist

    ๐Ÿ’กHelpful? Kudos are appreciated.

    โœ”๏ธSolved? Mark as Solution so others can find it faster.