Forum Discussion

bhavya5903's avatar
bhavya5903
Icon for Advocate II rankAdvocate II
1 year ago
Solved

How to Append Data to a CSV File in Fabric Copy Data Activity?

I am ingesting JSON data from an API as a source in Microsoft Fabric and handling pagination within a loop. The data is successfully parsed and written to a CSV file using the Copy Data activity. How...
  • nilendraFabric's avatar
    1 year ago

    Hello bhavya5903 

     

    NandanHegde  is absolutely right. Here are more details 

     

    appending data directly to an existing CSV file using the Copy Data activity in Microsoft Fabric is not natively supported. The Copy Data activity in Fabric is designed to write to a destination in either overwrite or create mode, but it does not support appending data to an existing file directly.


    Why Appending Is Not Supported


    The Copy Data activity overwrites the file in the sink by default because it treats each execution as a new write operation. CSV files are not structured like databases, so they lack native support for incremental updates or appends within the same file.

     


    Workarounds to Achieve Appending


    If you need to append data to a CSV file, you can use one of the following approaches:
    1. Use a Staging File and Merge
    • Write each iteration of data into separate files (e.g., `file1.csv`, `file2.csv`).
    • At the end of the loop or pipeline execution, use a script or another process (e.g., a notebook or Dataflow) to merge all these files into a single CSV file.

     

    better approach would be to in Dataflow Gen2, ingest JSON by parsing or flattening it, then configure the Delta table as the destination with “Append” mode.

     

     

    hope this helps 

     

    if this is helpful please accept the answer and give kudos