Forum Discussion
Copy data activity - reverts to default behaviour
Hi All,
Don't know if anyone else has experienced this?
We have a lot of Copy data activities, moving files around our network and Lakehouses from "to be imported" -> Archive etc in multiple pipelines. In these activities, the "Delete files after completion" option was checked, files processed then moved to a new location and deleted from the original location - everything was fine.
However, during maintenance on several pipelines (adding activities -Teams notification, Fail Activity) this option reverted to its default state i.e. un-checked during the save operation - nobody went and deliberately un-checked this option on the pipelines. This subtle change to the behaviour wasn’t picked up in check-in and resulted, in files being re-processed.
This option is part of the Advanced properties and is hidden normally (you must click > to expanded it) when looking at the Source tab. Whilst doing the modifications on the pipelines, it’s just not something I or anyone else would have looked at, let alone un-check!
Has anyone experienced this behaviour? When editing pipelines?
Jeff
Hi jj44 ,
Thanks for reaching fabric community,will happy to assist.
As you mentioined the issue, Yes this is a known and frustrating behaviour in Fabric pipelines. Here's a quick breakdown:
What's Happening
- The "Delete files after completion" option lives under Source → Advanced (collapsed by default)
- When you edit a pipeline and save, Fabric can silently reset hidden/advanced settings back to their defaults
- Since it's tucked away and not visible at a glance, it's very easy to miss during reviews exactly as you described
Immediate Fix
- Open every affected Copy activity → Source tab → expand Advanced
- Re-check the "Delete files after completion" option
- Save and validate the pipeline runs as expected before going to PROD
Recommended Long-Term Workaround Separate Delete Activity
The recommended workaround is to uncheck the built-in delete option and instead add a separate Delete activity after the Copy activity with the same file mask/path. Microsoft Learn This gives you:
- Full visibility - delete step is a visible node on the canvas, not hidden in Advanced
- Immune to the reset bug - it won't silently disappear on save
- Safer - the built-in delete is per-file, so if the copy activity fails midway, some files will already be deleted from source Microsoft Lear a separate Delete activity gives you more control over when deletion happens
Pattern to use:
Copy Activity (no delete) → On Success → Delete Activity (same file path/mask)
Prevention Tips
- Always expand Advanced settings on Copy activities before saving any pipeline edits even unrelated ones
- Add a pipeline validation checklist for your team covering hidden settings
- Consider using Git integration for pipelines it lets you diff changes and catch silent resets before they hit PROD
- Log your pipeline runs enable activity logging so file deletions are traceable if something goes wrong
Summary
Problem
Fix
"Delete files" resets on save
Re-check & use separate Delete activity
Setting hidden in Advanced
Separate Delete activity = visible on canvas
Silent change caused reprocessing
Git integration for pipeline change tracking
Definitely worth raising this as a bug/feedback on the Fabric Ideas Forum advanced settings should not silently reset on save.
Please consider marking this as the Accepted Solution to help other community members find this fix more easily. If this helped you, Kudos are always appreciated!
Hope this helps, Jeff!
3 Replies
Hi jj44 ,
Thanks for reaching fabric community,will happy to assist.
As you mentioined the issue, Yes this is a known and frustrating behaviour in Fabric pipelines. Here's a quick breakdown:
What's Happening
- The "Delete files after completion" option lives under Source → Advanced (collapsed by default)
- When you edit a pipeline and save, Fabric can silently reset hidden/advanced settings back to their defaults
- Since it's tucked away and not visible at a glance, it's very easy to miss during reviews exactly as you described
Immediate Fix
- Open every affected Copy activity → Source tab → expand Advanced
- Re-check the "Delete files after completion" option
- Save and validate the pipeline runs as expected before going to PROD
Recommended Long-Term Workaround Separate Delete Activity
The recommended workaround is to uncheck the built-in delete option and instead add a separate Delete activity after the Copy activity with the same file mask/path. Microsoft Learn This gives you:
- Full visibility - delete step is a visible node on the canvas, not hidden in Advanced
- Immune to the reset bug - it won't silently disappear on save
- Safer - the built-in delete is per-file, so if the copy activity fails midway, some files will already be deleted from source Microsoft Lear a separate Delete activity gives you more control over when deletion happens
Pattern to use:
Copy Activity (no delete) → On Success → Delete Activity (same file path/mask)
Prevention Tips
- Always expand Advanced settings on Copy activities before saving any pipeline edits even unrelated ones
- Add a pipeline validation checklist for your team covering hidden settings
- Consider using Git integration for pipelines it lets you diff changes and catch silent resets before they hit PROD
- Log your pipeline runs enable activity logging so file deletions are traceable if something goes wrong
Summary
Problem
Fix
"Delete files" resets on save
Re-check & use separate Delete activity
Setting hidden in Advanced
Separate Delete activity = visible on canvas
Silent change caused reprocessing
Git integration for pipeline change tracking
Definitely worth raising this as a bug/feedback on the Fabric Ideas Forum advanced settings should not silently reset on save.
Please consider marking this as the Accepted Solution to help other community members find this fix more easily. If this helped you, Kudos are always appreciated!
Hope this helps, Jeff!
Thanks Lodha_Jaydeep,
You confirmed my suspicions!
Yes, once I discovered the problem, I did add Delete activities for all the effected pipelines - a belt and braces solution - whilst waiting to hear from the community on whether I'd done something really stupid.
Many thanks,
Jeff
Happy to hear!
thanks
