Enable SharePoint File Support in Fabric Copy Jobs
Idea description Microsoft Fabric Copy Jobs currently do not support reading from or writing files directly to SharePoint sites. This limitation makes it difficult to implement straightforward and reusable data integration workflows involving documents and files stored in SharePoint. We would like to see native SharePoint file support added to Fabric Copy Jobs, enabling users to copy files between SharePoint and Fabric storage destinations, without requiring additional workarounds or custom implementation. This capability would simplify data ingestion
Support parameterized file destination paths in Copy job
Currently, the file destination path in a Fabric Copy job is defined statically in the job configuration. Please add support for parameterizing the destination file path, including folder and file names, so that the output location can be dynamically controlled at runtime. For example, it should be possible to define an output path such as: Files/output/{run_date}/{run_id}/ where run_date and run_id are supplied dynamically at runtime. It should also be possible to pass these values from a pipeline or other orchestration process. This would enable a single Copy job definition to be reused across different executions without creating multiple Copy jobs that differ only in their destination paths. Use cases Create execution-specific output folders using run_date and run_id Dynamically partition output files by processing date Prevent outputs from different executions from being mixed or overwritten Pass destination path values from a pipeline or orchestration process Reuse the same Copy job definition across multiple executions and environments
Add ability to manage Fabric capacity by Operation-Kind(s), Night-hours and Workspaces
Fabric Capacity Surge protection and overages are a step in the right direction. Unfortunately the threshold percentage is same for each workspace in the current UI. The ability to have a higher percentage of CU (s) for interactive operations at specific times (for example, in a customer facing company 8-9am or 1-2pm) and the ability to schedule more CUs for background operations at different times in the day helps us control the spread better. Thirdly, let's face it, all workspaces are not equal - like this😀😁! Some workspaces are meant to be for CxOs and some for Accounting. To be able to control the CU (s) in a more granular fashion is the order of the day. I have been requesting Microsoft for years!
Informix support for Incremental Load
Microsoft Fabric Copy Job connector support matrix for Informix connector supports Full Load operations, but it does not currently support Incremental Load (watermark-based) operations Because Fabric Data Factory Copy Jobs don't yet support incremental/delta copy for IBM Informix, we've had to build a custom pipeline (rather than using a native Delta Copy Job) to approximate delta loads from our Informix/Pronto ERP data. This pipeline-based approach consumes considerably more Capacity Units than a native delta copy job would and may not fit sustainably within our Fabric capacity. If it doesn't, we'll be forced back to daily full copy jobs — a significant regression from the hourly delta refresh our legacy SSIS job was able to achieve. The CU cost of running delta via a pipeline (versus a native Copy Job) means the refresh cadence we can sustain is at risk of dropping from hourly to daily. In practice this means data that previously updated every hour now risks sitting stale for most of the day, with no clear way to restore that cadence without native incremental support for Informix. Requesting to add Informix to support Incremental Load as it is crucial part of the operation sustainability and efficiency.Copy Job needs more functionality
Experimenting with Copy Job. Yeah, OK, but not great. Some suggestions: 1) Make the option for OVERWRITE available for Warehouse destinations. Currently available for Lakehouses only. This article (What is Copy job in Data Factory - Microsoft Fabric | Microsoft Learn) does not differentiate that option. Honestly, how hard it is to program in a DROP TABLE IF EXISTS statement in the back end code? Warehouses support such actions. 2) Give the user the abiltiy to DISABLE a table in the list. If I have a list of 50 tables in my job, but during development only want to refresh 2 of them, I have to run the whole job. Waste of time and resources.Allow dynamic content in Activity Name (and other fields on general tab)
We copy a large number of tables by iterating through a ForEach activity and calling the copy activity for each. When trying to answer "which tables take a long time to copy", it's hard because all of the copy activities just have the generic activity name. If I were able to include dynamic content in the Activity name field, I could use code like @concat('Copy activity - ', Item().TableName) and then in the output we could tell which table was being copied without having to dive into the .json input and output. This would also be helpful for other fields on the general tab - like Timeout settings and the like. Then timeout durations could be calculated offline, and fed into the activity vs. hardcoding. That way if the job starts running slower over time, the timeout setting could be gradually increased. p.s. Unable to use the new "Copy Job" activity because it doesn't support dynamic partitioning, needed for good performance. Thanks, ScottLookback window feature for the incremental watermark in copy jobs
Hi! Late-arriving facts are a common problem when loading data incrementally. E.g. the max incremental timestamp is 10:00 after a load, and when we load the next batch with >10:00 we miss some rows tagged with 09:50, who arrived late in the source table. This is usually handled by using a simple lookback window, meaning we load from the latest incremental timestamp subtracting e.g. 1 hour. The new copy jobs are great, but to do a lookback window today we have to revert to the old copy activity with a custom watermark table and query. So thinking this could instead be implemented by adding a lookback window option per table in the new copy jobs! 😊 BR ErikAdd AI Smart Data Validation in Copy Job
I suggest adding an AI Smart Data Validation feature to Data Factory Copy Job. It should automatically detect missing values, duplicate records, data type errors, and mapping issues before copying. This will reduce errors, save time, improve data quality, and make Copy Job easier for all Microsoft Fabric users.Improve Copy Job Performance Monitoring in Microsoft Fabric Data Factory
Problem: Currently, monitoring Copy Jobs in Microsoft Fabric Data Factory can be difficult when handling multiple pipelines and large data transfers. Users have limited visibility into detailed execution metrics, bottlenecks, and historical performance trends. Suggested Improvement: 1. Add detailed Copy Job execution statistics. 2. Display source and destination throughput. 3. Show estimated completion time. 4. Highlight performance bottlenecks. 5. Provide historical execution comparison. 6. Export monitoring reports. 7. Enable custom alerts for slow jobs. 8. Improve filtering and search options. 9. Display failed record details. 10. Provide optimization recommendations. Benefits: • Faster troubleshooting • Better pipeline optimization • Improved monitoring experience • Reduced operational effort • Better performance analysis This enhancement would significantly improve the usability of Microsoft Fabric Data Factory and help organizations manage large-scale data movement more efficiently.
