Add ability to manage Fabric capacity by Operation-Kind(s), Night-hours and Workspaces
Fabric Capacity Surge protection and overages are a step in the right direction. Unfortunately the threshold percentage is same for each workspace in the current UI. The ability to have a higher percentage of CU (s) for interactive operations at specific times (for example, in a customer facing company 8-9am or 1-2pm) and the ability to schedule more CUs for background operations at different times in the day helps us control the spread better. Thirdly, let's face it, all workspaces are not equal - like this😀😁! Some workspaces are meant to be for CxOs and some for Accounting. To be able to control the CU (s) in a more granular fashion is the order of the day. I have been requesting Microsoft for years!
Informix support for Incremental Load
Microsoft Fabric Copy Job connector support matrix for Informix connector supports Full Load operations, but it does not currently support Incremental Load (watermark-based) operations Because Fabric Data Factory Copy Jobs don't yet support incremental/delta copy for IBM Informix, we've had to build a custom pipeline (rather than using a native Delta Copy Job) to approximate delta loads from our Informix/Pronto ERP data. This pipeline-based approach consumes considerably more Capacity Units than a native delta copy job would and may not fit sustainably within our Fabric capacity. If it doesn't, we'll be forced back to daily full copy jobs — a significant regression from the hourly delta refresh our legacy SSIS job was able to achieve. The CU cost of running delta via a pipeline (versus a native Copy Job) means the refresh cadence we can sustain is at risk of dropping from hourly to daily. In practice this means data that previously updated every hour now risks sitting stale for most of the day, with no clear way to restore that cadence without native incremental support for Informix. Requesting to add Informix to support Incremental Load as it is crucial part of the operation sustainability and efficiency.Copy Job needs more functionality
Experimenting with Copy Job. Yeah, OK, but not great. Some suggestions: 1) Make the option for OVERWRITE available for Warehouse destinations. Currently available for Lakehouses only. This article (What is Copy job in Data Factory - Microsoft Fabric | Microsoft Learn) does not differentiate that option. Honestly, how hard it is to program in a DROP TABLE IF EXISTS statement in the back end code? Warehouses support such actions. 2) Give the user the abiltiy to DISABLE a table in the list. If I have a list of 50 tables in my job, but during development only want to refresh 2 of them, I have to run the whole job. Waste of time and resources.
Allow dynamic content in Activity Name (and other fields on general tab)
We copy a large number of tables by iterating through a ForEach activity and calling the copy activity for each. When trying to answer "which tables take a long time to copy", it's hard because all of the copy activities just have the generic activity name. If I were able to include dynamic content in the Activity name field, I could use code like @concat('Copy activity - ', Item().TableName) and then in the output we could tell which table was being copied without having to dive into the .json input and output. This would also be helpful for other fields on the general tab - like Timeout settings and the like. Then timeout durations could be calculated offline, and fed into the activity vs. hardcoding. That way if the job starts running slower over time, the timeout setting could be gradually increased. p.s. Unable to use the new "Copy Job" activity because it doesn't support dynamic partitioning, needed for good performance. Thanks, ScottLookback window feature for the incremental watermark in copy jobs
Hi! Late-arriving facts are a common problem when loading data incrementally. E.g. the max incremental timestamp is 10:00 after a load, and when we load the next batch with >10:00 we miss some rows tagged with 09:50, who arrived late in the source table. This is usually handled by using a simple lookback window, meaning we load from the latest incremental timestamp subtracting e.g. 1 hour. The new copy jobs are great, but to do a lookback window today we have to revert to the old copy activity with a custom watermark table and query. So thinking this could instead be implemented by adding a lookback window option per table in the new copy jobs! 😊 BR ErikAdd AI Smart Data Validation in Copy Job
I suggest adding an AI Smart Data Validation feature to Data Factory Copy Job. It should automatically detect missing values, duplicate records, data type errors, and mapping issues before copying. This will reduce errors, save time, improve data quality, and make Copy Job easier for all Microsoft Fabric users.Improve Copy Job Performance Monitoring in Microsoft Fabric Data Factory
Problem: Currently, monitoring Copy Jobs in Microsoft Fabric Data Factory can be difficult when handling multiple pipelines and large data transfers. Users have limited visibility into detailed execution metrics, bottlenecks, and historical performance trends. Suggested Improvement: 1. Add detailed Copy Job execution statistics. 2. Display source and destination throughput. 3. Show estimated completion time. 4. Highlight performance bottlenecks. 5. Provide historical execution comparison. 6. Export monitoring reports. 7. Enable custom alerts for slow jobs. 8. Improve filtering and search options. 9. Display failed record details. 10. Provide optimization recommendations. Benefits: • Faster troubleshooting • Better pipeline optimization • Improved monitoring experience • Reduced operational effort • Better performance analysis This enhancement would significantly improve the usability of Microsoft Fabric Data Factory and help organizations manage large-scale data movement more efficiently.
Potential limitation in Fabric CopyJob connection parameterization via Variable Library
We have identified a potential limitation while using Microsoft Fabric Variable Library to parameterize SQL Server source connections in Copy Jobs across multiple environments (DEV/UAT/PROD). Scenario We are using Variable Library to manage environment-specific source connections and deploy the same Copy Job definition across environments. DEV Environment Source Connection: DEV SQL Connection Source Database: DEV Database UAT Environment Source Connection: UAT SQL Connection Source Database: UAT Database The objective is to deploy the same Copy Job to higher environments and override the source connection through Variable Library without requiring manual changes. Observation When the Variable Library value is updated to point from the DEV connection to the UAT connection: The connection reference/GUID is updated correctly. However, the database property remains unchanged in the Copy Job definition. Only the connection reference is overridden, while the source database value appears to remain static/hardcoded. As a result, the Copy Job attempts to execute using a mismatched combination of: Updated connection Original database value This leads to execution failures after deployment to higher environments. Impact This behavior prevents fully automated environment promotion and requires manual updates to the database value in each deployed Copy Job. For organizations managing a large number of Copy Jobs across multiple environments, this creates: Additional deployment effort Increased risk of manual configuration errors Reduced effectiveness of Variable Library-based environment parameterization Questions Is it expected behavior that Variable Library overrides only the connection reference while the database property remains unchanged? Is there a supported method to parameterize the source database value within Fabric Copy Jobs? Is this a known limitation or product bug? Are there any recommended workarounds to avoid manual database updates after deployment? Are there any planned enhancements to support full connection rebinding (server and database) during deployment and environment promotion? Expected Behavior When a source connection is overridden through Variable Library during deployment, all associated connection properties required for establishing connectivity, including the target database context, should be updated automatically or be configurable through environment-specific parameterization. Any guidance, clarification, or roadmap information would be greatly appreciated.
Feature request: Fabric Pipelines User Properties
Hello Fabric team, Support for User Properties in Fabric Pipelines would be very valuable for my team's monitoring capabilities. We would like to be able to add user properties to pipeline activities — for example dynamic values such as source file name, table name, or batch ID — and have them surface in the pipeline monitoring view, the same way it works in Azure Data Factory today. Reference: https://learn.microsoft.com/en-us/azure/data-factory/concepts-annotations-user-properties#create-and-use-annotations-and-user-properties Is there anything on the roadmap in this area? Best regards, Rebwar
