Enable Table-Level Execution in Fabric Copy Job
a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } Fabric Copy Job currently executes the entire copy job when the Run button is selected. It would be very useful to have the ability to select one or more individual tables and execute only the selected tables without modifying the Copy Job configuration. For example, if a Copy Job contains 10 tables and the mapping or configuration of only one table is modified, we may want to immediately run only that table to validate and apply the change. Currently, running the Copy Job also processes the other tables even though they have not been changed. A simple implementation could be to add checkboxes next to each table in the monitoring/table list and provide an action such as "Run selected tables." This would be particularly useful for: Re-running a table after changing its mapping or configuration Re-loading only the tables that require correction Testing changes without unnecessarily processing unrelated tables Reducing unnecessary data movement and processing Ideally, the existing Run button would continue to execute the entire Copy Job, while a new Run selected tables option would execute only the tables selected by the user. This would allow us to keep multiple related tables in a single production Copy Job while still having the flexibility to execute individual tables when needed.Improve Copy Job Validation Errors When Upsert Keys Are Missing
Improve Copy Job Validation Errors When Upsert Keys Are Missing Product: Microsoft Fabric Area: Data Factory – Copy Job Category: Error Handling / User Experience Problem When adding new tables to an existing Microsoft Fabric Copy Job configured for Change Data Capture (CDC) with Upsert behavior, the designer allows users to configure tables without specifying the required key columns used for matching records. When attempting to save the Copy Job, the following generic error is displayed: "Validation of models failed. Fix it before saving." Unfortunately, this message does not identify which table has an invalid configuration, what property is missing, or how to resolve the issue. This becomes particularly frustrating when a Copy Job contains dozens of tables, because users must manually inspect each table's configuration to identify the problem. Proposed Improvement Improve Copy Job validation to provide actionable, table-specific error messages when required configuration properties are missing or invalid. Specifically: Validate Upsert Keys: When a destination table is configured with Upsert behavior, validate that at least one matching key column has been selected. Identify the Table: Include the source or destination table name in the validation error. Provide Actionable Guidance: Explain which configuration property is missing and how to correct it. Highlight Invalid Tables: Display a validation indicator next to affected tables in the Copy Job designer. Validate Before Save: Detect missing required properties when configuring a table rather than waiting until the user attempts to save the entire Copy Job. Aggregate Validation Errors: When multiple tables have invalid configurations, display all identified issues so users can resolve them in one pass. Example Improved Error Message Copy Job validation failed: 2 tables require attention. dbo.Customer: Upsert requires at least one matching key column. Select a key column under Destination → Upsert settings. dbo.Invoice: Upsert requires at least one matching key column. Select a key column under Destination → Upsert settings. Select the affected table to correct its configuration. Expected Benefits Reduce troubleshooting time for Copy Jobs containing many tables. Make validation errors actionable instead of generic. Improve the experience of adding tables to existing CDC Copy Jobs. Reduce unnecessary support requests. Help users distinguish configuration validation errors from runtime execution failures. Current Behavior vs. Expected Behavior Current: A generic "Validation of models failed" error appears without identifying the invalid table or missing configuration. Expected: The designer identifies the specific table, missing Upsert key configuration, and steps required to resolve the issue. Additional Context This issue was encountered when adding tables to an existing Azure SQL Database → Microsoft Fabric Data Warehouse CDC Copy Job. The existing Copy Job executed successfully, but saving changes failed because newly added tables did not have their Upsert matching keys configured. The underlying configuration issue was straightforward to correct once identified, but the generic validation error provided no useful diagnostic information.📉 How Would You Investigate a 20% Drop in Revenue?
A data analyst shouldn’t stop at identifying what happened. The real questions are: Where did the decline occur? → Why did it happen? → What caused it? → What should we do next? By analyzing regions, products, customers, time periods, and key business metrics, raw data can reveal the root cause behind the numbers. Good data analysis turns numbers into decisions. 📊 #DataAnalysis #BusinessIntelligence #PowerBI #MicrosoftFabric #DataAnalyticsAdd Reliable Change Data Capture With Delete Detection to SAP ABAP Add-On for Copy Job
The SAP ABAP Add-On provides an important option for moving SAP data into Microsoft Fabric, but watermark-based incremental extraction is not sufficient for many production SAP scenarios. SAP tables do not always update a standard timestamp or change-date field for every business change. Deleted source records are also difficult to detect reliably through a watermark column. Please add a native change data capture option for the SAP ABAP Add-On that can reliably identify inserts, updates, and deletes without depending only on a user-selected timestamp field. The capability should maintain a restartable change checkpoint, preserve the operation type, support recovery after interrupted extraction, and expose the current change position and replication lag in monitoring. It should also allow teams to identify tables where native change capture is available and automatically recommend CDC instead of watermark processing. This would make the SAP ABAP Add-On much more suitable for production replication of large SAP ECC and S/4HANA estates and reduce the need for custom reconciliation and delete-detection logic.Allow CDC and Watermark Tables in the Same Copy Job Without Falling Back to Watermark
Enterprise databases commonly contain a mix of tables. Some tables support change data capture, while others need watermark-based incremental processing. When CDC-enabled and non-CDC-enabled tables are included in the same Copy Job, each table should be able to retain the change detection method best suited for that table. Please allow replication mode to be configured independently for each table in a Copy Job. CDC-enabled tables should continue using CDC, while other tables can use a configured watermark column or full copy. Each table should maintain its own checkpoint, reset state, monitoring information, and error status. This would avoid creating multiple Copy Jobs for the same source simply because different tables have different change-tracking capabilities. It would also make Copy Job easier to operate for large enterprise source systems containing hundreds of tables.
Fabric Copy Jobs desperately need basic editability after creation.
Requiring users to recreate an entire Copy Job just to change something as fundamental as the source connection or connection URL is unnecessarily restrictive and creates significant maintenance overhead. A Copy Job should allow users to: Change the source connection Change the destination connection Edit connection details Repoint a job to a replacement connection Preserve existing mappings, scheduling, and other configuration when changing connections Having to rebuild a working job because one connection property needs to change makes Copy Jobs much less useful for production data workflows. Please add the ability to edit or replace connections on an existing Copy Job.Lakehouse connector with Service Principal and Workspace Identity authentication
Currently, the only available authentication method is the Organizational Account. Service Principal and Workspace Identity authentication methods are required. Lakehouse connector overview - Microsoft Fabric | Microsoft LearnBuilt a free tool to inventory & risk-rate legacy SSIS packages before a Fabric migration
Hi all, Sharing something I built that might save some of you the painful "open every .dtsx in SSDT one by one" phase of an SSIS → Fabric migration. The problem it addresses: most teams sitting on a legacy SSIS estate have dozens to hundreds of undocumented packages and no clear starting point. Before any actual migration work, someone has to answer: what does each package do, which components map cleanly to Fabric, and which need a full rewrite? What it does: Parses .dtsx package XML and extracts control flow tasks, data flow components, connection managers, variables, and precedence logic Maps each component against a Fabric equivalent — Data Flow Task → Dataflow Gen2, Execute SQL Task → Script/Stored Procedure Activity, Script Task → Notebook Activity, Foreach Loop → ForEach Activity, and so on Gives each package a Low/Medium/High risk rating based on concrete signals (Script Task usage, Fuzzy Lookup, deprecated CDC/Attunity components, nesting depth, dynamic connection strings) Flags every component individually as auto-mappable or needs-manual-redesign, not just the package as a whole Across a batch of packages, produces a portfolio rollup: risk tier counts, the blockers showing up across the most packages, deprecated components flagged separately, and a suggested migration order What it's built as: a Claude Skill (Anthropic's Claude AI), with the actual XML parsing done by a plain Python script with zero dependencies — not an LLM guessing at package structure. It's designed to run fully offline, no live Azure/Fabric API access assumed, though it flags anywhere that access would improve accuracy (resolving a dynamic connection string, for example). What it deliberately doesn't do yet: this covers the Inventory/Classify phase only. It won't draft actual Fabric pipeline JSON or claim a package is production-ready — every future draft artifact gets an explicit DRAFT/NEEDS REVIEW label. I'd rather it under-claim than over-claim. Repo's on GitHub with a sample package and example report included, so you can see the actual output before running it against your own packages: https://github.com/HBBH11/ssis-fabric-migration-assistant Genuinely interested in feedback from people who've done these migrations for real — particularly whether the risk heuristics hold up against messier packages than my test case, and whether the Fabric equivalence mappings match what you've found in practice. Happy to take suggestions on what a Phase 2 (draft pipeline generation) should prioritize too.
Enable SharePoint File Support in Fabric Copy Jobs
Idea description Microsoft Fabric Copy Jobs currently do not support reading from or writing files directly to SharePoint sites. This limitation makes it difficult to implement straightforward and reusable data integration workflows involving documents and files stored in SharePoint. We would like to see native SharePoint file support added to Fabric Copy Jobs, enabling users to copy files between SharePoint and Fabric storage destinations, without requiring additional workarounds or custom implementation. This capability would simplify data ingestionSupport parameterized file destination paths in Copy job
Currently, the file destination path in a Fabric Copy job is defined statically in the job configuration. Please add support for parameterizing the destination file path, including folder and file names, so that the output location can be dynamically controlled at runtime. For example, it should be possible to define an output path such as: Files/output/{run_date}/{run_id}/ where run_date and run_id are supplied dynamically at runtime. It should also be possible to pass these values from a pipeline or other orchestration process. This would enable a single Copy job definition to be reused across different executions without creating multiple Copy jobs that differ only in their destination paths. Use cases Create execution-specific output folders using run_date and run_id Dynamically partition output files by processing date Prevent outputs from different executions from being mixed or overwritten Pass destination path values from a pipeline or orchestration process Reuse the same Copy job definition across multiple executions and environments
