Forum Discussion
Copy Job Interface
The checkbox in the table/query selection pane is not a runtime toggle, it is a configuration selector. When you uncheck a query, Fabric treats it as “not part of the job” and removes it from the activity definition rather than just skipping execution. The “Exclude” option, on the other hand, is evaluated at runtime but still assumes the query exists in the config.
Once a custom SQL query is removed this way, dont think there is a native recovery unless you have versioning in place (for ex, git integration or a saved JSON/export of the pipeline). Without that, you will need to recreate those queries manually.
For dev, the practical approach is to avoid using the checkbox as a control mechanism. Instead, split workloads into smaller logical copy activities (for ex, per domain or batch of tables), parameterise the table/query list and drive execution via parameters, or use filters/flags in a metadata table to control which tables run. Another simple workaround is to duplicate the pipeline for testing and prune it there, keeping the original intact. In short, treat copy Job configuration as static and control execution dynamically rather than editing the selection each time.
Hi,
In my experience, this is one of the reasons why I usually prefer a more flexible ingestion approach based on Data Pipelines (Copy Activity) or Notebooks rather than Copy Jobs.
With a metadata-driven design, you can store the configuration for your source tables in a metadata file (for example JSON) and version it in GitHub or Azure DevOps. The ingestion pipeline itself stays generic (typically Lookup → ForEach → Copy Activity) and reads the metadata at runtime.
Once the framework is in place, onboarding a new table is almost as simple as adding it to a Copy Job, but you gain much more flexibility:
- Enable/disable tables without losing configuration
- Version and track changes through Git
- Apply source-specific settings when needed
- Scale to hundreds of tables more easily
- Add custom logic, validation, and error handling
Personally, I prefer storing the metadata in a version-controlled JSON file instead of a metadata table, as it keeps the entire ingestion configuration under source control.
For development and debugging scenarios like yours, this approach makes it very easy to temporarily exclude tables without the risk of accidentally deleting their configuration.
Best regards
Matteo