Copy Job Interface

[rafalencar1997]

Hi everyone,

I'm currently using a Copy Job in Microsoft Fabric to ingest data from an on-premises Oracle database into OneLake.

I have more than 100 tables to load, and for each one I'm using a custom SQL query. Since I'm still developing and debugging this ingestion process, some tables have already been successfully loaded, and I don't want to re-run them every time I test the remaining ones.

I assumed that unchecking the queries I didn't want to execute would simply skip them during the run. However, it appears that when I uncheck a query, Fabric actually removes/excludes that query from the Copy Job configuration.

This behavior doesn't seem intuitive to me, especially because there is already an explicit "Exclude" option available.

My questions are:

1. Is this the expected behavior of Copy Jobs in Microsoft Fabric?
2. Does unchecking a query permanently remove it from the job configuration?
3. Is there any way to recover previously created custom SQL queries that were removed in this way?
4. What is the recommended approach for temporarily disabling certain tables during development and debugging without losing their configurations?

Any guidance or best practices would be greatly appreciated.

Thank you!

[ssrithar]

Hi @rafalencar1997  , 

Based on the current behavior, it appears that the table selection checkboxes are treated as part of the Copy Job definition rather than as a runtime execution option. As a result, deselecting a table and saving the job removes that table (and its custom SQL) from the configuration instead of simply skipping it during execution.

For ingestion scenarios involving 100+ tables, I'd recommend considering a metadata-driven Data Pipeline (Lookup → ForEach → Copy Activity) instead of a single Copy Job. This allows you to enable or disable tables through metadata, version the configuration in Git, and add logging, retries, validation, and environment-specific behavior without modifying the pipeline itself.

If you're using Git integration with your Fabric workspace, it's also worth checking whether a previous version of the artifact can be restored. Otherwise, there doesn't appear to be a native recovery mechanism once a custom SQL definition has been removed.

I also think this would make a good product enhancement—a "Skip for this run" or "Disable" option that preserves the configuration during development and testing would be much more intuitive.

If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

[v-achippa]

Hi @rafalencar1997,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @Vinodh247, @Prince0011 and @matteo_palmieri for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked?  or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @rafalencar1997,

We wanted to kindly follow up to check if the solution provided by the user's for the issue worked?  or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[matteo_palmieri]

Hi,

In my experience, this is one of the reasons why I usually prefer a more flexible ingestion approach based on Data Pipelines (Copy Activity) or Notebooks rather than Copy Jobs.

With a metadata-driven design, you can store the configuration for your source tables in a metadata file (for example JSON) and version it in GitHub or Azure DevOps. The ingestion pipeline itself stays generic (typically Lookup → ForEach → Copy Activity) and reads the metadata at runtime.

Once the framework is in place, onboarding a new table is almost as simple as adding it to a Copy Job, but you gain much more flexibility:

• Enable/disable tables without losing configuration
• Version and track changes through Git
• Apply source-specific settings when needed
• Scale to hundreds of tables more easily
• Add custom logic, validation, and error handling

Personally, I prefer storing the metadata in a version-controlled JSON file instead of a metadata table, as it keeps the entire ingestion configuration under source control.

For development and debugging scenarios like yours, this approach makes it very easy to temporarily exclude tables without the risk of accidentally deleting their configuration.

Best regards

Matteo

[Prince0011]

Hi,

Yeah, this is expected behavior right now, unchecking a table doesn't pause it, it removes it from the saved config once you save. Confusing given "Exclude" already exists and sounds like it should do exactly this.

Quick answers:

1. Expected behavior, not a bug.
2. Yes, unchecking + saving removes that table's config, not just hides it.
3. No recovery option currently, you'd have to rebuild the query manually.
4. For now, best workaround is keeping your custom SQL saved somewhere outside Fabric (Notepad, Excel, whatever) as you write it, so nothing's truly lost. Splitting into smaller Copy Jobs by batch also helps so you're not touching configs you've already validated. Worth testing if Exclude actually behaves differently before relying on unchecking.

Might be worth posting this as a feature idea too, a real pause/skip toggle without losing config seems like a fair ask

[Vinodh247 · Accepted solution]

The checkbox in the table/query selection pane is not a runtime toggle, it is a configuration selector. When you uncheck a query, Fabric treats it as “not part of the job” and removes it from the activity definition rather than just skipping execution. The “Exclude” option, on the other hand, is evaluated at runtime but still assumes the query exists in the config.

Once a custom SQL query is removed this way, dont think there is a native recovery unless you have versioning in place (for ex, git integration or a saved JSON/export of the pipeline). Without that, you will need to recreate those queries manually.

For dev, the practical approach is to avoid using the checkbox as a control mechanism. Instead, split workloads into smaller logical copy activities (for ex, per domain or batch of tables), parameterise the table/query list and drive execution via parameters, or use filters/flags in a metadata table to control which tables run. Another simple workaround is to duplicate the pipeline for testing and prune it there, keeping the original intact. In short, treat copy Job configuration as static and control execution dynamically rather than editing the selection each time.