We currently achieve metadata-driven table and column selection in Copy Data activities by using dynamic queries. This allows us to dynamically determine which tables and columns should be processed based on metadata, without manually modifying the pipeline configuration.
We would like to have similar capabilities available with Copy Job.
Our requirement is to control the Copy Job dynamically at both table and column level using metadata.
At the table level, we would like to maintain a metadata flag such as Load_Enabled to determine whether a table should be included in the Copy Job. For example, Table 1 can be enabled, Table 2 can be enabled, and Table 3 can be disabled. The Copy Job should dynamically process only the enabled tables.
At the column level, we would like to use metadata to determine which columns should be selected and loaded. For example, only columns where Load_Enabled = 1 should be included in the Copy Job.
We currently achieve similar functionality in Copy Data using dynamic queries. If a similar metadata-driven or dynamic-query mechanism could be supported in Copy Job, it would be extremely helpful for our implementation.
The requested capability would allow Copy Job to dynamically determine the tables and columns to process based on metadata while maintaining a reusable and scalable ingestion framework.
This would reduce the need for manual configuration changes whenever tables or columns need to be enabled, disabled, added, or excluded, and would make Copy Job much more suitable for enterprise-scale metadata-driven ingestion.
