Hi.
We are trying to use the CopyJob to do Merge on some table in order to do incremental loads to a Lakehouse in Fabric from an onpremises SQL Server. Some of the tables we use are not very well constructed so we sometimes have to get rid of some columns in the source for which we use a Query instead of bringing the whole table which sometimes can have more than 100 columns (again not very well constructed table).
On the Copy Activity we have the ability so send a Query or ingest a whole table with all the columns. We would like to see this option available for the CopyJob if at all possible.
Thanks!
4 Comments
- yexuMicrosoft EmployeeStatus changed:NewtoNeeds Votes
- Miwa1Microsoft EmployeeQuery is already supported for the following connectors and we're expanding it to more. Lakehouse AzureSql AzureSqlMI FabricSql SqlServer AzureSQLDatawarehouse Snowflake Google BigQuery Oracle Kusto
- AntonioPinedaRegular VisitorYES! We noticed it is added now. Thanks a lot!
- yexuMicrosoft EmployeeStatus changed:Needs VotestoCompleted
Use query as source has been added in Copy job. See details in Truncate Destination, Queries and Multiple Folders – Simplifying Data Ingestion with Copy Job | Microsoft Fabric Blog | Microsoft Fabric
Recent ideas
Allow External OneLake Data Shares to Be Accepted into Multiple Fabric Item Types
Fabric External Data Sharing provides an excellent zero-copy mechanism for sharing OneLake data across tenants. However, the receiving tenant currently has to accept the share into a Lakehouse. I w...Mourak22 hours agoNew MemberNew5Views0likes0CommentsAllow OneLake Shortcut Authentication Mode to Be Changed In Place
OneLake shortcuts support passthrough and delegated authentication, but changing the authentication model currently requires deleting and recreating the shortcut. I would like shortcut authenticatio...Mourak22 hours agoNew MemberNew7Views0likes0CommentsTenant-Wide OneLake Shortcut Lineage and Impact Analysis
This is grounded in a documented limitation: shortcut lineage is currently workspace-scoped, and Microsoft specifically says lineage for shortcuts to warehouses and semantic models isn't currently av...Mourak22 hours agoNew MemberNew16Views0likes1CommentAdd an End-to-End Production Reference Architecture for Microsoft Fabric Real-Time Intelligence
I would like to see a complete end-to-end Microsoft Fabric Real-Time Intelligence reference architecture that demonstrates how the individual RTI capabilities work together in a production-oriented s...Mourak22 hours agoNew MemberNew5Views0likes0CommentsImprove CI/CD consistency for Microsoft Fabric Real-Time Intelligence Eventstreams
Microsoft Fabric Real-Time Intelligence has made strong progress with Git integration and deployment pipelines, but lifecycle management can still be inconsistent across Eventstream sources, destinat...Mourak23 hours agoNew MemberNew3Views0likes0Comments