Forum Discussion

bf49152's avatar
bf49152
Frequent Visitor
8 months ago
Solved

CopyJob for Incremental Copy from Fabric Lakehouse to SQL Server

Hello, 

 

I am trying to configure a CopyJob to perform an incremental copy from a Fabric Lakehouse to an on-premises SQL Server. The Lakehouse table has change data capture enabled. The on-prem connection to SQL Server has been tested.

With the CopyJob, I'm the error message: 

ErrorCode=UserErrorInvalidPluginType,'Type=Microsoft.DataTransfer.Common.Shared.PluginNotRegisteredException,Message=Invalid type 'LakehouseTable' is provided in 'linkedServices'. Please correct the type in payload and retry.,Source=Microsoft.DataTransfer.ClientLibrary,' 

 

A CopyActivity in a Fabric Pipeline works, but it's not an ideal solution for me.

data engineering
lakehouse
  • Anonymous's avatar
    Anonymous
    8 months ago

    Hi bf49152 ,

    CopyJob does not currently support LakehouseTable as a valid linked service type. At present, the only supported way to copy data from a Fabric Lakehouse to SQL Server is through a Copy Activity in a Fabric Pipeline, not via CopyJob.

     

    Try below alternative workarounds.

     

    1. Use Fabric Pipeline Copy Activity 

     

    Pipelines fully support Lakehouse as a source.

     

    You can configure incremental copy using CDC-enabled tables.

     

    This is the officially documented approach

    https://learn.microsoft.com/en-us/fabric/data-factory/connector-lakehouse-copy-activity

     

    2. Intermediate Staging Approach:

     

    Export Lakehouse data to a supported format (e.g., Parquet/CSV in OneLake or ADLS).

     

    Configure CopyJob to move from that staging location into SQL Server.

     

    This adds an extra step but avoids pipeline dependency.

     

    3. Hybrid Strategy:

     

    Use Pipeline for Lakehouse → Staging (incremental CDC).

     

    Use CopyJob for Staging → SQL Server if you want to keep CopyJob in play for downstream automation.

     

    If my response as resolved your issue please mark it as solution and give kudos.

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    8 months ago

    Hi bf49152 ,

    CopyJob does not currently support LakehouseTable as a valid linked service type. At present, the only supported way to copy data from a Fabric Lakehouse to SQL Server is through a Copy Activity in a Fabric Pipeline, not via CopyJob.

     

    Try below alternative workarounds.

     

    1. Use Fabric Pipeline Copy Activity 

     

    Pipelines fully support Lakehouse as a source.

     

    You can configure incremental copy using CDC-enabled tables.

     

    This is the officially documented approach

    https://learn.microsoft.com/en-us/fabric/data-factory/connector-lakehouse-copy-activity

     

    2. Intermediate Staging Approach:

     

    Export Lakehouse data to a supported format (e.g., Parquet/CSV in OneLake or ADLS).

     

    Configure CopyJob to move from that staging location into SQL Server.

     

    This adds an extra step but avoids pipeline dependency.

     

    3. Hybrid Strategy:

     

    Use Pipeline for Lakehouse → Staging (incremental CDC).

     

    Use CopyJob for Staging → SQL Server if you want to keep CopyJob in play for downstream automation.

     

    If my response as resolved your issue please mark it as solution and give kudos.

    • bf49152's avatar
      bf49152
      Frequent Visitor
      8 months ago

      Hi Anonymous ,

       

      What you're saying contradicts Microsoft's recommendation to use CopyJob to move data from Fabric Lakehouse to on-premises SQL Server. The blog article below was recently published and addresses my use case.

      Simplifying Data Ingestion with Copy job – Replicate data from Dataverse through Fabric to multiple destinations | Microsoft Fabric Blog | Microsoft Fabric


      The link below shows that the connectors I am using are supported. 


      What is Copy job in Data Factory - Microsoft Fabric | Microsoft Learn

       

      I'm not sure what I'm doing wrong. There is not much to setting up a CopyJob. 

      • v-menakakota's avatar
        v-menakakota
        Community Support
        8 months ago

        Hi bf49152 ,

        Thanks for reaching out to the Microsoft fabric community forum.

         

        Thanks for the update. The blog does show that CopyJob supports Lakehouse as a connector, but CopyJob still does not support LakehouseTable as a direct linked service type. That’s why the job fails with the “Invalid type 'LakehouseTable'” error.

         

        Pipelines support LakehouseTable fully, which is why your Pipeline version works. CopyJob can only use Lakehouse through file-based sources (like Parquet/CSV), not the table object itself.

         

        So your setup is correct it’s just a current limitation of CopyJob. Until Microsoft updates this capability, the only working options are using Pipeline Copy Activity or using a staging folder with CopyJob.

         

        Best Regards, 
        Community Support Team