Forum Discussion

Joshrodgers123's avatar
Joshrodgers123
Icon for Kudo Collector rankKudo Collector
2 years ago

Copy Activity - On-Prem to Warehouse - Staging

Trying to copy data from on-prem SQL Server to a Warehouse. It says that staging is required, but only allows you to use ADLS or Blob storage - not OneLake.

 

Is this really the expected functionality? Do we need to bring additional Azure resources to get data from on-prem to a Warehouse?

 

 

pipelines

13 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi Joshrodgers123 

     

    Thanks for using Microsoft Fabric Community.

    Apologies for the inconvenience.

    Data Warehouse COPY command directly supports Azure Blob Storage and Azure Data Lake Storage Gen2 as source data stores. If your source data meets the criteria described in this section, use COPY command to copy directly from the source data store to Data Warehouse.

    For more details please refer : Direct copy by using COPY command.

                                                   : Create a table.

    If the issue still persists, please do let us know. Glad to help.

     

    I hope this information helps.

     

    Thanks.

    • Joshrodgers123's avatar
      Joshrodgers123
      Icon for Kudo Collector rankKudo Collector
      2 years ago

      My question was referring to the copy activity in data pipelines (SQL server source/warehouse destination), not the copy command in the warehouse. 

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Icon for Microsoft Employee rankMicrosoft Employee
        2 years ago

        Hi Joshrodgers123 

         

        Apologies for the delay in response from my end.

        In order to understand your specific data copy scenario, Could you please share some more details about your copy data activity. This will help me to understand the query and guide in better way. If possible, could you provide information on the following:

         

        Data Transformation Specifications: Are there any transformations being applied to the data during the copy process?

        Screenshots: If feasible, sharing screenshots of the source and sink configuration within your data copy activity would be highly beneficial for visualization purposes.

         

        Thanks.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi Joshrodgers123 

       

      Glad that you query got resolved and matkvaid thank you for sharing the same with the community as it can be helpful to others.
      Please continue using Fabric Community for further queries.

       

      Thank you.

       

  • dbrowne's avatar
    dbrowne
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Supporting OneLake as staging for on-premise to DW is on the roadmap.  In the mean time, it's better to simply load to a Lakehouse in the same workspace as the Warehouse.  Those lakehouse tables will be accessable from the SQL endpoint, and you can load from there to the Warehouse if you want to.

     

    • matkvaid's avatar
      matkvaid
      Icon for Helper III rankHelper III
      2 years ago

      I was waiting for on premise connection support for pipelines, because dataflows gen2 had many issues and with them resolved, they still use much more CU's than pipelines (at least copying from warehouse to lakehouse test promises). Now setting up lakehouse and then warehouse again would need adittional resources. Now i understand i should move to lakehouse and rebuild all reports, views, etc. Lakehouse tables does not support spaces in column names, o it is one of the things that is a nightmare to set up copy activities when renaming hundreds of columns...