Forum Discussion

sudhav's avatar
sudhav
Icon for Helper V rankHelper V
3 years ago

ingesting data from on premise sql server to Fabric Via Notebook

Hi Team,  i need to ingest the data from on premise sql server to Fabric. i am able to do this using Dataflows, but my requirement is here to use Notebook.  is there any way or code to do that. ple...
data engineering
Data Integration
DennesTorres's avatar
DennesTorres
Icon for Power Participant rankPower Participant
3 years ago

Hi,

I don't think the notebooks on Fabric can access data through the data gateway. The pipelines can't also, only the dataflows can.

I'm curious: Wouldn't it be enough to bring the data to the Fabric environment using a dataflow and later transform the data using a notebook?

Kind Regards,

Dennes

sudhav's avatar
sudhav
Icon for Helper V rankHelper V
3 years ago

Hi, I agree with you dataflows are enough, but those are very slow and my customer asking me to do it in another way because huge volume of the dta in on prem. so dataflows are getting failed oftenely while publishing and also getting performance issues.

  • DennesTorres's avatar
    DennesTorres
    Icon for Power Participant rankPower Participant
    3 years ago

    Hi,

    I don't think the dataflows are the ones to be blamed here, although this is a very specific experience and it would be interesting to hear from someone from Microsoft.

    Dataflows are linked to the Fabric capacity as the notebooks, the performance should not have this huge difference, it should be only a matter of choice - I believe so.

     

    Anyway, the bottleneck on this case doesn't seems to be the dataflow execution, but the transfer through data gateway, which notebooks don't support yet.

    First, I would suggest to analyse the volume of data: would it be possible to make incremental load to reduce it?

    Second, I would consider alternate ways. What if you make some automation on premises, may be SSIS, to generate files and upload these files to the "Files" area of a lakehouse, than start in the cloud from this point?

    At the moment you may be tied by the lack of notebook support to the data gateway. In the future, you may need a support ticket to ask about the best option and performance issues.

     

    Kind Regards,

     

    Dennes

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Thanks for your explaination here Dennes.  This issue bother me as well for these days.

      Now I have to thinking use dataflow to ingest source data instead of Notebook since it does not support data gateway.

      Would like to add one point :  you know sometimes the source data tables could be hundreds or thousands and I need to manually config the destination one by one in dataflow as below. 

       

      In addtion, notebook will be more easier to mainten if anything need to be changed, like datasource connection string change., data base tables change, etc.