Forum Discussion

davidz106's avatar
davidz106
Helper III
2 years ago

Building flat file pipeline

Hi everyone, I've recently come across Microsoft Fabric and its connected services, and I'm intrigued by the functionality where you can simply drop files in an explorer interface, and they beco...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi davidz106 ,

Thanks for using Fabric Community.

In order to guide you better I have few queries,
1. Can you please share some dummy data or sample files ( 2 or 3 )
2. How you are expecting data to load in SQL DB? I would like to understand the sink table structure.
3. How big is the data?
4. Can you please share challenges in your current python pipelines implementation?
5. Are you looking for real time triggers? Like whenever we upload a file in One lake File explorer it starts processing the file and loads to SQL DB.

Please help me sharing your response, so that I can guide you better.


Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi davidz106 ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to look into above reply. 

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi davidz106 ,

    We haven’t heard from you on the last response and was just checking back to see if you got a chance to look into above reply.