Forum Discussion
Building flat file pipeline
Hi davidz106 ,
Thanks for using Fabric Community.
In order to guide you better I have few queries,
1. Can you please share some dummy data or sample files ( 2 or 3 )
2. How you are expecting data to load in SQL DB? I would like to understand the sink table structure.
3. How big is the data?
4. Can you please share challenges in your current python pipelines implementation?
5. Are you looking for real time triggers? Like whenever we upload a file in One lake File explorer it starts processing the file and loads to SQL DB.
Please help me sharing your response, so that I can guide you better.
Hi davidz106 ,
We haven’t heard from you on the last response and was just checking back to see if you got a chance to look into above reply.Hi davidz106 ,
We haven’t heard from you on the last response and was just checking back to see if you got a chance to look into above reply.
Hi v-gchenna-msft,
1. and 3. Sample data could be a simple 20x20 table with one empty row. By 'parsing' I mean operation like skipping empty rows in df or such.
2. We have thousands of files so uploading them directly into PBI is not an option, therefore a data pipeline is needed. Currently parsing functions are already written in python but we could use other tool if needed.
4. Solutions are ok but a more robust approach is wanted, preferably inside MS enviroment since we would then use this database as input to PBI.5. This is the gist of my question and exactly what I or we are aiming for, you put it nicely: we upload a file in One lake File explorer it starts processing the file and loads to SQL DB.
A tutorial on that would be very welcome.
