Forum Discussion

SuperFiets_'s avatar
SuperFiets_
Helper I
2 years ago
Solved

Ingesting .xml files from SharePoint library into Lakehouse

Hi everyone,


I was looking for some guidance and some thoughts on best practises around using Fabric, specifically Lakehouses/Warehouses and ingesting data with Dataflow Gen2 or other options if you can recommend any.

I want to ingest .xml files stored in a SharePoint document library into the Lakehouse. The only way I find this is possible is using a Dataflow Gen2, getting the sharepoint library and opening the .xml files binary content with Xml.Tables. Then expanding all the needed columns and thus loading in all the tables that come forth of the .xml file into the Lakehouse.

 

Does anyone know of some other way to parse these .xml files (a lot) faster than using Dataflow Gen2? What are my options? Would using OneLake help me getting/loading the files faster, should I use PySpark to get the tables from the .xml or maybe even SQL (we are talking about hundreds of .xml files ranging from 1mb to 100mb to 5gb)?

 

Would love to get some extra eyes and experiences on this 🙂

 

Thanks!

Dataflows
Need Help
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi SuperFiets_ 
    You cannot use the above scenario for a sharepoint folder. Best option would be to use Dataflow Gen2.
    Thanks

4 Replies

    • SuperFiets_'s avatar
      SuperFiets_
      Helper I
      2 years ago

      Hi,

       

      Thanks for your response!

       

      How would I be able to implement the use case you specified into a pipeline that gets the data from SharePoint instead of an API? And more importantly would this be able to work for multiple .xml files?

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi SuperFiets_ 
        You cannot use the above scenario for a sharepoint folder. Best option would be to use Dataflow Gen2.
        Thanks