topic Re: Ingesting .xml files from SharePoint library into Lakehouse in Dataflow

Ingesting .xml files from SharePoint library into Lakehouse

SuperFiets_ — Mon, 17 Jun 2024 20:20:08 GMT

Hi everyone,

I was looking for some guidance and some thoughts on best practises around using Fabric, specifically Lakehouses/Warehouses and ingesting data with Dataflow Gen2 or other options if you can recommend any.

I want to ingest .xml files stored in a SharePoint document library into the Lakehouse. The only way I find this is possible is using a Dataflow Gen2, getting the sharepoint library and opening the .xml files binary content with Xml.Tables. Then expanding all the needed columns and thus loading in all the tables that come forth of the .xml file into the Lakehouse.

Does anyone know of some other way to parse these .xml files (a lot) faster than using Dataflow Gen2? What are my options? Would using OneLake help me getting/loading the files faster, should I use PySpark to get the tables from the .xml or maybe even SQL (we are talking about hundreds of .xml files ranging from 1mb to 100mb to 5gb)?

Would love to get some extra eyes and experiences on this 🙂

Thanks!

Re: Ingesting .xml files from SharePoint library into Lakehouse

Anonymous — Tue, 18 Jun 2024 10:02:19 GMT

Hi SuperFiets_
Thanks for using Fabric Community.
You can refer to this link for using pipelines :
Microsoft Fabric - Ingest XML into Lakehouse - Dan Ambler

Hope this helps. Please let me know if you have any further questions.

Re: Ingesting .xml files from SharePoint library into Lakehouse

SuperFiets_ — Tue, 18 Jun 2024 13:46:43 GMT

Hi,

Thanks for your response!

How would I be able to implement the use case you specified into a pipeline that gets the data from SharePoint instead of an API? And more importantly would this be able to work for multiple .xml files?

Re: Ingesting .xml files from SharePoint library into Lakehouse

Anonymous — Thu, 20 Jun 2024 18:07:35 GMT

Hi SuperFiets_
You cannot use the above scenario for a sharepoint folder. Best option would be to use Dataflow Gen2.
Thanks

Re: Ingesting .xml files from SharePoint library into Lakehouse

SuperFiets_ — Tue, 25 Jun 2024 07:41:45 GMT

Thank you, in that case I will be using dataflows for now.