Forum Discussion
Solved
XML parsing fabric notebooks
Hi, TLDR: I have trouble in parsing an xml file in fabric lakehouse. Spark xml parser somehow misses _value of one of the tags. The xml file related to abstract looks like below and I have tro...
How I personally would approach loading 8k files with inconsistent schemas;
- If I can't identify the schema from the filename/metadata, process each file separately.
- Convert each file to flat pyspark/pandas data frames - the schemas I've see so far are hierarchical which can be a bit painful to merge.
- Once everything has been flattened I'd then consider writing to either a single table or a single pyspark data frame making sure I allow the process to add new columns (easier when appending to a table, but not hard)
- I should now have a table with 1 row per entry, and null values where columns don't exist in one row but do in another.
- From this point onwards it's now a data processing/analysis problem 😊
Hi Amir_m_h
Thank you for reaching out microsoft fabric community forum.
- It looks like your XML structure has <abstract> nested inside <us-patent-application>, but Spark’s XML parser might not be handling that well, especially with large, combined XML files.
- The spark-xml library usually extracts attributes correctly, but when it comes to text inside deeply nested elements (like <p> inside <abstract>), it might miss the actual content if the structure isn’t processed properly.
- Also, if you’ve simply concatenated 8,000 XML documents into one big file without wrapping them in a proper root element, Spark might struggle to parse them. Ideally, each XML document should be treated as a separate row.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.