Forum Discussion
XML parsing fabric notebooks
How I personally would approach loading 8k files with inconsistent schemas;
- If I can't identify the schema from the filename/metadata, process each file separately.
- Convert each file to flat pyspark/pandas data frames - the schemas I've see so far are hierarchical which can be a bit painful to merge.
- Once everything has been flattened I'd then consider writing to either a single table or a single pyspark data frame making sure I allow the process to add new columns (easier when appending to a table, but not hard)
- I should now have a table with 1 row per entry, and null values where columns don't exist in one row but do in another.
- From this point onwards it's now a data processing/analysis problem 😊
Hi Amir_m_h
Thank you for reaching out microsoft fabric community forum.
- It looks like your XML structure has <abstract> nested inside <us-patent-application>, but Spark’s XML parser might not be handling that well, especially with large, combined XML files.
- The spark-xml library usually extracts attributes correctly, but when it comes to text inside deeply nested elements (like <p> inside <abstract>), it might miss the actual content if the structure isn’t processed properly.
- Also, if you’ve simply concatenated 8,000 XML documents into one big file without wrapping them in a proper root element, Spark might struggle to parse them. Ideally, each XML document should be treated as a separate row.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.
In my case I have included a root element as header and footer of the document but still it didn't solve the problem.
what is strange is that individual documents are parsed correctly but when put together this problem happens. So instead I also tried an approach to split the file to 8000 different xml files and then load them all in the data frame and the problem still persists.
Another thing I noticed is that abstract descriptions sometimes have <b> <i> tags for bold and italic. I tried to remove those and still the problem is there. I don't know what other option I need to try. (makes me think I prefer JSON over xml any day 🙂 )