Forum Discussion
Advocate IStoring and parsing HL7 data
Any healthcare folks can share what best practices they have found around storing HL7 data for analytics ?
Hi ipkus,
I would start by separating two questions: whether the source is HL7 v2 or FHIR, and whether the ingestion is batch or real-time.
For HL7 v2, my usual pattern in Fabric would be to keep the original message unchanged in a raw/Bronze layer and then create a structured analytical representation downstream.
Something like:
Raw/Bronze
-> original HL7 message + source/message metadataStructured/Silver
-> parsed clinical entities in Delta tables, or a FHIR-based canonical representationAnalytics/Gold
-> domain-specific tables for encounters, orders, labs, admissions, etc. that are convenient for semantic models and Power BII would not use a single JSON column as the only long-term analytical representation. Keeping JSON/raw text is useful for replay and auditability, but repeatedly extracting every HL7 field from that representation becomes harder to govern and query as the number of message types grows.
Microsoft's current Healthcare data solutions architecture follows a similar medallion approach. Its unified storage design includes healthcare formats such as HL7 and FHIR, while the structured healthcare model in the Silver layer is based on FHIR.
If your source is HL7 v2 and you want FHIR as the canonical model, Azure Health Data Services also provides $convert-data, which supports HL7v2 -> FHIR R4 conversion. Microsoft treats that as one component of an ETL pipeline rather than the complete ingestion solution.
One practical point for HL7 v2 is that I would use an HL7-aware parser rather than relying only on splitting by |. Real messages also contain components, repetitions, escaping and message-specific segment structures that become difficult to manage with delimiter logic alone.
For batch workloads, files can land in OneLake and be parsed/transformed with pipelines/notebooks. For real-time workloads, I would keep the same Bronze/Silver/Gold separation but change the ingestion path and make sequencing/idempotency explicit where the clinical workflow depends on message order.
There is also a current product consideration: Microsoft is changing the delivery model for Healthcare data solutions in Fabric. A customer-managed source-code package is now available, and Microsoft is transitioning away from new deployments of the current managed solution, so I would take that roadmap into account for a new implementation.
If you can share whether the source is HL7 v2 or FHIR, and whether it is batch files or a live interface feed, the recommended implementation can be narrowed down quite a bit.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
5 Replies
- v-csrikanth
Community Support
Hi ipkus
We would like to inquire whether have you got the chance to check the solutions provided by other users in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community. - v-csrikanth
Hi ipkus
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Thank you.
- Kagiyama_yutaka
Skilled Sharer
Could you add a bit more detail about your setup? How you store HL7 for analytics can change a lot depending on whether you're working with HL7 v2 or FHIR, and whether your workflow is real‑time or batch.
nathancwatkinsHelper II
Bring in all fields by parsing with |, grab and fill down each Message ID and segment code, make the parsed values a list type, then into JSON, so you have a column containing the values as JSON text for that message ID and segment. The values compress efficiently, and there’s your base ingestion table. Now your ETL query can easily extract values and zip them to a list of field names.
—Nate
- ShivekMaharaj
Community Champion
Hi ipkus,
I would start by separating two questions: whether the source is HL7 v2 or FHIR, and whether the ingestion is batch or real-time.
For HL7 v2, my usual pattern in Fabric would be to keep the original message unchanged in a raw/Bronze layer and then create a structured analytical representation downstream.
Something like:
Raw/Bronze
-> original HL7 message + source/message metadataStructured/Silver
-> parsed clinical entities in Delta tables, or a FHIR-based canonical representationAnalytics/Gold
-> domain-specific tables for encounters, orders, labs, admissions, etc. that are convenient for semantic models and Power BII would not use a single JSON column as the only long-term analytical representation. Keeping JSON/raw text is useful for replay and auditability, but repeatedly extracting every HL7 field from that representation becomes harder to govern and query as the number of message types grows.
Microsoft's current Healthcare data solutions architecture follows a similar medallion approach. Its unified storage design includes healthcare formats such as HL7 and FHIR, while the structured healthcare model in the Silver layer is based on FHIR.
If your source is HL7 v2 and you want FHIR as the canonical model, Azure Health Data Services also provides $convert-data, which supports HL7v2 -> FHIR R4 conversion. Microsoft treats that as one component of an ETL pipeline rather than the complete ingestion solution.
One practical point for HL7 v2 is that I would use an HL7-aware parser rather than relying only on splitting by |. Real messages also contain components, repetitions, escaping and message-specific segment structures that become difficult to manage with delimiter logic alone.
For batch workloads, files can land in OneLake and be parsed/transformed with pipelines/notebooks. For real-time workloads, I would keep the same Bronze/Silver/Gold separation but change the ingestion path and make sequencing/idempotency explicit where the clinical workflow depends on message order.
There is also a current product consideration: Microsoft is changing the delivery model for Healthcare data solutions in Fabric. A customer-managed source-code package is now available, and Microsoft is transitioning away from new deployments of the current managed solution, so I would take that roadmap into account for a new implementation.
If you can share whether the source is HL7 v2 or FHIR, and whether it is batch files or a live interface feed, the recommended implementation can be narrowed down quite a bit.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
