Forum Discussion
Getting json payload from Azure Event Grid / Event Hubs and just save it to the Lakehouse
- 7 months ago
Hi foks,
Thank you for reaching out to the Microsoft fabric community forum.When event data from Event Grid / Event Hub is ingested into a Lakehouse using streaming paths, Fabric automatically parses the JSON and stores it as a Delta table for analytics. There is not a supported option today to tell Fabric to keep the incoming payload as a raw JSON file in the Lakehouse Files area.
If your requirement is to preserve the events exactly as received raw JSON, you will need to land them outside the default streaming-to-Lakehouse experience.
For example capture the events using a custom ingestion Azure Function, Logic App, or Data Factory and write them directly as JSON files to storage/Lakehouse Files. If the goal is analytics on event or telemetry data, Eventhouse (KQL) is the recommended approach, as it’s designed for this type of time-series data.
Kindly refer to the below documentation links for better understanding:
Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft LearnHope this clarifies why the data is getting flattened and what the available options are today.
Regards,
Community Support Team.
Hi foks,
Thank you for reaching out to the Microsoft fabric community forum.
When event data from Event Grid / Event Hub is ingested into a Lakehouse using streaming paths, Fabric automatically parses the JSON and stores it as a Delta table for analytics. There is not a supported option today to tell Fabric to keep the incoming payload as a raw JSON file in the Lakehouse Files area.
If your requirement is to preserve the events exactly as received raw JSON, you will need to land them outside the default streaming-to-Lakehouse experience.
For example capture the events using a custom ingestion Azure Function, Logic App, or Data Factory and write them directly as JSON files to storage/Lakehouse Files. If the goal is analytics on event or telemetry data, Eventhouse (KQL) is the recommended approach, as it’s designed for this type of time-series data.
Kindly refer to the below documentation links for better understanding:
Add a lakehouse destination to an eventstream - Microsoft Fabric | Microsoft Learn
Hope this clarifies why the data is getting flattened and what the available options are today.
Regards,
Community Support Team.
- v-hjannapu7 months agoCommunity Support
Hi foks,
I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.
Regards,
Community Support Team.- v-hjannapu7 months agoCommunity Support
Hi foks,
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We are always here to support you.
Regards,
Community Support Team.