Forum Discussion
Sudden Format change from JSON to TXT in Datastream
We created a data stream for EventHub using the "Get Data" functionality of Eventhouse. Initially, the data was coming in a multi-JSON format, but it suddenly changed to a TXT format we found this when we created new connection and tried to pull data. Previously, we were able to pull some system properties columns along with the original data. However, after the format change, even after selecting the system properties columns, we are unable to populate them.
We are pulling the same data from EventHub via Databricks, and it is working without any issues. However, in Fabric, our older data stream is unable to ingest the data which i am suspecting is due to format change.
Please help us resolve this issue or find the root cause. I am attaching screenshots for both scenarios.
New Data connection
Old Connection
Since Databricks is still ingesting the data correctly, the issue is likely within Fabric’s data ingestion or transformation process.
First, verify your Fabric data stream settings, especially the ingestion method and format configurations, as the initial multi-JSON format might have been altered due to a setting change. Check if the schema of the EventHub data has changed, if metadata or system properties are missing, Fabric might be defaulting to a TXT format instead of JSON.
Ensure that system properties are still selected in the Fabric data ingestion settings and try reconfiguring the ingestion to force their inclusion. Also, check if any recent Microsoft Fabric updates may have impacted how data streams handle JSON formats.
If nothing works, try recreating the data stream from scratch and explicitly setting the format to multi-JSON during configuration. Since the issue persists within Fabric but not in Databricks, it strongly suggests an internal setting or schema mismatch rather than a problem with EventHub.
4 Replies
TomOsResolver I
Since Databricks is still ingesting the data correctly, the issue is likely within Fabric’s data ingestion or transformation process.
First, verify your Fabric data stream settings, especially the ingestion method and format configurations, as the initial multi-JSON format might have been altered due to a setting change. Check if the schema of the EventHub data has changed, if metadata or system properties are missing, Fabric might be defaulting to a TXT format instead of JSON.
Ensure that system properties are still selected in the Fabric data ingestion settings and try reconfiguring the ingestion to force their inclusion. Also, check if any recent Microsoft Fabric updates may have impacted how data streams handle JSON formats.
If nothing works, try recreating the data stream from scratch and explicitly setting the format to multi-JSON during configuration. Since the issue persists within Fabric but not in Databricks, it strongly suggests an internal setting or schema mismatch rather than a problem with EventHub.
v-prasareCommunity Support
Hi Anonymous,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
TomOs , Thanks for your promt response
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
- v-prasare
Hi Anonymous,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
- v-prasare
Hi @SanchoPL,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query