Forum Discussion
Allow manual schema edits without live samples.
Issue
The schema editor depends on live samples, which prevents mapping for rare events.
Example
Patricia designs medical devices. Her device only outputs certain fields when a medical condition is present. She should not have to wait or force it just to define schema fields.
Expected Result
Manual schema editing works without a live sample.
Optional: provide a textbox to enter simulated data for the schema.
Allow adding new fields at any time for future records.
Actual Result
Schema editor requires a live record to add fields.
Impact
Rare events cannot be mapped on demand.
Schema setup is incomplete until the event occurs live.
Repro Steps
Add one IoT Hub source in Eventstream.
Send steady telemetry
Open schema editor.
Observe all event types in one stream.
Try to add new fields without a live sample.
Acceptance Criteria
Given no live messages, schema editor still accepts new fields.
Future records populate those fields when they arrive.
Hi uriel-kluk,
Great ask - you’ve hit a real limitation today.
What’s possible right now
- Pre-declare optional fields downstream with "Manage fields".
In the Event Processor, add a Manage fields step and use Add field to create placeholders (e.g., set them to null or a constant). This publishes those fields in the output schema even if they aren’t present in live samples yet. See “Manage fields” in the Event Processor editor (docs) and tutorial steps that show adding a Manage fields tile (tutorial). - If you control a simulator, use a Custom App source + Schema Registry.
You can upload a JSON schema to the Schema Registry and attach it when using Custom app/endpoint sources — which lets you define a schema up front without waiting for a live message. Today this is supported for Custom app/endpoint and Azure SQL CDC sources (not IoT Hub), and it must be enabled when you create the eventstream (docs, add Custom app). - Know how destinations behave.
If you’re landing in a Lakehouse table, extra incoming columns are dropped and missing table columns get written as null. Pre-creating the table with your optional columns avoids data loss for rare fields (docs).
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this post as the solution.
- Pre-declare optional fields downstream with "Manage fields".
2 Replies
Hi uriel-kluk,
Great ask - you’ve hit a real limitation today.
What’s possible right now
- Pre-declare optional fields downstream with "Manage fields".
In the Event Processor, add a Manage fields step and use Add field to create placeholders (e.g., set them to null or a constant). This publishes those fields in the output schema even if they aren’t present in live samples yet. See “Manage fields” in the Event Processor editor (docs) and tutorial steps that show adding a Manage fields tile (tutorial). - If you control a simulator, use a Custom App source + Schema Registry.
You can upload a JSON schema to the Schema Registry and attach it when using Custom app/endpoint sources — which lets you define a schema up front without waiting for a live message. Today this is supported for Custom app/endpoint and Azure SQL CDC sources (not IoT Hub), and it must be enabled when you create the eventstream (docs, add Custom app). - Know how destinations behave.
If you’re landing in a Lakehouse table, extra incoming columns are dropped and missing table columns get written as null. Pre-creating the table with your optional columns avoids data loss for rare fields (docs).
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this post as the solution.
- Pre-declare optional fields downstream with "Manage fields".
Hi uriel-kluk,
We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.
tayloramy ,Thanks for your prompt responseThank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support
