Forum Discussion
Solved
Allow manual schema edits without live samples.
Issue The schema editor depends on live samples, which prevents mapping for rare events. Example Patricia designs medical devices. Her device only outputs certain fields when a medical condition i...
Hi uriel-kluk,
Great ask - you’ve hit a real limitation today.
What’s possible right now
- Pre-declare optional fields downstream with "Manage fields".
In the Event Processor, add a Manage fields step and use Add field to create placeholders (e.g., set them to null or a constant). This publishes those fields in the output schema even if they aren’t present in live samples yet. See “Manage fields” in the Event Processor editor (docs) and tutorial steps that show adding a Manage fields tile (tutorial). - If you control a simulator, use a Custom App source + Schema Registry.
You can upload a JSON schema to the Schema Registry and attach it when using Custom app/endpoint sources — which lets you define a schema up front without waiting for a live message. Today this is supported for Custom app/endpoint and Azure SQL CDC sources (not IoT Hub), and it must be enabled when you create the eventstream (docs, add Custom app). - Know how destinations behave.
If you’re landing in a Lakehouse table, extra incoming columns are dropped and missing table columns get written as null. Pre-creating the table with your optional columns avoids data loss for rare fields (docs).
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this post as the solution.
- Pre-declare optional fields downstream with "Manage fields".
Hi uriel-kluk,
Great ask - you’ve hit a real limitation today.
What’s possible right now
- Pre-declare optional fields downstream with "Manage fields".
In the Event Processor, add a Manage fields step and use Add field to create placeholders (e.g., set them to null or a constant). This publishes those fields in the output schema even if they aren’t present in live samples yet. See “Manage fields” in the Event Processor editor (docs) and tutorial steps that show adding a Manage fields tile (tutorial). - If you control a simulator, use a Custom App source + Schema Registry.
You can upload a JSON schema to the Schema Registry and attach it when using Custom app/endpoint sources — which lets you define a schema up front without waiting for a live message. Today this is supported for Custom app/endpoint and Azure SQL CDC sources (not IoT Hub), and it must be enabled when you create the eventstream (docs, add Custom app). - Know how destinations behave.
If you’re landing in a Lakehouse table, extra incoming columns are dropped and missing table columns get written as null. Pre-creating the table with your optional columns avoids data loss for rare fields (docs).
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, please mark this post as the solution.