Forum Discussion

TroelsHJ's avatar
TroelsHJ
New Member
2 years ago

Formatting AVRO during Eventstreaming from ConfluentCloud

Hi
 
I'm attempting to consume data from our Confluent Could Kafka environment, but I keep getting an error saying that the payloads are not formatted correctly according to the AVRO format.  
 
The exact message is this: 
Data preview "MasterDataStream_Test": ["Source 'EventHubInputAdapter' had 1 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2024-08-23T14:02:15.4723431Z' and '2024-08-23T14:02:15.4723431Z'. Invalid Avro Format, drop invalid record.","Source 'EventHubInputAdapter' had 1 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2024-08-23T14:02:15.4723431Z' and '2024-08-23T14:02:15.4723431Z'. Invalid Avro Format, drop invalid record.","Source 'EventHubInputAdapter' had 1 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2024-08-23T14:02:15.4723431Z' and '2024-08-23T14:02:15.4723431Z'. Invalid Avro Format, drop invalid record.","Source 'EventHubInputAdapter' had 1 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2024-08-23T14:02:15.4723431Z' and '2024-08-23T14:02:15.4723431Z'. Invalid Avro Format, drop invalid record."]
 
Looking at the data insight within the eventstream, it appears that my connection can consume data, but is not able to deserialize and display it. 
 
We're using the AVRO format when distributing data via Kafka and we have multiple other in-house services that can consume, deserialize, and work with the formatted data.
 
The following is an example of a message payload: 
 

 

[
    {
        "exceededFields": null,
        "headers": [
        ],
        "key": {
            "data": [
                0,
                0,
                0,
                0,
                0,
                13,
                23,
                106
            ],
            "type": "Buffer"
        },
        "offset": 392735,
        "partition": 1,
        "timestamp": 1706708606183,
        "timestampType": "CREATE_TIME",
        "value": {
            "created": {
                "long": 1706708606038
            },
            "created_date": {
                "long": 1702637923000
            },
            "is_deleted": {
                "boolean": false
            },
            "order_number": {
                "string": "ABC123"
            },
            "order_states_id": {
                "long": 30
            },
            "orders_id": {
                "long": 123123
            },
            "public_id": {
                "long": 123123
            },
            "source_data_version": {
                "long": 879127895786
            },
            "styles_public_id": {
                "long": 123123
            },
            "updated": {
                "long": 1706708606038
            },
            "vendor_name": {
                "string": "VENDOR_NAME"
            },
            "vendor_number": {
                "string": "VENDOR_NUMBER"
            }
        }
    }
]

 

 
So my concern is regarding the AVRO deserialization within the eventstream activity. 
Are we wrongful in our AVRO format or could there be some issue with the way the format is handled within the eventstream? 
 
Eventstream

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi TroelsHJ ,

     

    The error messages you've received suggest that the data is not in the expected AVRO format. Ensure that the AVRO schema used by the event stream is fully compatible with the schema used to serialize the data in Kafka.

    For more detail, please refer: Process event data with the event processor editor - Microsoft Fabric | Microsoft Learn

     

    Best Regards,

    Neeko Tang

    If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

    • TroelsHJ's avatar
      TroelsHJ
      New Member
      2 years ago

      Hi Anonymous 

       

      Thanks for the reply. 

       

      As per your suggestion, I'm trying to figure out exactly what schema the event stream is using when trying to deserialize the payload, but I can't seem to work it out. I find it very hard to debug in the event stream. 

       

      Can you possibly tell me where, within the event stream, I would find the schema it is using or perhaps tell me how I could debug the event stream? 

       

      Thanks

       

      Best, 

      Troels