Improving the error message when a data type conversion issue occurs writing onto a lakehouse
Hi,
for an evenstream that reads from an Azure Event Hubs to write the events onto a lakehouse I've encountered a data type conversion error and looking to the Run-time logs I've seen a such generic error message:
{
"ErrorCode": {
"value": "Warning",
"type": "string"
},
"TimeGenerated": {
"value": "2024-10-14 13:39:48Z",
"type": "datetime"
},
"Level": {
"value": "Warning",
"type": "string"
},
"Category": {
"value": "Warning",
"type": "string"
},
"ResourceId": {
"value": "",
"type": "string"
},
"SubscriptionId": {
"value": "",
"type": "string"
},
"ResourceGroup": {
"value": "",
"type": "string"
},
"JobName": {
"value": "",
"type": "string"
},
"OperationName": {
"value": "",
"type": "string"
},
"Status": {
"value": "Warning",
"type": "string"
},
"Source": {
"value": "",
"type": "string"
},
"TypeProperty": {
"value": "Warning",
"type": "string"
},
"DataErrorType": {
"value": "",
"type": "string"
},
"ErrorCategory": {
"value": "Warning",
"type": "string"
},
"BriefMessage": {
"value": "",
"type": "string"
},
"Message": {
"value": "First Occurred: 10/14/2024 1:39:46 PM UTC | Resource Name: 234dbbd1-4cb0-ab3b-7e66-a02a98fe0732 | Message: Source '234dbbd1-4cb0-ab3b-7e66-a02a98fe0732' had 1 occurrences of kind 'OutputDataConversionError.TypeConversionError' between processing times '2024-10-14T13:39:46.5109807Z' and '2024-10-14T13:39:46.7587399Z'. \r\n",
"type": "string"
},
"FromTimestamp": {
"value": null,
"type": "datetime"
},
"ToTimestamp": {
"value": null,
"type": "datetime"
},
"EventCount": {
"value": 0,
"type": "long"
},
"GroupSize": {
"value": 1,
"type": "long"
}
}
It could be very useful to have a major detail in order to detect the specific event data that causes the issue. Thanks
1 Comment
- fbcideas_migusrNew MemberStatus added:New
Recent ideas
Workspace Folder & Subfolder Inventory View for Power BI Reports and Apps
Problem: Many organisations use Power BI workspace folders and nested subfolders to organise reports by department, business area, project, or function. While this structure helps keep content organ...rijesh13 minutes agoFrequent VisitorNew2Views0likes0CommentsDeployment pipeline: Deployment rules for Direct Lake on OneLake semantic models
Currently, it is not possible to use deployment rules with Direct Lake on OneLake semantic models. The option is greyed out. Please enable this, so we can automatically change the data so...frithjof_v4 hours agoCommunity ChampionNew6KViews137likes17CommentsNative Master Data Management Solution
Microsoft Fabric currently lacks a native Master Data Management (MDM) capability, creating a gap for organizations that need to manage and govern business-critical reference data such as customers, ...bradcoles5 hours agoRegular VisitorNew213Views1like3Comments