Forum Discussion

Manuel_Grutsch's avatar
Manuel_Grutsch
New Member
2 years ago

Escape character vanishes on load. JSON field. Databricks connector

Hi everyone

 

We access a table containing a JSON column with the Power BI connector for SQL Warehouse cluster in Databricks.

 

The field looks like this in Databricks (complex datatype):

 

 

[{"key":"FOO","label":"FOOBAR","value":"ABC 54\""},{...}]

 

 

 

It also looks like that when we connect to the same SQL Warehouse cluster with Python.

 

However, in Power BI it looks like this (string). The escape character \ is missing:

 

 

[{"key":"FOO","label":"FOOBAR","value":"ABC 54""},{...}]

 

 

 

 

This leads to an error when we try to parse this JSON.

There are no other steps done in Power Query. That is just how the data comes in.

 

Could this be a bug? Some kind of parsing on the way?

 

Thanks for the help!

Kind regards

Manuel

Need Help

3 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Manuel_Grutsch ,

     

    After loading the data into Power BI, use the Power Query Editor to manually insert the missing escape characters. You can use the Replace Values feature to find all instances of 54"" and replace them with 54\"".

    Or before attempting to parse the JSON in Power BI, use a custom column in Power Query to validate and possibly correct the JSON string. This can be done using the M language to ensure that the JSON string is correctly formatted for parsing.

     

    Best Regards,

    Stephen Tao

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • Manuel_Grutsch's avatar
    Manuel_Grutsch
    New Member
    2 years ago

    Hi Anonymous

     

    Thanks for the reply.

     

    Your first suggestion sadly won't work, because the fields we are dealing with a custom fields in SAP. New ones can be added and the are barely any restriction for neither label or value. In other words, at any point there can be double quotes in the strings. Therefore we have no reliable way to do the replace value without messing with the required double quotes of the JSON structure.

     

    As for your second suggestion, how exactly would you go about validating and correcting the JSON string with a custom column and M?

     

    Thanks and beste regards

    Manuel