Escape character vanishes on load. JSON field. Databricks connector

[Manuel_Grutsch]

Hi everyone

We access a table containing a JSON column with the Power BI connector for SQL Warehouse cluster in Databricks.

The field looks like this in Databricks (complex datatype):

[{"key":"FOO","label":"FOOBAR","value":"ABC 54\""},{...}]

It also looks like that when we connect to the same SQL Warehouse cluster with Python.

However, in Power BI it looks like this (string). The escape character \ is missing:

[{"key":"FOO","label":"FOOBAR","value":"ABC 54""},{...}]

This leads to an error when we try to parse this JSON.

There are no other steps done in Power Query. That is just how the data comes in.

Could this be a bug? Some kind of parsing on the way?

Thanks for the help!

Kind regards

Manuel

[Manuel_Grutsch]

Hi @Anonymous

Thanks for the reply.

Your first suggestion sadly won't work, because the fields we are dealing with a custom fields in SAP. New ones can be added and the are barely any restriction for neither label or value. In other words, at any point there can be double quotes in the strings. Therefore we have no reliable way to do the replace value without messing with the required double quotes of the JSON structure.

As for your second suggestion, how exactly would you go about validating and correcting the JSON string with a custom column and M?

Thanks and beste regards

Manuel

[Anonymous]

Hi @Manuel_Grutsch ,

Maybe you could try using escape characters?

Here's the reference:

Chris Webb's BI Blog: Character Escape Sequences In M (crossjoin.co.uk)

Best Regards,

Stephen Tao

[Anonymous]

Hi @Manuel_Grutsch ,

After loading the data into Power BI, use the Power Query Editor to manually insert the missing escape characters. You can use the Replace Values feature to find all instances of 54"" and replace them with 54\"".

Or before attempting to parse the JSON in Power BI, use a custom column in Power Query to validate and possibly correct the JSON string. This can be done using the M language to ensure that the JSON string is correctly formatted for parsing.

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.