Forum Discussion

prabhatnath's avatar
prabhatnath
Advocate III
1 year ago
Solved

Need help to parse JSON string that has escape chars

Hi Friends, I am working on a PySpark notebook in Fabric that has below code fragment and the initialized value for log_data is  good working example where the code gives outptu as: JSON is valid! ...
workspaces
  • v-pnaroju-msft's avatar
    v-pnaroju-msft
    1 year ago

    Hi prabhatnath,

    Thank you for reaching out to the Microsoft Fabric Community Forum.

    Please find attached the screenshot and the relevant log_data, which may assist in resolving the issue:


    log_data = "[{\"ApplicationName\": \"Ap001\",\"WorkspaceId\": \"af00f007-7e26-4654-803c-d82f11108b79\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_People\",\"Operation\": \"ETL\",\"Run_Id\": \"f5f780ba-b5b7-452c-80e7-77b05d898799\",\"SessionId\": \"\",\"Message\": \"[DEV] - Ap001 - Pipeline Execution Failed - 20250313\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_People\\nRun Id: f5f780ba-b5b7-452c-80e7-77b05d898799\\nError Message: Notebook Error Message: An error occurred while calling o4570.load.\n: Invalid URI The ABFS endpoint for host: mydomain.dfs.fabric.microsoft.com1 is not supported. It should match one of the valid configured endpoints [fabric.microsoft.com, data.microsoft.com, pbidedicated.windows.net, core.windows.net]\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.validateHostnameEndpointsIfRequired(AzureBlobFileSystemStore.java:503)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getURIBuilder(AzureBlobFileSystemStore.java:455)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1649)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:258)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:192)\n\tat com.microsoft.vegas.vfs.VegasFileSystem.initialize(VegasFileSystem.java:133)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)\n\tat org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)\n\tat org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:178)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource$.parsePathIdentifier(DeltaDataSource.scala:357)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.x$1$lzycompute(DeltaTableV2.scala:73)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.x$1(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelByPath$lzycompute(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelByPath(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.$anonfun$timeTravelSpec$1(DeltaTableV2.scala:122)\n\tat scala.Option.orElse(Option.scala:447)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelSpec$lzycompute(DeltaTableV2.scala:122)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelSpec(DeltaTableV2.scala:118)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.snapshot$lzycompute(DeltaTableV2.scala:126)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.snapshot(DeltaTableV2.scala:125)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.toBaseRelation(DeltaTableV2.scala:200)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.$anonfun$createRelation$5(DeltaDataSource.scala:230)\n\tat org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:141)\n\tat org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:139)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.recordFrameProfile(DeltaDataSource.scala:49)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:188)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:346)\n\tat org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:236)\n\tat org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:219)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:219)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/af00f007-7e26-4654-803c-d82f11108b79/pipelines/PL_People/f5f780ba-b5b7-452c-80e7-77b05d898799?experience=power-bi\\nApp Name: Ap001\",\"CorrelationId\": \"\",\"User\": \"System\"}]"  

    If you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will help other community members encountering similar queries.

    Thank you.

v-pnaroju-msft's avatar
v-pnaroju-msft
Community Support
1 year ago

Hi prabhatnath,

Thank you for reaching out to the Microsoft Fabric Community Forum.

Please find attached the screenshot and the relevant log_data, which may assist in resolving the issue:


log_data = "[{\"ApplicationName\": \"Ap001\",\"WorkspaceId\": \"af00f007-7e26-4654-803c-d82f11108b79\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_People\",\"Operation\": \"ETL\",\"Run_Id\": \"f5f780ba-b5b7-452c-80e7-77b05d898799\",\"SessionId\": \"\",\"Message\": \"[DEV] - Ap001 - Pipeline Execution Failed - 20250313\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_People\\nRun Id: f5f780ba-b5b7-452c-80e7-77b05d898799\\nError Message: Notebook Error Message: An error occurred while calling o4570.load.\n: Invalid URI The ABFS endpoint for host: mydomain.dfs.fabric.microsoft.com1 is not supported. It should match one of the valid configured endpoints [fabric.microsoft.com, data.microsoft.com, pbidedicated.windows.net, core.windows.net]\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.validateHostnameEndpointsIfRequired(AzureBlobFileSystemStore.java:503)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getURIBuilder(AzureBlobFileSystemStore.java:455)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1649)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:258)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:192)\n\tat com.microsoft.vegas.vfs.VegasFileSystem.initialize(VegasFileSystem.java:133)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)\n\tat org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)\n\tat org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)\n\tat org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:178)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource$.parsePathIdentifier(DeltaDataSource.scala:357)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.x$1$lzycompute(DeltaTableV2.scala:73)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.x$1(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelByPath$lzycompute(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelByPath(DeltaTableV2.scala:68)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.$anonfun$timeTravelSpec$1(DeltaTableV2.scala:122)\n\tat scala.Option.orElse(Option.scala:447)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelSpec$lzycompute(DeltaTableV2.scala:122)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.timeTravelSpec(DeltaTableV2.scala:118)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.snapshot$lzycompute(DeltaTableV2.scala:126)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.snapshot(DeltaTableV2.scala:125)\n\tat org.apache.spark.sql.delta.catalog.DeltaTableV2.toBaseRelation(DeltaTableV2.scala:200)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.$anonfun$createRelation$5(DeltaDataSource.scala:230)\n\tat org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:141)\n\tat org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:139)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.recordFrameProfile(DeltaDataSource.scala:49)\n\tat org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:188)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:346)\n\tat org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:236)\n\tat org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:219)\n\tat scala.Option.getOrElse(Option.scala:189)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:219)\n\tat org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/af00f007-7e26-4654-803c-d82f11108b79/pipelines/PL_People/f5f780ba-b5b7-452c-80e7-77b05d898799?experience=power-bi\\nApp Name: Ap001\",\"CorrelationId\": \"\",\"User\": \"System\"}]"  

If you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will help other community members encountering similar queries.

Thank you.

prabhatnath's avatar
prabhatnath
Advocate III
1 year ago

Hi Friend,

Below is another example that is failing as the Details section has text enclosed with " ("Internal Server Error") and I am not sure how do I parse these unescaped " chars insode the message. I cant directly do replace for entire file as the element names need to be enclosed with ".

Please suggest how can I parse these types of messages that are having text enclosed with ".

 

var_str_log_data = "[{\"ApplicationName\": \"APP\",\"WorkspaceId\": \"0addb382-fa0d-4ce1-9c3c-95e32b957955\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_APP_Ingest\",\"Operation\": \"ETL\",\"Run_Id\": \"88276e4f-4e60-47ec-a6d6-dd9e475d2c3a\",\"SessionId\": \"\",\"Message\": \"[DEV] - APP - Pipeline Execution Failed - 20250329\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_APP_Ingest\\nRun Id: 88257e4f-4e60-34ec-a6d6-dd9e475d2c3a\\nError Message: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o4628.load.\n: java.util.concurrent.ExecutionException: Operation failed: \"Internal Server Error\", 500, HEAD, http://msit-onelake.dfs.fabric.microsoft.com/MYWORKSPACE/dataverse_name_cds2_workspace_unq60feaf930cbf236eb2519b27f2d6b.Lakehouse/Tables/app_riskparcel/_delta_log?upn=false&action=getStatus&timeout=90\n	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)\n	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)\n	... 37 more\n' : \\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/0addb122-fa0d-4ce1-9c3c-95e25b957955/pipelines/PL_APP_Ingest/88223e4f-4e60-47ec-a6d6-dd9e475d2c3a?experience=power-bi\\nApp Name: APP\",\"CorrelationId\": \"\",\"User\": \"System\"}]"

Thanks,

Prabhat