Need help to parse JSON string that has escape chars

Thank you for your help on this. And the statement did worked for Example-1 and Example-2.

Here is what I added:  

# Escape control characters in the JSON string
log_data = log_data.replace('\n', '\\n').replace('\t', '\\t')

I tried below example and that failed as looks like need help to handle \". It will be great help if you can assit in this as well.

log_data = "[{\"ApplicationName\": \"App003\",\"WorkspaceId\": \"af00f007-7e26-4654-803c-d82f11108b79\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_App003_Ingest\",\"Operation\": \"ETL\",\"Run_Id\": \"e0dc524a-b36a-49da-8af7-dac7ead251af\",\"SessionId\": \"\",\"Message\": \"[DEV] - App003 - Pipeline Execution Failed - 20250319\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_App003_Ingest\\nRun Id: e0dc524a-b36a-49da-8af7-dac7ead251af\\nError Message: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o5298.save.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 299.0 failed 4 times, most recent failure: Lost task 0.3 in stage 299.0 (TID 3552) (vm-89b62646 executor 1): org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME] You may get a different result due to the upgrading to Spark >= 3.0:\nreading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z\nfrom Parquet files can be ambiguous, as the files may be written by\nSpark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar\nthat is different from Spark 3.0+'s Proleptic Gregorian calendar.\nSee more details in SPARK-31404. You can set the SQL config \"spark.sql.parquet.datetimeRebaseModeInRead\" or\nthe datasource option \"datetimeRebaseMode\" to \"LEGACY\" to rebase the datetime values\nw.r.t. the calendar difference during reading. To read the datetime values\nas it is, set the SQL config or the datasource option to \"CORRECTED\".\n	at org.apache.spark.sql.errors.QueryExecutionErrors$.sparkUpgradeInReadingDatesError(QueryExecutionErrors.scala:763)\n	at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInRead(DataSourceUtils.scala:178)\n	at org.apache.spark.sql.execution.datasources.DataSourceUtils.newRebaseExceptionInRead(DataSourceUtils.scala)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.rebaseTimestamp(ParquetVectorUpdaterFactory.java:1100)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.rebaseMicros(ParquetVectorUpdaterFactory.java:1113)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$LongWithRebaseUpdater.decodeSingleDictionaryId(ParquetVectorUpdaterFactory.java:577)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdater.decodeDictionaryIds(ParquetVectorUpdater.java:75)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:240)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:328)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:219)\n	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)\n	at org.apache.spark.sql.execution.datasources.RecordReaderIterator$$anon$1.hasNext(RecordReaderIterator.scala:61)\n	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:158)\n	at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:615)\n	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)\n	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)\n	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:764)\n	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:424)\n	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)\n	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)\n	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)\n	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:368)\n	at org.apache.spark.rdd.RDD.iterator(RDD.scala:332)\n	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)\n	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\n	at org.apache.spark.scheduler.Task.run(Task.scala:139)\n	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:574)\n	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)\n	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:577)\n	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n	at java.base/java.lang.Thread.run(Thread.java:829)\n\nDriver stacktrace:\n	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2871)\n	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2807)\n	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2806)\n	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\n	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\n	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\n	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2806)\n	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1229)\n	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1229)\n	at scala.Option.foreach(Option.scala:407)\n	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1229)\n	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3070)\n	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3009)\n	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2998)\n	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\n	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:988)\n	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2418)\n	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2439)\n	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2458)\n	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2483)\n	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1029)\n	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)\n	at org.apache.spark.rdd.RDD.withScope(RDD.scala:409)\n	at org.apache.spark.rdd.RDD.collect(RDD.scala:1028)\n	at org.apache.spark.sql.execution.SparkPlan.executeCollectIterator(SparkPlan.scala:491)\n	at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.$anonfun$relationFuture$1(BroadcastExchangeExec.scala:137)\n	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:236)\n	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n	at java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.spark.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME] You may get a different result due to the upgrading to Spark >= 3.0:\nreading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z\nfrom Parquet files can be ambiguous, as the files may be written by\nSpark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar\nthat is different from Spark 3.0+'s Proleptic Gregorian calendar.\nSee more details in SPARK-31404. You can set the SQL config \"spark.sql.parquet.datetimeRebaseModeInRead\" or\nthe datasource option \"datetimeRebaseMode\" to \"LEGACY\" to rebase the datetime values\nw.r.t. the calendar difference during reading. To read the datetime values\nas it is, set the SQL config or the datasource option to \"CORRECTED\".\n	at org.apache.spark.sql.errors.QueryExecutionErrors$.sparkUpgradeInReadingDatesError(QueryExecutionErrors.scala:763)\n	at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInRead(DataSourceUtils.scala:178)\n	at org.apache.spark.sql.execution.datasources.DataSourceUtils.newRebaseExceptionInRead(DataSourceUtils.scala)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.rebaseTimestamp(ParquetVectorUpdaterFactory.java:1100)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.rebaseMicros(ParquetVectorUpdaterFactory.java:1113)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$LongWithRebaseUpdater.decodeSingleDictionaryId(ParquetVectorUpdaterFactory.java:577)\n	at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdater.decodeDictionaryIds(ParquetVectorUpdater.java:75)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:240)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:328)\n	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:219)\n	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)\n	at org.apache.spark.sql.execution.datasources.RecordReaderIterator$$anon$1.hasNext(RecordReaderIterator.scala:61)\n	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:158)\n	at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:615)\n	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)\n	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)\n	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:764)\n	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:424)\n	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)\n	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)\n	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)\n	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:368)\n	at org.apache.spark.rdd.RDD.iterator(RDD.scala:332)\n	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)\n	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)\n	at org.apache.spark.scheduler.Task.run(Task.scala:139)\n	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:574)\n	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)\n	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:577)\n	... 3 more\n' : \\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/af00f007-7e26-4654-803c-d82f11108b79/pipelines/PL_App003_Ingest/e0dc524a-b36a-49da-8af7-dac7ead251af?experience=power-bi\\nApp Name: DCRM\",\"CorrelationId\": \"\",\"User\": \"System\"}]"

Thanks,

Prabhat

Hi Friend,

Below is another example that is failing as the Details section has text enclosed with " ("Internal Server Error") and I am not sure how do I parse these unescaped " chars insode the message. I cant directly do replace for entire file as the element names need to be enclosed with ".

Please suggest how can I parse these types of messages that are having text enclosed with ".

var_str_log_data = "[{\"ApplicationName\": \"APP\",\"WorkspaceId\": \"0addb382-fa0d-4ce1-9c3c-95e32b957955\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_APP_Ingest\",\"Operation\": \"ETL\",\"Run_Id\": \"88276e4f-4e60-47ec-a6d6-dd9e475d2c3a\",\"SessionId\": \"\",\"Message\": \"[DEV] - APP - Pipeline Execution Failed - 20250329\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_APP_Ingest\\nRun Id: 88257e4f-4e60-34ec-a6d6-dd9e475d2c3a\\nError Message: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o4628.load.\n: java.util.concurrent.ExecutionException: Operation failed: \"Internal Server Error\", 500, HEAD, http://msit-onelake.dfs.fabric.microsoft.com/MYWORKSPACE/dataverse_name_cds2_workspace_unq60feaf930cbf236eb2519b27f2d6b.Lakehouse/Tables/app_riskparcel/_delta_log?upn=false&action=getStatus&timeout=90\n	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)\n	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)\n	... 37 more\n' : \\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/0addb122-fa0d-4ce1-9c3c-95e25b957955/pipelines/PL_APP_Ingest/88223e4f-4e60-47ec-a6d6-dd9e475d2c3a?experience=power-bi\\nApp Name: APP\",\"CorrelationId\": \"\",\"User\": \"System\"}]"

Thanks,

Prabhat