Cannot load delta lake table into Fabric created by Streaming job

Hi all,

I'm using event hub with streaming analytics job, which is capturing incoming messages and stores it into delta lake table stored in ADLS Gen2 storage. Incoming json message contains 5 nested levels of properties/elements. I want to store the json payload into one column.

When message is processed using Stream analytics job, delta table is created ok. In Fabric, I've created shortcut to that delta lake table successfully. When I want to load that file into table using spark, I've got an error:

Spark_User_UserApp_ArrayIndexOutOfBoundsException

Please verify if the application is attempting to access a non-existent array index. Remember that arrays are 0-indexed, so a 3-element array will have indexes 0, 1, 2.

The same happens when I wanted to load table using GUI in Fabric.

When I change the json message and remove 5th nested level, surprisingly the load finished successfully.

Any thoughts?

data engineering

one lake

HimanshuS-msft
2 years ago
Hi KarolPapaj
I did a quick check if this is an issue with the JSON . I did took that the JSON which you shared and tried to read it from the notebook . Honestly I thought it will fail as you had mentioned , surprisingly it did worked fine 🙂

df = spark.read.option("multiline", "true").json("Files/Karol.json")

# df now is a Spark DataFrame containing JSON data from "Files/Karol.json".

display(df)

Now since you are using the ASA to create the delta table , I think it will best if you create a support ticket and let them investigate this .

Thanks
Himanshu

5 Replies

v-cboorla-msft
Microsoft Employee
2 years ago
Hi KarolPapaj

Thanks for using Microsoft Fabric Community.

Can you please provide the spark code which you are using to load the tables? This would help me in understanding the issue better.
Please provide the schema of the files along with the screenshots.
Thanks.
- KarolPapaj
  New Member
  2 years ago
  Sure, here is the spark code:
  # Load data from parquet file from ADLS Gen2 shrtcut
  df = spark.read.parquet("Files/landing/rabbit_mq_messages_test/8248458b7943442c81e1e48a41cf83c7_1.parquet")
  # df now is a Spark DataFrame containing parquet data from "Files/landing/rabbit_mq_messages_test/bee298959ac9439283831be21db4f0df_1.parquet".
  display(df)
  
  The file structure is like this:
  {
  "event": "SomeText",
  "messageDateTime": "2023-09-21T00:21:58.677387Z",
  "metadata": {
  "countryCode": "SomeText",
  "relationUid": "SomeText",
  "processUid": "SomeText",
  "entityUid": "SomeText",
  "productCode": "SomeText",
  "channel": "SomeText"
  },
  "additionalParams": {
  "POLICY_RULES": {
  "requestUid": "17bcb656-0ec2-4f44-a722-f4e32f3f4f1e",
  "rulesetUids": ["SomeText"],
  "productResults": [
  {
  "outcome": "NotOK",
  "productCode": "SomeText",
  "statusCode": "NotOK",
  "statusDescription": "SomeText",
  "policyCheckRecords": [
  {
  "outcome": "SomeText",
  "statusCode": "SomeText",
  "statusDescription": "SomeText"
  },
  {
  "outcome": "SomeText",
  "statusCode": "SomeText",
  "statusDescription": "SomeText"
  }
  ]
  }
  ]
  }
  }
  }
  - v-cboorla-msft
    Microsoft Employee
    2 years ago
    Hi KarolPapaj
    
    Apologies for the issue you have been facing. I would like to check are you still facing this issue?
    
    If the issue still persists, please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link
    
    After creating a Support ticket please provide the ticket number as it would help us to track for more information.
    
    Hope this helps. Please let us know if you have any other queries.
HimanshuS-msft
Microsoft Employee
2 years ago
Hi KarolPapaj
I did a quick check if this is an issue with the JSON . I did took that the JSON which you shared and tried to read it from the notebook . Honestly I thought it will fail as you had mentioned , surprisingly it did worked fine 🙂

df = spark.read.option("multiline", "true").json("Files/Karol.json")

# df now is a Spark DataFrame containing JSON data from "Files/Karol.json".

display(df)

Now since you are using the ASA to create the delta table , I think it will best if you create a support ticket and let them investigate this .

Thanks
Himanshu
v-cboorla-msft
Microsoft Employee
2 years ago
Hi KarolPapaj

Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

Thanks