Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started

Reply
dphugo
Frequent Visitor

Fabric Notebook job complete but cell still running + ERROR AzureBlobFileSystemStore getUnixTimeInMi

I'm reading JSON files from a S3 shorcut.

It's 78112 small files, total size is 313MB. Largest file size is 20kb and smallest is 2kb.

The Spark Job shows completed (after 9m16s), but the cell is still in a Running state and never finishes (waited over 60 minutes)

This is the code in the cell:

df_raw = spark.read.option("multiline", "true").json("Files/source/S3/text/2024-01-20/*.json")
display(df_raw.printSchema())
df_raw.show(10, False)

dphugo_0-1706892395950.png

 

The "Code Snippets" under the Job->Stage says: "Listing leaf files and directories for 78115 paths:<br/>abfss://<GUID>@onelake.dfs.fabric.microsoft.com/<GUID>/Files/source/S3/text/2024-01-20/<filename>.json, ...

 

The Log shows the same ERROR being generated every few miliseconds:

ERROR AzureBlobFileSystemStore [Thread-54]: getUnixTimeInMillisFromVersion has Exceptionjava.lang.NumberFormatException

dphugo_1-1706892470060.png

The timestamps on these error messages are delayed it seems by a growing window of 10-30 minutes.

 

Spark cluster config is 3 nodes, Runtime 1.2 (Spark 3.4, Delta 2.4) Compute Small, Memory Optimized.

All spark logs show as completed succesfully apart from the error output above. 

 

Anyone else experiencing this and what the resolution was?

 

5 REPLIES 5
mkulikowski
Frequent Visitor

Hi @dphugo  - have they fixed it?

I am getting exactly the same error with shortcuts and parquet files. I wonder if there is any fix for that?

Hi @mkulikowski, unfortunately not.

There were a few email exchanges between myself and the support team, where I sent them logs and metadata over the period of a week.

They did then reach out to meet to go through the issue, but by then we deemed the approach infeasible and went a different route.

Anonymous
Not applicable

Hi @dphugo ,

Thanks for using Fabric Community.

 

Apologies for the issue you have been facing. I would like to check are you still facing this issue? 

It's difficult to tell what could be the reason for this performance. I would request you to wait for sometime and try again.

 

 

If the issue still persists, please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Hope this helps. Please let us know if you have any other queries.

Hi @Anonymous,

 

I'm still experiencing the issue even after creating a new workspace, notebook, lakehouse and s3 shortcut.

I've logged a ticked: 

Support request number:

2402050050000341

 

Anonymous
Not applicable

Hi @dphugo ,

Thanks for sharing the support ticket number.
Support Team will reach out to you and will try to resolve the issue.
Please continue using Fabric Community for your further queries.

Helpful resources

Announcements
Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!