Forum Discussion

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Cannot read Parquet from local Spark cluster storage

Hi, I've got an issue in a Synapse Spark cluster whereby a 3rd party tool generates data and saves to the clusters local tmp directory.  What I would like to do is use Spark to read the parquet file...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi AndyDDC,

I think this issue should be related to the tmp folder, normally it used to cache some temporary data and they may been cleanup to release resources.
I'd like to suggest you check the folder to confirm if this parquet file stay in the folder and you have enough permission during the notebook operations.(if files have been occupancy with other processing, they may be lockup and you can't read and edit it)
You can also try to move the file to a general folder instead of the specific folders and try again with your operations.
Regards,

Xiaoxin Sheng