Cannot read Parquet from local Spark cluster storage

[AndyDDC]

Hi,

I've got an issue in a Synapse Spark cluster whereby a 3rd party tool generates data and saves to the clusters local tmp directory.  What I would like to do is use Spark to read the parquet files that are saved locally, problem is I don't seem to be able to do that with syntax in a Notebook:

spark.read.parquet("File:///tmp/Files/tableone/*.parquet")

I get "file not found" even though the error message actually shows the parquet file, so not sure why it returns "file not found".

Any help appreciated.

[Anonymous]

HI @AndyDDC,

After further test with this, I think this should related to the path. Currently the notebook will try to find the file from the notebook run time environment. (obviously, your file should not exist in it)

Perhaps you can try to move the file to file server or share this file/folder and allow to access remotely. Then you can use correspond file path and use your notebook to access and read data remotely.
Regards,

Xiaoxin Sheng

[Anonymous]

Hi @AndyDDC,

I think this issue should be related to the tmp folder, normally it used to cache some temporary data and they may been cleanup to release resources.
I'd like to suggest you check the folder to confirm if this parquet file stay in the folder and you have enough permission during the notebook operations.(if files have been occupancy with other processing, they may be lockup and you can't read and edit it)
You can also try to move the file to a general folder instead of the specific folders and try again with your operations.
Regards,

Xiaoxin Sheng