Forum Discussion

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Cannot read Parquet from local Spark cluster storage

Hi, I've got an issue in a Synapse Spark cluster whereby a 3rd party tool generates data and saves to the clusters local tmp directory.  What I would like to do is use Spark to read the parquet file...
data engineering
Anonymous's avatar
Anonymous
Not applicable
2 years ago

HI AndyDDC,

After further test with this, I think this should related to the path. Currently the notebook will try to find the file from the notebook run time environment. (obviously, your file should not exist in it)

Perhaps you can try to move the file to file server or share this file/folder and allow to access remotely. Then you can use correspond file path and use your notebook to access and read data remotely.
Regards,

Xiaoxin Sheng