Forum Discussion

st_0999's avatar
st_0999
Helper II
3 years ago

DataSource.Error: Microsoft SQL: Error handling external file: 'Invalid: Parquet file size is 0 byte

Ok so here is my journey:   Tried to use PQ Online - but face a cap of 10 minutes, and kept getting a 10 minute timeout error, and the data flow would fail. So, I did 80% in PQ, Online in a Gen2 D...
Pipelines
st_0999's avatar
st_0999
Helper II
3 years ago

I think I know why this was happening

 

When Notebook writes the Parquet files, e.g. using the line:

sorted_df.write.mode("overwrite").format("parquet").save("Files/ + parquet file names)
 
It also writes a _SUCCESS file. 
 
The Copy step in the Data Pipeline also copies this, which messes up querying the Delta Table. Probably worth the Fabric team looking into fixing?
 
So I had to in the Data Pipeline:
 

1. Original Gen2 Dataflow (80%) 

2. Notebook Pyspark (for the remaining 20% - it's super quick!, only 30 seconds - a short time on complex steps)

3. Delete the _SUCCESS file only (before the Copy Activity)

4. A Copy Activitiy to pick up the Parquet files from the Files space, and destination as a Delta Table in Lake House (here I can actually set the name of Delta Table as UPPER CASE - which is what I wanted)

5. A Delete file steps to delete the temporary Parque files that the Notebook loaded into the File space

 

Not ideal, but its a workaround for me! SPARK should really support UPPER CASE table names.

ajarora's avatar
ajarora
Microsoft Employee
3 years ago

You can exclude the SUCCESS file during copy, by using wild card to specify the files to be copied. Something like this:

 

If you used "file path" option, there too if you specify the "file format" as parquet, it shouldnt have copied the success file. If that happens, it looks like a bug, please let me know if that was the case.

  • st_0999's avatar
    st_0999
    Helper II
    3 years ago

    Thank you. I will try it. It never occured to me to use a WildCard Copy.