Forum Discussion

st_0999's avatar
st_0999
Helper II
3 years ago

DataSource.Error: Microsoft SQL: Error handling external file: 'Invalid: Parquet file size is 0 byte

Ok so here is my journey:   Tried to use PQ Online - but face a cap of 10 minutes, and kept getting a 10 minute timeout error, and the data flow would fail. So, I did 80% in PQ, Online in a Gen2 D...
pipelines
st_0999's avatar
st_0999
Helper II
3 years ago

I think I know why this was happening

 

When Notebook writes the Parquet files, e.g. using the line:

sorted_df.write.mode("overwrite").format("parquet").save("Files/ + parquet file names)
 
It also writes a _SUCCESS file. 
 
The Copy step in the Data Pipeline also copies this, which messes up querying the Delta Table. Probably worth the Fabric team looking into fixing?
 
So I had to in the Data Pipeline:
 

1. Original Gen2 Dataflow (80%) 

2. Notebook Pyspark (for the remaining 20% - it's super quick!, only 30 seconds - a short time on complex steps)

3. Delete the _SUCCESS file only (before the Copy Activity)

4. A Copy Activitiy to pick up the Parquet files from the Files space, and destination as a Delta Table in Lake House (here I can actually set the name of Delta Table as UPPER CASE - which is what I wanted)

5. A Delete file steps to delete the temporary Parque files that the Notebook loaded into the File space

 

Not ideal, but its a workaround for me! SPARK should really support UPPER CASE table names.

  • ajarora's avatar
    ajarora
    Microsoft Employee
    3 years ago

    You can exclude the SUCCESS file during copy, by using wild card to specify the files to be copied. Something like this:

     

    If you used "file path" option, there too if you specify the "file format" as parquet, it shouldnt have copied the success file. If that happens, it looks like a bug, please let me know if that was the case.

    • st_0999's avatar
      st_0999
      Helper II
      3 years ago

      Thank you. I will try it. It never occured to me to use a WildCard Copy.