topic Re: Corrupted metadata detected while processing the Direct Lake table in Data Engineering

Corrupted metadata detected while processing the Direct Lake table

rks — Wed, 30 Apr 2025 05:43:56 GMT

Hi, I have tried to refresh a data set based on the new OneLake Mode for Direct Lake. I ran into this error:
Corrupted metadata detected while processing the Direct Lake table (TableTMID='xxx'). The source parquet file '<oii>xxxxxx2841696645%29.00000000000000000000.parquet</oii>' is corrupted or malformed

i have retrieved the data from the following process:

1) I have a regular, good old import model

2) I have enabled OneLake Integration

3) I have created an empty, schema bound lake house

4) and copied the Tables from the Onelake-Integration into the Tables file from the new lakehouse

5) I have made a copy of the original import model and changed the partitions with TMDL (Deep dive into Direct Lake on OneLake and creating Direct Lake semantic models in Power BI Desktop | Microsoft Power BI Blog | Microsoft Power BI)

Then I wanted to refresh (reframe) the Model in order to test performance with existing reports. The refresh failed due to the error mentioned above.

Is there a way to "repair" the corrupted meta data for a given set of tables?

Best regards

Konstantin

Re: Corrupted metadata detected while processing the Direct Lake table

v-prasare — Wed, 30 Apr 2025 12:34:05 GMT

Hi @rks,

Can you try once below trobleshooting steps and let me know if this helps?

The error may be due to corrupted or improperly copied Parquet files. Avoid manual file copying and use Dataflows or notebooks to move data properly.
Schema mismatches between source and target lakehouses can cause issues. Ensure the schema is identical by comparing with a Spark notebook.
Hidden metadata files might be missing after manual copying. These are essential for correct operation and are not copied with normal file tools.
Modifying partitions using TMDL can cause issues if model bindings don't align with the data files. Double-check your partition paths and formats.
To resolve the issue, recreate the table using Fabric tools, re-ingest the data, validate the schema, and rebind the model accurately.

Thanks,
Prashanth Are
MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Corrupted metadata detected while processing the Direct Lake table

rks — Fri, 09 May 2025 06:05:16 GMT

Hi @v-prasare
Maybe you can help me once more? I was able to follow the documentation up to a point where I created a lakehouse with shortcuts to the semantic model. Then I wanted to use the Copy Job fabric item to move the data phyiscally into another lakehouse. However there's an error message preventing me to create the job:

Is there another way to move the data? I am not proficient with Python (notebooks)...