Forum Discussion

Koritala's avatar
Koritala
Icon for Post Patron rankPost Patron
2 months ago
Solved

Is there a solution to re-load the deleted records into datalake table

Hi,

If I deleted records of specific customer ID by mistake using delete command in notebook in lakehouse, can I recover deleted  customer records and load into the same lakehouse table (Parquet format).

From the developer side is there any solution for the same instead of going to platform team or admin?

Appriciate if anyone can give answer.

 

Thanks,

Sri.

  • Hi Koritala,

     

    If your Lakehouse table is a Delta table, then yes, you may be able to recover the deleted customer records without going to the platform team/admin.

     

    First check the table history:

     

    DESCRIBE HISTORY your_schema.your_table;​

     

    Find the version before the DELETE happened.
    Option 1: Restore the whole table
    RESTORE TABLE your_schema.your_table TO VERSION AS OF 5;​

     

    This makes the old version the current table state again.
    But be careful: this restores the whole table back to that version. If other valid changes happened after the delete, they may be rolled back too.

     

    Option 2: Recover only the deleted customer records
    If you only want to reload the deleted customer ID, read the previous version and append only those rows back.
    Example in PySpark:
    deleted_customer_id = "C123"
     
    
    old_df = (
        spark.read
            .format("delta")
            .option("versionAsOf", 5)
            .table("your_schema.your_table")
            .filter(f"CustomerID = '{deleted_customer_id}'")
    )
     
    
    old_df.write \
        .format("delta") \
        .mode("append") \
        .saveAsTable("your_schema.your_table")

     

    This is usually safer than restoring the entire table if other data changed after the delete.

     

    Important:
    This only works if the table is Delta and the previous version is still available. If VACUUM or retention cleanup has already removed the old files, then time travel/restore might not work.

     

    If it is only plain Parquet files without Delta transaction log, then you do not have Delta time travel/restore. In that case, you would need to reload the records from the original source, backup, or another copy.

     

    Reference:
     
     

    🔍Parchitect
    Solutions Architect · Microsoft Fabric Specialist

    💡Helpful? Kudos are appreciated.
    ✔️Solved? Mark as Solution so others can find it faster.

  • hey Koritala depending on how the table is stored and configured, yes you have options to reload deleted records.

    Is your table stored in lakehouse as delta table or parquet files?

     

    If your Lakehouse table is a Delta table (most Fabric managed tables are), you may be able to recover the deleted records using Delta Lake Time Travel (refer to Parchitect answer above).

    If the table is stored as Parquet files only, recovery becomes much more difficult. Parquet files do not natively provide transaction history or time travel capabilities. In that case your recovery options are eitherr eload from the source system or restore from a backup (if existed), for this ask the platform/admin team if backup or retention mechanisms are available in the source system.

     

    Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.

  • Hi Sri,

    Yes, it may be possible, depending on how the delete was performed and whether the underlying data is still available.

    If your Lakehouse table is a Delta table, you have a few recovery options:

    • Time Travel: If the deleted data hasn't been removed by VACUUM, you can query an earlier version of the table using VERSION AS OF or TIMESTAMP AS OF to retrieve the deleted records.

    • RESTORE: If you want to revert the entire table to a previous version, you can use the Delta RESTORE command (if supported in your Fabric environment).

    • MERGE/INSERT: If only specific customer records need to be recovered, you can read the previous table version with Time Travel, filter the required customer IDs, and INSERT or MERGE those records back into the current table instead of restoring the entire table.

    If the table is Parquet only (non-Delta), recovery is much more difficult because Parquet doesn't maintain transaction history. In that case, your options are generally limited to:

    • Restoring from a backup or another copy of the data.

    • Reloading the affected records from the original source system.

    • Requesting assistance from your platform team if storage-level recovery or backups are available.

    Before involving the platform team, I'd first verify:

    1. Is the table actually a Delta table or just Parquet files?

    2. Has a VACUUM operation been run since the delete?

    3. Can you view the table history using DESCRIBE HISTORY <table_name>?

    If the history is still available, there's a good chance you can recover just the deleted customer records yourself without restoring the entire table.

    Could you confirm whether your Lakehouse table is a managed Delta table or a Parquet table, and whether the delete was executed using SQL (DELETE FROM) or by deleting the underlying files?

    If this helps, please consider giving it a Like or marking it as the Accepted Solution so others with a similar issue can find it easily.

4 Replies

  • Hi Koritala,

     

    If your Lakehouse table is a Delta table, then yes, you may be able to recover the deleted customer records without going to the platform team/admin.

     

    First check the table history:

     

    DESCRIBE HISTORY your_schema.your_table;​

     

    Find the version before the DELETE happened.
    Option 1: Restore the whole table
    RESTORE TABLE your_schema.your_table TO VERSION AS OF 5;​

     

    This makes the old version the current table state again.
    But be careful: this restores the whole table back to that version. If other valid changes happened after the delete, they may be rolled back too.

     

    Option 2: Recover only the deleted customer records
    If you only want to reload the deleted customer ID, read the previous version and append only those rows back.
    Example in PySpark:
    deleted_customer_id = "C123"
     
    
    old_df = (
        spark.read
            .format("delta")
            .option("versionAsOf", 5)
            .table("your_schema.your_table")
            .filter(f"CustomerID = '{deleted_customer_id}'")
    )
     
    
    old_df.write \
        .format("delta") \
        .mode("append") \
        .saveAsTable("your_schema.your_table")

     

    This is usually safer than restoring the entire table if other data changed after the delete.

     

    Important:
    This only works if the table is Delta and the previous version is still available. If VACUUM or retention cleanup has already removed the old files, then time travel/restore might not work.

     

    If it is only plain Parquet files without Delta transaction log, then you do not have Delta time travel/restore. In that case, you would need to reload the records from the original source, backup, or another copy.

     

    Reference:
     
     

    🔍Parchitect
    Solutions Architect · Microsoft Fabric Specialist

    💡Helpful? Kudos are appreciated.
    ✔️Solved? Mark as Solution so others can find it faster.

  • hey Koritala depending on how the table is stored and configured, yes you have options to reload deleted records.

    Is your table stored in lakehouse as delta table or parquet files?

     

    If your Lakehouse table is a Delta table (most Fabric managed tables are), you may be able to recover the deleted records using Delta Lake Time Travel (refer to Parchitect answer above).

    If the table is stored as Parquet files only, recovery becomes much more difficult. Parquet files do not natively provide transaction history or time travel capabilities. In that case your recovery options are eitherr eload from the source system or restore from a backup (if existed), for this ask the platform/admin team if backup or retention mechanisms are available in the source system.

     

    Appreciate if you can "Kudos" and/or "Accept as Solution" if this answered your query.

  • Hi Sri,

    Yes, it may be possible, depending on how the delete was performed and whether the underlying data is still available.

    If your Lakehouse table is a Delta table, you have a few recovery options:

    • Time Travel: If the deleted data hasn't been removed by VACUUM, you can query an earlier version of the table using VERSION AS OF or TIMESTAMP AS OF to retrieve the deleted records.

    • RESTORE: If you want to revert the entire table to a previous version, you can use the Delta RESTORE command (if supported in your Fabric environment).

    • MERGE/INSERT: If only specific customer records need to be recovered, you can read the previous table version with Time Travel, filter the required customer IDs, and INSERT or MERGE those records back into the current table instead of restoring the entire table.

    If the table is Parquet only (non-Delta), recovery is much more difficult because Parquet doesn't maintain transaction history. In that case, your options are generally limited to:

    • Restoring from a backup or another copy of the data.

    • Reloading the affected records from the original source system.

    • Requesting assistance from your platform team if storage-level recovery or backups are available.

    Before involving the platform team, I'd first verify:

    1. Is the table actually a Delta table or just Parquet files?

    2. Has a VACUUM operation been run since the delete?

    3. Can you view the table history using DESCRIBE HISTORY <table_name>?

    If the history is still available, there's a good chance you can recover just the deleted customer records yourself without restoring the entire table.

    Could you confirm whether your Lakehouse table is a managed Delta table or a Parquet table, and whether the delete was executed using SQL (DELETE FROM) or by deleting the underlying files?

    If this helps, please consider giving it a Like or marking it as the Accepted Solution so others with a similar issue can find it easily.

  • v-kathullac's avatar
    v-kathullac
    Icon for Community Support rankCommunity Support

    Thankyou  Prince0011 , rizalard0684 , Parchitect   for Addressing the issue.

     

    Hi  Koritala  ,

    Thank you for reaching out to Microsoft Fabric Community Forum,

     

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?


    Regards,

    Chaithanya