Forum Discussion
Soobramoney
Advocate I
Advocate ILakehouse backup table
Hi, I have a data pipeline that loads data into a staging table with the destination setting set to "Overwrite." This seems to create a _backup_XXXXXXXXXX table each time I load data. Prior to us...
It seems you cannot prevent the backup tables when you are using the Overwrite setting. I had to create a clean job that runs once a day.
import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("BackupTableCleanup") backup_tables = spark.sql("SHOW TABLES") drop_backup_tables = backup_tables.filter( (backup_tables["tableName"].like('%_backup_%')) & ) drop_statements = drop_backup_tables.rdd.map(lambda row: f"DROP TABLE {row['tableName']};").collect() for statement in drop_statements: try: logger.info(f"Executing: {statement}") print(f"Executing: {statement}") spark.sql(statement) logger.info(f"Successfully executed: {statement}") except Exception as e: logger.error(f"Failed to execute: {statement}. Error: {e}")
SoobramoneyAdvocate I
Advocate IIt seems you cannot prevent the backup tables when you are using the Overwrite setting. I had to create a clean job that runs once a day.
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("BackupTableCleanup")
backup_tables = spark.sql("SHOW TABLES")
drop_backup_tables = backup_tables.filter(
(backup_tables["tableName"].like('%_backup_%')) &
)
drop_statements = drop_backup_tables.rdd.map(lambda row: f"DROP TABLE {row['tableName']};").collect()
for statement in drop_statements:
try:
logger.info(f"Executing: {statement}")
print(f"Executing: {statement}")
spark.sql(statement)
logger.info(f"Successfully executed: {statement}")
except Exception as e:
logger.error(f"Failed to execute: {statement}. Error: {e}")
- ToddChitt
Super User
I know this post is over a year old, but is this really an acceptable solution? Seems more like "Defect support system" (meaning we have to add extra coding and stuff to 'fix' stuff that Microsoft won't address.
