Forum Discussion

Dinesh09's avatar
Dinesh09
Frequent Visitor
2 years ago
Solved

How to truncate a lakehouse table.

I have a multiple dataflowgen2 inserts data into the same lakehouse table. Before inserting data into lakehouse table, I would like to truncate the lakehouse table. How can I do that. The lakehouse t...
data engineering
Fundamentals
  • Dinesh09's avatar
    Dinesh09
    2 years ago

    Notebook is the only option to drop the table right ?

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

what are the repercussions if the destination table didn't exist downstream? Eg I'm thinking that dropping the table beforehand and then recreating via the data flow may have less overhead 

Dinesh09's avatar
Dinesh09
Frequent Visitor
2 years ago

Notebook is the only option to drop the table right ?

  • Scott_Powell's avatar
    Scott_Powell
    Icon for Helper V rankHelper V
    2 years ago

    Yep I think so. But again a Gen2 dataflow just set to "replace" will truncate the table before it starts loading the new data. 

     

    Hope this helps!

    Scott

    • Scott_Powell's avatar
      Scott_Powell
      Icon for Helper V rankHelper V
      1 year ago

      Just saw this thread pop up again, and I will say that we 100% don't use gen2 dataflows any more. Months ago (March or April 2024 I think) Microsoft released the capability for a copy activity in a pipeline to source on-prem data. 

       

      All of our copy jobs use this now, and with dynamic partitioning we're able to load data very quickly - I can load 10+ billion rows of data in a few hours. And we don't have to worry about updating a ton of gen2 dataflows when something changes.

       

      Hope this helps,

      Scott