Forum Discussion

amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago
Solved

Duplicated rows between notebook and SQL Endpoint

I'm having troubles when trying to count the number of rows in my table from Lakehouse.   If I count the rows from SQL Endpoint, I get 221555 rows, if I read the table from notebook and then I coun...
data engineering
  • Scott_Powell's avatar
    Scott_Powell
    2 years ago

    Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

     

    mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
     
    Hope this helps,
    Scott
Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

 

mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
 
Hope this helps,
Scott
DCELL's avatar
DCELL
Resolver I
1 year ago

Does this mean that when reading lakehouse tables from a notebook that I need to run these 3 lines of code for every table I read??