Forum Discussion

joshuaking's avatar
joshuaking
Regular Visitor
1 year ago
Solved

write_deltalake with Python Notebook is creating an "Unidentified" folder in my lakehouse.

Hi there, all!   Due to the smaller size of our data at my organization, we decided to use Polars and PyArrow for our data transformation in lieu of PySpark, Spark SQL, and other distributed data p...
  • nilendraFabric's avatar
    1 year ago

    Hello joshuaking 

     

    Here is what I am thinking could happened. 

    when writing a Delta table using Python functions like `write_deltalake` or Polars’ `write_delta`, the underlying data files are correctly created and queryable, yet the lakehouse metadata may not recognize or register the table properly. This can lead to the table being shown under an “Unidentified” folder

     

    Writing data to a specific path does not always automatically update the lakehouse catalog. 

    try using ddl command like in your notebook 
    `CREATE TABLE student USING DELTA LOCATION '/lakehouse/your_schema/your_path/student';`
    to inform the system about the logical table name

     

    if this helps please accept the solution and give kudos

  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Yes it is automatic in Fabric

     

    When files in Delta format (Parquet + `_delta_log` folder) are placed in the managed area of a Lakehouse, Fabric automatically registers them as tables in the catalog.

     

    In most thr cases properly structured delta file from Unidentified folder eventually moves to Table