Column and table descriptions in Lakehouse tables

[zdlfabric]

We suggest the idea that it becomes possible to add column and table descriptions directly to Lakehouse tables in Fabric. 

Currently, it is only possible to add column and table descriptions to tables in semantic models in Fabric.

In Microsoft Purview, data quality scans are no longer possible on semantic model tables (SQL endpoint tables), instead only data quality scans on Lakehouse tables are possible. We would like to be able to see the column and table descriptions in these scans, similarly as how these could be seen in Purview for semantic model column descriptions from Fabric. For this the possibility to add the descriptions to the Lakehouse directly in Fabric is needed.

[gotshalld]

You can add table and column descriptions directly to a lakehouse table using a notebook. This adds comments to the underlying lakehouse delta table. I am guessing Purview would then pick up that metadata of the delta table. Some spark sql code that would accomplish this can be structured like below:

-- 1) Table comment

COMMENT ON TABLE dbo.Sales IS 'Transactions end customer sales.';

-- (2) Inspect your column types (so you know what to re-declare):

DESCRIBE dbo.Sales;

-- (3) Use ALTER TABLE … CHANGE COLUMN to add comments to each column:

ALTER TABLE dbo.Sales CHANGE COLUMN SaleID SaleID BIGINT COMMENT 'Primary key of sale.';
ALTER TABLE dbo.Sales CHANGE COLUMN SaleDate SaleDate DATE COMMENT 'Date when sale occurred.';
ALTER TABLE dbo.Sales CHANGE COLUMN Amount Amount BIGINT COMMENT 'Total dollar amount.';

[zdlfabric]

@gotshalld I can indeed add these comments to the lakehouse, but it seems like we can only view them in the lakehouse delta log files in Fabric or if we query them. They do not show up in Purview, so it is useless for our data steward

[frithjof_v]

This would be great for providing end user documentation.

[nschachter]

We are looking into implementing this in OneLake Catalog.

[zdlfabric]

@nschachter Nice! However, I would like to note that we have private link enabled in our Fabric tenant and unfortunately a lot of features in OneLake Catalog are not available. Please take this into consideration when implementing this in OneLake Catalog, i.e. by making this feature available for private linked tenants as well. Thanks!

[00009]

I would love to be able to define column-level comments directly within materialised lake views. This would allow to have it in one place, including git integration etc. In addition I could imagine that we should be able to re-use existing comments from the source schema enabled lakehouse+table in case of an medallion architecture. This would allow an materialised lake view to re-use the comment from "bronze-schema.table1.columnA" within "silver-schema.table2.columnA".