Column and table descriptions in Lakehouse tables

[zdlfabric]

We suggest the idea that it becomes possible to add column and table descriptions directly to Lakehouse tables in Fabric. 

Currently, it is only possible to add column and table descriptions to tables in semantic models in Fabric.

In Microsoft Purview, data quality scans are no longer possible on semantic model tables (SQL endpoint tables), instead only data quality scans on Lakehouse tables are possible. We would like to be able to see the column and table descriptions in these scans, similarly as how these could be seen in Purview for semantic model column descriptions from Fabric. For this the possibility to add the descriptions to the Lakehouse directly in Fabric is needed.

[gotshalld]

You can add table and column descriptions directly to a lakehouse table using a notebook. This adds comments to the underlying lakehouse delta table. I am guessing Purview would then pick up that metadata of the delta table. Some spark sql code that would accomplish this can be structured like below:

-- 1) Table comment

COMMENT ON TABLE dbo.Sales IS 'Transactions end customer sales.';

-- (2) Inspect your column types (so you know what to re-declare):

DESCRIBE dbo.Sales;

-- (3) Use ALTER TABLE … CHANGE COLUMN to add comments to each column:

ALTER TABLE dbo.Sales CHANGE COLUMN SaleID SaleID BIGINT COMMENT 'Primary key of sale.';
ALTER TABLE dbo.Sales CHANGE COLUMN SaleDate SaleDate DATE COMMENT 'Date when sale occurred.';
ALTER TABLE dbo.Sales CHANGE COLUMN Amount Amount BIGINT COMMENT 'Total dollar amount.';

[zdlfabric]

@gotshalld I can indeed add these comments to the lakehouse, but it seems like we can only view them in the lakehouse delta log files in Fabric or if we query them. They do not show up in Purview, so it is useless for our data steward