Forum Discussion

Kuladeep's avatar
Kuladeep
Icon for Advocate II rankAdvocate II
1 day ago

Table level or Columns level lineage

Hallo Community,

is it possible to trace the source tables and columns used in a report?

I understand that intermediate transformations can change the tables and columns during the process. However, I would like to know if there is any workaround or best practice to trace the fields in a report back to their original source tables and columns.

We are having Medallion architecture and using Noteboos (pyspark) to transform the data.

Any tips or suggestions would be greatly appreciated.

5 Replies

  • Hi Kuladeep​,

    Recommended: For table level lineage use Fabric Lineage view.

    Fabric can provide lineage across items such as:

    Source → Bronze → Silver → Gold → Semantic Model → Report

    This is the easiest way to understand dependencies between tables, notebooks, semantic models, and reports.

    If you need column level lineage then you need to build meta data table across your workflow, this is where things will become difficult.

     you can create this by using DAX studio but

    However, DAX Studio will not give you complete lineage back through your PySpark notebooks, it will only limit up to sematic layer, for lineage back to pyspark you must create techinal meta data table in your workspace using pyspark. Something like this

    Target   LayerTarget  TableTarget  Column  Source  TableSource
     


    If this helps, ✓ Mark as Kudos | Help Other





  • Hallo,

    Yes, it is possible — and the fabric-toolbox repository provides a ready-made solution for this exact scenario.

    Microsoft's fabric-toolbox includes a Lineage Extractor notebook that automatically extracts column-level metadata from Fabric artifacts and uploads it to Microsoft Purview to create a graphical lineage graph.

    You can take a look at 

    https://github.com/microsoft/fabric-toolbox/tree/9dcbc14496f4f6342f3af1f482073ff0be0d91e0/tools/Lineage_Extractor

     

    • Kuladeep's avatar
      Kuladeep
      Icon for Advocate II rankAdvocate II

      I think this works solution upto some extent of our usecase. I will test this and get back here

  • I am already aware of it. and also the item level lineage.

    Interested to know if anyone in the community has developed any work around or tricks to achieve table level lineage. Or at least list all the related tables (bronze till gold) to the report

  • Hi Kuladeep​,

    Fabric doesn't have any native column level lineage. There is data source lineage, so you can see which model your report is getting data from, but there is nothing more granular than that.