Forum Discussion

Kuladeep's avatar
Kuladeep
Icon for Advocate II rankAdvocate II
1 day ago

Table level or Columns level lineage

Hallo Community,

is it possible to trace the source tables and columns used in a report?

I understand that intermediate transformations can change the tables and columns during the process. However, I would like to know if there is any workaround or best practice to trace the fields in a report back to their original source tables and columns.

We are having Medallion architecture and using Noteboos (pyspark) to transform the data.

Any tips or suggestions would be greatly appreciated.

data engineering
lakehouse
Notebook

5 Replies

  • EduardoCastro's avatar
    EduardoCastro
    New Member
    1 day ago

    Hallo,

    Yes, it is possible — and the fabric-toolbox repository provides a ready-made solution for this exact scenario.

    Microsoft's fabric-toolbox includes a Lineage Extractor notebook that automatically extracts column-level metadata from Fabric artifacts and uploads it to Microsoft Purview to create a graphical lineage graph.

    You can take a look at 

    https://github.com/microsoft/fabric-toolbox/tree/9dcbc14496f4f6342f3af1f482073ff0be0d91e0/tools/Lineage_Extractor

     

    • Kuladeep's avatar
      Kuladeep
      Icon for Advocate II rankAdvocate II
      11 hours ago

      I think this works solution upto some extent of our usecase. I will test this and get back here

  • tayloramy's avatar
    tayloramy
    Icon for Super User rankSuper User
    1 day ago

    Hi Kuladeep​,

    Fabric doesn't have any native column level lineage. There is data source lineage, so you can see which model your report is getting data from, but there is nothing more granular than that.  

  • Kuladeep's avatar
    Kuladeep
    Icon for Advocate II rankAdvocate II
    1 day ago

    I am already aware of it. and also the item level lineage.

    Interested to know if anyone in the community has developed any work around or tricks to achieve table level lineage. Or at least list all the related tables (bronze till gold) to the report

  • Murtaza_Ghafoor's avatar
    Murtaza_Ghafoor
    Icon for Super User rankSuper User
    12 hours ago

    Hi Kuladeep​,

    Recommended: For table level lineage use Fabric Lineage view.

    Fabric can provide lineage across items such as:

    Source → Bronze → Silver → Gold → Semantic Model → Report

    This is the easiest way to understand dependencies between tables, notebooks, semantic models, and reports.

    If you need column level lineage then you need to build meta data table across your workflow, this is where things will become difficult.

     you can create this by using DAX studio but

    However, DAX Studio will not give you complete lineage back through your PySpark notebooks, it will only limit up to sematic layer, for lineage back to pyspark you must create techinal meta data table in your workspace using pyspark. Something like this

    Target   LayerTarget  TableTarget  Column  Source  TableSource     


    If this helps, ✓ Mark as Kudos | Help Other