After loading a table from a CSV file in Lakehouse, is there a way to determine which CSV file

[brittade]

I am needing to determine which File a Table in Fabric Lakehouse was loaded from. Whether that's using a REST API or within Lakehouse itself, preferably both. I am using the REST APIs and I don't see any way to determine which file a table was loaded from. Recently I had created a table from a CSV, and that table shows the original file as the relative path and "External" as the type (in Fabric Lakehouse as well as using the REST APIs), not "Managed". But any new tables I create simply show "Managed" as the type and no path to the CSV file. I don't see a way to determine the original File the Table was loaded from, am I missing something?

[v-prasare]

Thanks for participating in fabric community forum.

[v-prasare]

@brittade, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[brittade]

I may have replied to the wrong comment. 

Unfortunately we have to rely on APIs to get this type of metadata for customers for a feature we were once able to provide them, so unfortunately we can't implement something in their lakehouses to track lineage. So we'll have to let our customer know this is something we can no longer provide to them. So it's not fixed, but it sounds like there's no fix. So I can close it i suppose. 

[v-prasare]

Thank you for the update. We will now proceed to close this ticket.

If you have any further queries or require assistance in the future, please feel free to raise a new ticket in the community forum.

We appreciate your engagement and support.

[brittade]

I tried the _delta_log json file and there doesn't appear to be any mention of the file there either.

[v-prasare]

Hi @brittade,

Recent changes in Fabric Lakehouse mean that CSV ingestions now create Managed tables, and the system no longer preserves the file source in the table metadata. As a result, we can no longer determine the original CSV file path through the REST API or by inspecting the _delta_log.

To address this, we recommend implementing custom lineage tracking at ingestion time. This involves creating a metadata table that records the table name, source file path, ingestion method, user, and timestamp as soon as the data is ingested. We can then refer to this metadata for auditing and traceability needs.

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[brittade]

We have to rely on APIs to get this type of metadata for customers, so unfortunately we can't implement something in their lakehouses to track lineage. So we'll have to let our customer know this is something we can no longer provide to them. 

[v-prasare]

Hi @brittade , As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

Hi @brittade,

Thanks for reaching Ms Fabric community support.

when you create a table in a Fabric Lakehouse, the way its source file is referenced depends on how it was ingested:

• If your table is "External," the REST API will show the source file.
• If you ingest data using SQL, Spark, or other Fabric ingestion mechanisms, the table becomes "Managed." Fabric does not track the source file after ingestion.
• You may need to implement workarounds like Delta logs, Spark transformations, or an audit table. If your table was created from a Delta table, you can check the transaction log (_delta_log) inside the Files section of the Lakehouse:

                                Navigate to: Files -> Tables -> YourTableName -> _delta_log, Open the latest JSON log file                                        (00000xxxx.json) and look for add operations, which may reference source file locations.

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[brittade]

@v-prasare I clicked on a csv file in Lakehouse in Fabric and selected to load to a table. When I did this most recently, it created a table that did not have a type "External" it only says managed and doesn't point back to the CSV. I don't see how I can tell that the table was loaded from a particular CSV file. I have an example i created from a couple of months ago in the same manner, and in that previous example when i loaded a file to a table it showed type "External" and the URL in the properties for the table actually points to a csv path. So I feel that something has changed in the last couple of months and i can no longer determine that a table was loaded from a particular csv file. So I want to know how I can find this information now for the source CSV file that a table was loaded from, for example.