Forum Discussion

cadams123's avatar
cadams123
New Member
4 months ago
Solved

INFORMATION_SCHEMA

Hi all,
I’m currently migrating tables from Oracle into Microsoft Fabric (Lakehouse) and I’m trying to show progress / coverage of the migration to management.

 

What I’m trying to achieve The requirement is straightforward:

 

  • Take a list of all Oracle tables (source of truth)
  • Take a list of tables currently landed in Fabric
  • Compare the two to show: What has already been migrated What remains

 

What I’ve done so far
On the Oracle side, I’ve extracted table metadata using:

SELECT DISTINCT owner, table_name FROM all_tab_cols;

 

That result is loaded into a dbo table in my Fabric Lakehouse and works well.

 

On the Fabric side, I’m querying the SQL analytics endpoint using:

SELECT * FROM INFORMATION_SCHEMA.TABLES;

 

This correctly returns the Lakehouse tables and matches what I see in the Explorer.

 

The problem
I can select from INFORMATION_SCHEMA.TABLES, but I can’t seem to do anything useful with it:

Joining it to my Oracle metadata table fails (distributed processing errors)

 

  • CTAS / INSERT / materialising it into a table fails
  • Using it in notebooks (Spark SQL / DataFrames) isn’t possible — Spark can’t see INFORMATION_SCHEMA
    Same behaviour whether I use Lakehouse SQL or Warehouse
  • So effectively, INFORMATION_SCHEMA feels read‑only and isolated, usable only for inspection, not comparison.

 

What I’m trying to understand

 

Is this:
An expected Fabric design / control‑plane limitation?
Something others have run into when doing governance or migration tracking?
A case where the recommended pattern is: Dataflow Gen2? Pipelines? JDBC from a notebook? Warehouse‑only metadata?

 

I’m not looking for workarounds for the sake of it — I’d really like to understand what the supported approach is for comparing source‑system metadata with Lakehouse contents in Fabric.

 

Any insights or real‑world patterns would be greatly appreciated.

Thanks!

  • Hi cadams123 

     

    INFORMATION_SCHEMA in Fabric’s SQL analytics endpoint is a read-only system catalog that can’t be joined to Lakehouse Delta tables or materialized via CTAS . that’s by design, not a bug.

     

    The standard workaround is to snapshot the metadata into a Delta table so you can join it with your Oracle metadata normally.

     

    The spark.catalog.listTables() approach in a notebook remains your best bet for snapshotting Fabric metadata into a joinable Delta table.

     

     

3 Replies

  • Hi cadams123 

     

    INFORMATION_SCHEMA in Fabric’s SQL analytics endpoint is a read-only system catalog that can’t be joined to Lakehouse Delta tables or materialized via CTAS . that’s by design, not a bug.

     

    The standard workaround is to snapshot the metadata into a Delta table so you can join it with your Oracle metadata normally.

     

    The spark.catalog.listTables() approach in a notebook remains your best bet for snapshotting Fabric metadata into a joinable Delta table.

     

     

  • v-priyankata's avatar
    v-priyankata
    Community Support

    Hi cadams123 

    Thank you for reaching out to the Microsoft Fabric Forum Community.

    nilendraFabric Thanks for the inputs.

    I hope the information provided by user was helpful. If you still have questions, please don't hesitate to reach out to the community.

     

    • v-priyankata's avatar
      v-priyankata
      Community Support

      Hi cadams123 

      Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.