Forum Discussion

Ira_27's avatar
Ira_27
Icon for Helper II rankHelper II
5 months ago
Solved

Data Extraction from Oracle

Hello Community,   I have a design question, we are building a enterprise data warehouse using Fabric Warehouse and I have to stage the data from our Oracle database (source). Currently the way i d...
data engineering
data pipeline
notebook
  • v-hjannapu's avatar
    v-hjannapu
    5 months ago

    Hi Ira_27  ,
    Thanks for the clarification. I understand you are using notebook only to read the .sql file and pass it to Copy activity.

    To avoid notebook, one option you can try is storing the SQL query in a config table and reading it using Lookup activity in pipeline. This way you can directly pass it to Copy activity and remove the HC notebook dependency.

    If maintaining queries in table is difficult, another option is to store the SQL files in OneLake and use pipeline activities like Lookup/Get Metadata + read content to read the file instead of notebook. This helps you keep the files separate and also avoids the 5 connection limit from HC notebooks.

    Please check if this works for you. Let us know if you need any help on this.
    Regards,
    Community Support Teams.

spaceman127's avatar
spaceman127
Icon for Super User rankSuper User
5 months ago

Hi Ira_27,

That’s an interesting approach you’re taking.


We run a few Oracle databases ourselves, and we use Oracle Mirroring to get the data into Fabric in the first place. In my opinion, this works very well, and it’s free up to your capacity limit. That means you don’t pay anything for storage.


Here you can read this.

https://learn.microsoft.com/en-us/fabric/mirroring/overview


Once you have the data in Fabric, you can easily process it using notebooks and populate your lakehouse.

Here’s an overview of how we do it.

1. Transfer Oracle data to Fabric using mirroring
2. Process the data further using notebooks (we use Bronze, Silver, and Gold)
3. Use the data in reports.


I hope that helps

 

Many greetz

Rene