Forum Discussion
Helper IIData Extraction from Oracle
Hi Ira_27 ,
Thanks for the clarification. I understand you are using notebook only to read the .sql file and pass it to Copy activity.To avoid notebook, one option you can try is storing the SQL query in a config table and reading it using Lookup activity in pipeline. This way you can directly pass it to Copy activity and remove the HC notebook dependency.
If maintaining queries in table is difficult, another option is to store the SQL files in OneLake and use pipeline activities like Lookup/Get Metadata + read content to read the file instead of notebook. This helps you keep the files separate and also avoids the 5 connection limit from HC notebooks.
Please check if this works for you. Let us know if you need any help on this.
Regards,
Community Support Teams.
Helper IIThank you spaceman127 for your response and that is exactly what i am doing right now. The only caviat is that since my Oracle source queries are stored in .sql file i am using notebook to read the content of the file and passing it as a variable in the copy activity. I initially thought of saving the source queries as TEXT in a table but then it add's overhead when someone has to review or update the source queries. Do you have any other recommendation?
Community SupportHi Ira_27 ,
Thanks for the clarification. I understand you are using notebook only to read the .sql file and pass it to Copy activity.
To avoid notebook, one option you can try is storing the SQL query in a config table and reading it using Lookup activity in pipeline. This way you can directly pass it to Copy activity and remove the HC notebook dependency.
If maintaining queries in table is difficult, another option is to store the SQL files in OneLake and use pipeline activities like Lookup/Get Metadata + read content to read the file instead of notebook. This helps you keep the files separate and also avoids the 5 connection limit from HC notebooks.
Please check if this works for you. Let us know if you need any help on this.
Regards,
Community Support Teams.
- Ira_27
Thanks, i totally missed out on using the Get Metadata, i guess i was too much stuck on using Notebook that i overlooked our good old "Get Metadata" friend.