topic Re: When to use T-SQL notebook in Data Warehouse

When to use T-SQL notebook

Srisakthi — Thu, 07 Nov 2024 03:59:57 GMT

Hello All,

Can you please suggest on what scenario i should consider using T-SQL notebooks for transformations.

1. I have to use only If I have warehouse tables and i need to do transformations on those data?

2. Can I use T-SQL notebooks on lakehouse tables for transformation and push the transformed data to warehouse?

Regards,

Srisakthi

Re: When to use T-SQL notebook

Anonymous — Thu, 07 Nov 2024 06:27:16 GMT

Hi Srisakthi,

#1, I think you can also use it to transform and show the temporary reuslt or save queries as view.

#2, I think dataflow or pipeline features should more suitable to push data from the Lakehouse. If you only use notebook, you may need to use some libraries and pyspark scripts to acheive these. (e.g. use mssparkutils and jdbc driver to write data from dataframe)

Solved: Re: Dataframe write to Warehouse through Notebook.... - Microsoft Fabric Community

Regards,

Xiaoxin Sheng

Re: When to use T-SQL notebook

Srisakthi — Thu, 07 Nov 2024 07:05:03 GMT

Hi Anonymous ,

Thanks for your response. What I'm looking for is in which Scenario I should be using T-SQL notebooks. Because in data pipeline there is no option for T-SQL notebooks. T-SQL notebook to be used for manual analysis and has to be ran manually, no automation for this ?

Regards,

Srisakthi

Re: When to use T-SQL notebook

Anonymous — Fri, 08 Nov 2024 02:24:54 GMT

Hi Srisakthi,

As you said, current it seems not support directly use t-sql model notebook.
For this scenario, I'd like to suggest create a new common notebook to invoke in pipeline and enter the notebook and manually switch its mode to 't-sql'. (I test with this and it works normally)

Regards,

Xiaoxin Sheng

Re: When to use T-SQL notebook

AndyDDC — Fri, 08 Nov 2024 17:10:59 GMT

Hi Srisakthi IMHO T-SQL support in Notebooks is really for adhoc SELECT analysis over the Lakehouse SQL Endpoint. I would not look do do any ETL/ELT code in Notebooks.

So for me, I use it for analysis of data, not for loading/engineering (Lakehouse SQL Endpoint is read only, cannot write data using T-SQL).

Hope that helps.