Forum Discussion
Spark SQL query data in one datalake and store transformed data in another data lake
Hi SergioTorrinha , Hope your doing well.
In this scenario i suggest you to raise a support ticket here. so, that they can assit you in addressing the issue you are facing. please follow below link on how to raise a support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thanks,
Prashanth Are
MS Fabric community support
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query
Hi nilendraFabric !
Thanks for you'r input.
Do I really need to create shortcuts, for this kind of task, even if these lakehouses are within the same tenant/domain/workspace?
Imagine I have multiple tables in bronze_lh, for which I would like to do the same sort of operations. Would I need to create those shortcuts in the silver_lh? How is one automating such task?
Thank you.
Hello SergioTorrinha
In your notebook, you need access to the Bronze table. If both lakehouses share the same workspace, you can reference the table directly (for example, using its fully qualified name). Otherwise, create a shortcut in your Silver lakehouse that points to the Bronze table
-- Using fully qualified name (catalog.schema.table)
CREATE OR REPLACE TABLE silver_lh.tableA_silver AS
SELECT
Cola
FROM bronze_lh.tableA
Ok, now this is making more sense, I'll have to try this one out.
Thank you for your help!Hi again nilendraFabric !
Sorry to pull this topic again, but after testing I didn't quite got the results I was expecting after your input. I somewhat feel this has a really simple solution, but aparently I cant get there alone.
Below code:%%sql CREATE OR REPLACE TABLE lh_silver.dbo.test_table_pls_drop_me AS SELECT * FROM dbo.bronze_table LIMIT 10
Throws the following error message:
[REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `lh_silver`.`dbo`.
Knowing that:- the notebook I am running this code is associated to lh_bronze
- both lh_bronze and lh_silver are located in the same workspace
what am I doing wrong?
Apretiate your help in advance.Hi everyone!
can someone from the support team, help with this one, please?
Much apretiated.Hi everyone!
I keep facing this issue, can someone help with this one, please?
Thank you.