Forum Discussion

jovianaditya's avatar
jovianaditya
Frequent Visitor
2 years ago

Spark managed table automatically update/insert

I've created a spark-managed table from a parquet source path in my ADLSgen2 as a delta table.

Can my spark-managed table automatically get updated/inserted every time new data is in my ADLSgen2 source path?

data engineering
one lake

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi jovianaditya,

    Thanks for using Fabric Community.


    As I understand your source is ADLS Gen2 and you want to see the latest data present in ADLS Gen2 in Fabric Lakehouse Tables automatically.

    I would suggest you to use shortcuts in One Lake as it as a capability to view the latest data in Fabric Lakehouse Tables.

    Please refer this document: OneLake shortcuts - Microsoft Fabric | Microsoft Learn

    Hope it is helpful. Please let me incase of further queries.




    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi jovianaditya ,

      We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi jovianaditya ,

        We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

  • HimanshuS-msft's avatar
    HimanshuS-msft
    Microsoft Employee
    2 years ago

    Hello jovianaditya 
    If the ask to have the new rows / updated rows show up the delta table , then you will have to run the spark code again , You can set it on a schedule  . Its not going to be a realtime scenario .

     





    Thanks 

    Himanshu