Forum Discussion

felicial's avatar
felicial
Icon for Microsoft Employee rankMicrosoft Employee
1 year ago
Solved

Shortcut of mirrored database won't update

I created a Cosmos DB for NoSQL account and container and enabled mirroring. 

Then I created a shortcut to run a Notebook on my data.

I add new data to Cosmos DB and thhe mirror upadtes. However, the shortcut won't. I need to recreate it everytime to get new data.

 

I have two questions:

1) Why do I have to create a shortcut instead of running the Notebook over the mirrored database?

2) Why won't it update? 

 

Thanks for your support.

data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hi felicial 

    You can access the mirrored database directly from notebooks using Spark SQL queries

    df = spark.sql("SELECT * FROM YourMirroredDatabase.YourTable")
    display(df)

     

    This approach will give you latest data.with a typical latency of up to 60 seconds

     

     

    Ideally both Shortcut and mirrored DB should provide the same infomation.

     

4 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hell0 felicial 

    Thank you for reaching out to the Microsoft Fabric Forum Community.

    the shortcut doesn't dynamically update when new data is added or mirrored in your Cosmos DB. this might be due to shortcut is using cached data or The data refresh mechanism isn't properly configured to reflect updates in real-time

    to avoid recreating shortcuts in your Cosmos DB integration with notebooks. check below resources to track changes to your data and let me know if this helps?
    The Change Feed feature(https://learn.microsoft.com/en-us/azure/cosmos-db/change-feed) in Cosmos DB allows you to listen for changes in your Cosmos DB container and process new or updated data in real-time. This is useful when you want to track and handle changes automatically.
    Using the Change Feed, you can continuously stream new data from Cosmos DB into your notebook environment without needing to manually recreate any shortcuts.


    If this information is helpful, please “Accept it as a solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
    Thank you.

  • felicial's avatar
    felicial
    Icon for Microsoft Employee rankMicrosoft Employee
    1 year ago

    Hi Prasannag,

    Thanks for your answer.

     

    My intention is to use the mirroring feature in Fabric, which relies on the continous backup of Cosmos DB and not the change feed. I am trying to build the Notebook in Fabric. 

    What options are there for "data refresh mechanisms"? I thought it was automatic.

     

    Thanks!

  • nilendraFabric's avatar
    nilendraFabric
    Icon for Super User rankSuper User
    1 year ago

    Hi felicial 

    You can access the mirrored database directly from notebooks using Spark SQL queries

    df = spark.sql("SELECT * FROM YourMirroredDatabase.YourTable")
    display(df)

     

    This approach will give you latest data.with a typical latency of up to 60 seconds

     

     

    Ideally both Shortcut and mirrored DB should provide the same infomation.

     

    • felicial's avatar
      felicial
      Icon for Microsoft Employee rankMicrosoft Employee
      1 year ago

      Thanks, I will try using the Notebook directly on the mirror. The shortcut won't update...

      Will post the results. Thanks a lot!