Refresh Dataflow after Databricks job

[johnpjustus]

Hi All,

I have an requirement like this.

I have a job in databricks, basically this job refreshes the datamodel in databricks (refreshes data in Gold layer)

Now as soon as this job completes, I woud like to refresh the dataflow in PowerBI service.

Does anyone tried this?

Thanks,

John

[Anonymous]

Hi @johnpjustus ,

You can use Power Automate is scheduled checks for periodic triggers, depending on the job is expected completion time.

After confirming that the Databricks job is complete, use Power Automate to call the Power BI REST API to trigger a refresh of the data flow.

For detailed steps on setting up a process in Power Automate, including how to use the Power BI REST API in Power Automate, please see the following documentation:

Create a cloud flow in Power Automate - Power Automate | Microsoft Learn

Dataflows - Refresh Dataflow - REST API (Power BI Power BI REST APIs) | Microsoft Learn

If you have any further questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[johnpjustus]

Thank you Yang for the suggestions.

The thing is, I wouldnt be knowing when the databricks job completes, so I can't stick to an expected completion time and then trigger the power automate to run.

Any ideas?

Thanks,

John

[Anonymous · Accepted solution]

Hi @johnpjustus ,

The official document mentioned:

In some cases, you may need to pass some values from the notebook back to the service. These values can be used for control flow (conditional inspection) in the service or use by downstream activities (limited to 2 MB).

I think you are in line with this situation.

You can return a value after your homework is completed. This value can be used in services such as Azure data factory, and then the Power Automate stream can be triggered.

For details, please refer to the screenshots and documents below:

Azure Databricks activity - Microsoft Fabric | Microsoft Learn

If you have other questions, please contact me at any time.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

[johnpjustus]

Thank you so much. I am sure this is going to be helpful.