Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
LB-Tech
Helper I
Helper I

Suggest an efficient automated pipeline idea

Hi,

I am currently working with two softwares, Meditos (Lab) and Wondersoft Shopaid (Pharmacy), which provide automated backups at specific intervals in the form of SQL and .bak files respectively.

I want to set up a real-time data pipeline to integrate these backups with Power BI for reporting.

Could you suggest me some best ways to 

     1. connect these backups to Power BI in real-time*?
     2. Any techniques, or workflows (e.g., ETL tools, connectors) that can help automate this process effectively(cost effectively)?
 

1 ACCEPTED SOLUTION
v-prasare
Community Support
Community Support

Hi  @LB-Tech ,

Thanks for reaching out to Microsoft Fabric community support.

 

from above question your trying to set up a real-time data pipeline that integrates SQL and .bak backups. please follow below mentioned steps and let me know if this helps:

 

  1. Setup a Centralized Database
  1. Automate Backup Processing

For SQL Files:

  • Write a script (Python, PowerShell, or Bash) to Detect new SQL files in the backup folder. Execute the SQL files to update the database.

For .bak Files:

  • Use SQL Server commands or tools to automate .bak file restoration. please refer to similar community post

Solved: Re: Is it possible to connect to a .BAK file? - Microsoft Fabric Community

 

Automate this process using:

  • Windows Task Scheduler or CRON jobs, ADF (Azure Data Factory) or custom scripts for orchestration.
  1. Data Transformation
    • Use an ETL tool or custom scripts toClean the data, Join and aggregate data, Apply business rules for reporting
  1. Real-Time Data Integration
    • DirectQuery for Real-Time Reporting: Connect Power BI directly to the centralized database. Changes in the database are reflected in Power BI in near-real-time.
    • Scheduled Refresh: Use Power BI Import mode for pre-aggregated data. Set a refresh schedule to pull the latest data.

 

Thanks,

Prashanth Are

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

View solution in original post

6 REPLIES 6
v-prasare
Community Support
Community Support

Hi @LB-Tech ,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

 

 

Your feedback is important to us, Looking forward to your response. 

 

Thanks,

Prashanth Are

MS Fabric community support.

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

v-prasare
Community Support
Community Support

Hi  @LB-Tech ,

Thanks for reaching out to Microsoft Fabric community support.

 

from above question your trying to set up a real-time data pipeline that integrates SQL and .bak backups. please follow below mentioned steps and let me know if this helps:

 

  1. Setup a Centralized Database
  1. Automate Backup Processing

For SQL Files:

  • Write a script (Python, PowerShell, or Bash) to Detect new SQL files in the backup folder. Execute the SQL files to update the database.

For .bak Files:

  • Use SQL Server commands or tools to automate .bak file restoration. please refer to similar community post

Solved: Re: Is it possible to connect to a .BAK file? - Microsoft Fabric Community

 

Automate this process using:

  • Windows Task Scheduler or CRON jobs, ADF (Azure Data Factory) or custom scripts for orchestration.
  1. Data Transformation
    • Use an ETL tool or custom scripts toClean the data, Join and aggregate data, Apply business rules for reporting
  1. Real-Time Data Integration
    • DirectQuery for Real-Time Reporting: Connect Power BI directly to the centralized database. Changes in the database are reflected in Power BI in near-real-time.
    • Scheduled Refresh: Use Power BI Import mode for pre-aggregated data. Set a refresh schedule to pull the latest data.

 

Thanks,

Prashanth Are

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Can we setup a On-Prem Gateway for this task?

Yes, setting up an On-Premises Data Gateway is a practical solution for integrating your on-premises data with Power BI in near real-time.

for more info related refer to this doc: Power BI implementation planning: Data gateways - Power BI | Microsoft Learn 

 

 

 

Thanks,

Prashanth Are

MS Fabric community team

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

Hi @LB-Tech,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

 

 

Your feedback is important to us, Looking forward to your response. 

 

Thanks,

Prashanth Are

MS Fabric community support.

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

spencer_sa
Super User
Super User

I'm assuming you're missing a footnote regarding real-time.  What do you mean by 'real-time'?  Actual real time or polling at intervals?  Actual real time is likely to be Event Stream and Kusto driven.

Helpful resources

Announcements
FabCon Global Hackathon Carousel

FabCon Global Hackathon

Join the Fabric FabCon Global Hackathon—running virtually through Nov 3. Open to all skill levels. $10,000 in prizes!

September Fabric Update Carousel

Fabric Monthly Update - September 2025

Check out the September 2025 Fabric update to learn about new features.

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.