Suggest an efficient automated pipeline idea

[LB-Tech]

Hi,

I am currently working with two softwares, Meditos (Lab) and Wondersoft Shopaid (Pharmacy), which provide automated backups at specific intervals in the form of SQL and .bak files respectively.

I want to set up a real-time data pipeline to integrate these backups with Power BI for reporting.

Could you suggest me some best ways to 

     1. connect these backups to Power BI in real-time*?
     2. Any techniques, or workflows (e.g., ETL tools, connectors) that can help automate this process effectively(cost effectively)?
 

[v-prasare]

Hi @LB-Tech ,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

Your feedback is important to us, Looking forward to your response. 

Thanks,

Prashanth Are

MS Fabric community support.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

[v-prasare · Accepted solution]

Hi  @LB-Tech ,

Thanks for reaching out to Microsoft Fabric community support.

from above question your trying to set up a real-time data pipeline that integrates SQL and .bak backups. please follow below mentioned steps and let me know if this helps:

1. Setup a Centralized Database

• Use a SQL Server instance(on-premise or cloud)
• Centralize all data by, Loading SQL backups directly into the database and Restoring .bak files periodically.

2. Automate Backup Processing

For SQL Files:

• Write a script (Python, PowerShell, or Bash) to Detect new SQL files in the backup folder. Execute the SQL files to update the database.

For .bak Files:

• Use SQL Server commands or tools to automate .bak file restoration. please refer to similar community post

Solved: Re: Is it possible to connect to a .BAK file? - Microsoft Fabric Community

Automate this process using:

• Windows Task Scheduler or CRON jobs, ADF (Azure Data Factory) or custom scripts for orchestration.

3. Data Transformation

• Use an ETL tool or custom scripts toClean the data, Join and aggregate data, Apply business rules for reporting

4. Real-Time Data Integration

• DirectQuery for Real-Time Reporting: Connect Power BI directly to the centralized database. Changes in the database are reflected in Power BI in near-real-time.
• Scheduled Refresh: Use Power BI Import mode for pre-aggregated data. Set a refresh schedule to pull the latest data.

Thanks,

Prashanth Are

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

[LB-Tech]

Can we setup a On-Prem Gateway for this task?

[v-prasare]

Yes, setting up an On-Premises Data Gateway is a practical solution for integrating your on-premises data with Power BI in near real-time.

for more info related refer to this doc: Power BI implementation planning: Data gateways - Power BI | Microsoft Learn 

Thanks,

Prashanth Are

MS Fabric community team

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

[v-prasare]

Hi @LB-Tech,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

Your feedback is important to us, Looking forward to your response. 

Thanks,

Prashanth Are

MS Fabric community support.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

[spencer_sa]

I'm assuming you're missing a footnote regarding real-time.  What do you mean by 'real-time'?  Actual real time or polling at intervals?  Actual real time is likely to be Event Stream and Kusto driven.