topic Re: Suggest an efficient automated pipeline idea in Data Engineering

Suggest an efficient automated pipeline idea

LB-Tech — Fri, 13 Dec 2024 07:05:28 GMT

Hi,

I am currently working with two softwares, Meditos (Lab) and Wondersoft Shopaid (Pharmacy), which provide automated backups at specific intervals in the form of SQL and .bak files respectively.

I want to set up a real-time data pipeline to integrate these backups with Power BI for reporting.

Could you suggest me some best ways to

1. connect these backups to Power BI in real-time*?
2. Any techniques, or workflows (e.g., ETL tools, connectors) that can help automate this process effectively(cost effectively)?

Re: Suggest an efficient automated pipeline idea

spencer_sa — Fri, 13 Dec 2024 08:40:38 GMT

I'm assuming you're missing a footnote regarding real-time. What do you mean by 'real-time'? Actual real time or polling at intervals? Actual real time is likely to be Event Stream and Kusto driven.

Re: Suggest an efficient automated pipeline idea

v-prasare — Fri, 20 Dec 2024 04:32:52 GMT

Hi LB-Tech ,

Thanks for reaching out to Microsoft Fabric community support.

from above question your trying to set up a real-time data pipeline that integrates SQL and .bak backups. please follow below mentioned steps and let me know if this helps:

Setup a Centralized Database

Use a SQL Server instance(on-premise or cloud)
Centralize all data by, Loading SQL backups directly into the database and Restoring .bak files periodically.

Automate Backup Processing

For SQL Files:

Write a script (Python, PowerShell, or Bash) to Detect new SQL files in the backup folder. Execute the SQL files to update the database.

For .bak Files:

Use SQL Server commands or tools to automate .bak file restoration. please refer to similar community post

Solved: Re: Is it possible to connect to a .BAK file? - Microsoft Fabric Community

Automate this process using:

Windows Task Scheduler or CRON jobs, ADF (Azure Data Factory) or custom scripts for orchestration.

Data Transformation

Use an ETL tool or custom scripts toClean the data, Join and aggregate data, Apply business rules for reporting

Real-Time Data Integration

DirectQuery for Real-Time Reporting: Connect Power BI directly to the centralized database. Changes in the database are reflected in Power BI in near-real-time.
Scheduled Refresh: Use Power BI Import mode for pre-aggregated data. Set a refresh schedule to pull the latest data.

Thanks,

Prashanth Are

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Re: Suggest an efficient automated pipeline idea

LB-Tech — Sat, 14 Dec 2024 09:00:19 GMT

Can we setup a On-Prem Gateway for this task?

Re: Suggest an efficient automated pipeline idea

v-prasare — Mon, 16 Dec 2024 08:17:28 GMT

Yes, setting up an On-Premises Data Gateway is a practical solution for integrating your on-premises data with Power BI in near real-time.

for more info related refer to this doc: Power BI implementation planning: Data gateways - Power BI | Microsoft Learn

Thanks,

Prashanth Are

MS Fabric community team

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

Re: Suggest an efficient automated pipeline idea

v-prasare — Fri, 20 Dec 2024 09:12:53 GMT

Hi LB-Tech,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

Your feedback is important to us, Looking forward to your response.

Thanks,

Prashanth Are

MS Fabric community support.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

Re: Suggest an efficient automated pipeline idea

v-prasare — Thu, 26 Dec 2024 11:22:41 GMT

Hi LB-Tech ,
Hope your doing well.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution helps you? or let us know if you need any further assistance here?

Your feedback is important to us, Looking forward to your response.

Thanks,

Prashanth Are

MS Fabric community support.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.