Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
ManuelSAG
Frequent Visitor

Extend Database with live html connection

Hello Community,

 

I would like to build a database with data from our ATS.

The data in the ATS deletes after 4 months and I would like to build a data warehouse.

 

I have a live connection via html with our ATS and pull arround 11.300 data.

 

Problem 1:

When I connect it via Power Query, only 5.000 data is loaded. 
Is it possible to get the full data?

 

Problem 2:

Since the connection is live, when I lose data in out ATS, I also lose data in my Query.
Is it possible to append data in a data-warehouse-structure?

 

Problem 3:

When I figured out a way to append, is there a way I can merge rows with the latest data? So that if a certain column got an update, that the latest update is in the cell when t 2 rows merge?

 

Information:
I use Office 365.

The headers are always the same.

I have 2 IDs I can merge to get a unique value with whom i can identify which ones are duplicates.

 

I hope there is a solution.

Thanks in advance for your help. I am looking forward to the discussion.

 

Greetings

2 REPLIES 2
ManuelSAG
Frequent Visitor

Hi @v-zhangti,

thanks for your comment.
I know how append and merge works. In Problem 3 I am trying to figure out a way to overwrite duplicates instead of deleting it.

v-zhangti
Community Support
Community Support

Hi, @ManuelSAG 

 

Please refer to the following links in the hopes of helping you.
Combine or Append Data in Power BI / Power Query: Main Concepts — The Power User

Solved: Correct way to append/merge data files - Microsoft Power BI Community

 

Best Regards,

Community Support Team _Charlotte

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Fabcon_Europe_Social_Bogo

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Power BI Carousel June 2024

Power BI Monthly Update - June 2024

Check out the June 2024 Power BI update to learn about new features.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.