Extend Database with live html connection

[ManuelSAG]

Hello Community,

I would like to build a database with data from our ATS.

The data in the ATS deletes after 4 months and I would like to build a data warehouse.

I have a live connection via html with our ATS and pull arround 11.300 data.

Problem 1:

When I connect it via Power Query, only 5.000 data is loaded. 
Is it possible to get the full data?

Problem 2:

Since the connection is live, when I lose data in out ATS, I also lose data in my Query.
Is it possible to append data in a data-warehouse-structure?

Problem 3:

When I figured out a way to append, is there a way I can merge rows with the latest data? So that if a certain column got an update, that the latest update is in the cell when t 2 rows merge?

Information:
I use Office 365.

The headers are always the same.

I have 2 IDs I can merge to get a unique value with whom i can identify which ones are duplicates.

I hope there is a solution.

Thanks in advance for your help. I am looking forward to the discussion.

Greetings

[ManuelSAG]

Hi @v-zhangti,

thanks for your comment.
I know how append and merge works. In Problem 3 I am trying to figure out a way to overwrite duplicates instead of deleting it.

[v-zhangti]

Hi, @ManuelSAG 

Please refer to the following links in the hopes of helping you.
Combine or Append Data in Power BI / Power Query: Main Concepts — The Power User

Solved: Correct way to append/merge data files - Microsoft Power BI Community

Best Regards,

Community Support Team _Charlotte

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.