Is my strategy a good one? Need to pull in millions of rows of HubSpot data for the first time

[apmulhearn]

Hello,

I want to know if my data load strategy is a good one, or if there is a better way.

Goal: To be able to find correlations between EmailEvents and Client Conversions.

Background: We use HubSpot for our Email and Dynamics for our Sales CRM.  We have never pulled historical email send, open, and click data before.

My present plan is to query 30 days' worth of email events at a time. Each of these takes about 2 hours to return.
Then, to append them to one another and Turn Off "Include in Report Refresh" for all EXCEPT the most recent one.
Finally, to add an incremental refresh to the most recent one, so that old data will remain and only new data will be loaded.

Please advise if there is a smarter way to do this?

I would like to go back to 1/1/2019 emails. We email ~ 80k contacts ~2x per week, so the Email Events add up quickly. To avoid timeout issues, this is the plan I have. I would like to know if I am forgetting anything, or if there is a better way to do this?

[jdbuchanan71]

@apmulhearn 

HubSpot does have an option to set up a data feed to a database for your data that you can then pull from.
https://ecosystem.hubspot.com/marketplace/apps/marketing/analytics-data/powerbi

The once a day refresh is $99/mo so not too pricey which may make it worth looking into.  I have not used this myself but I came across it doing some research for a co-worker.

My though on your initial plan is that is would be **bleep**ile in the sense that any problem with the old data would require you to refresh using, I assume, the APIs?  That could be time consuming.