Background on our capacity: Our company has a P1 capacity with multiple workspaces in the same capacity, so my workspace doesn't recieve 100% of the resources.
I have several dataflows setup. These are very large with dataflows that simply import up to 400M rows of data into Power BI. The first batch was the largest, since they are incremental updates from now now, which is about 10M rows per week. Then I have additional dataflows that transform and filter the data; one of the dataflows filters this data into various timeframes, including 90 days of data, so about a quarter of the size of the dataflow. At first, I was having memory issues with loading this data, so I updated the dataflow settings to Max Memory = 100%, Container Size = 5000mb, Enhanced Computer Engine = On, Computer Engine Memory = 30%. This allowed me to upload the data into dataflows. However, now I'm having issues loading that data into datasets. These datasets hold the quarterly data, so they are about 50M rows in two tables. I am able to refresh these datasets on my local laptop with 16gb memory, but it fails on the P1 Server. I can't even import that 7GB file, since I receive a memory error doing that as well.
My question is, what are the recommended dataset and dataflow settings for a P1 capacity to get this to refresh? I'm thinking about reducing the dataflow max memory to 50% and 3.5GB container size, since the capacity monitor dashboard shows the max memory from a dataflow at 3.13GB.
