topic Re: Incremental Refresh Azure Blob in Power Query

Incremental Refresh Azure Blob

vitoriamariabr — Tue, 12 Mar 2024 20:56:42 GMT

Hi, I need help implementing incremental refresh on Azure Blob databases. I've already implemented it the standard way, but the incremental refresh takes forever and sometimes it doesn't even update, I don't think it's working correctly. I implemented it the basic way, creating the parameters and filtering RangeStart and RangeEnd in my date column (in azure blob the date column I used was Date modified) and I also tested it by inserting the incremental refresh filtering directly in the source:

"let
startDate = RangeStart,
endDate = RangeEnd,
Fonte =Table.SelectRows(AzureStorage.Blobs("https://xxxxxxxxx.blob.core.windows.net/xxxxx"),each [Date modified] >= startDate and [Date modified] <= endDate)".

However, the incremental update still doesn't work correctly. Can anyone help me?

olá! Preciso de ajuda para implementar atualização incremental em bases do Azure Blob. Já implementei da forma padrão, porém a atualização incremental demora uma eternidade e às vezes nem chega a atualizar, não acho que esteja funcionando corretamente. Cheguei a implementar da maneira básica, criando os parâmetros e filtrando RangeStart e RangeEnd na minha coluna de data (no azure blob a coluna de data que usei foi o Date modified) e testei também inserindo a filtragem da atualização incremental direto na fonte:

Porém, ainda assim a atualização incremental não funciona corretamente.

Alguém saberia me ajudar?

Re: Incremental Refresh Azure Blob

lbendlin — Tue, 12 Mar 2024 21:30:50 GMT

RangeStart and RangeEnd cannot be both inclusive. Usually RangeStart is inclusive and RangeEnd is exclusive.

You cannot use Date columns, it must be DateTime or DateInteger.

Re: Incremental Refresh Azure Blob

vitoriamariabr — Wed, 13 Mar 2024 11:24:01 GMT

I see. My column is a DateTime. I'll adjust the RangeStart and RangeEnd, but I don't think that's the problem.

Re: Incremental Refresh Azure Blob

lbendlin — Wed, 13 Mar 2024 12:11:02 GMT

What is your reasoning for doing this in the first place?

Re: Incremental Refresh Azure Blob

vitoriamariabr — Wed, 13 Mar 2024 12:20:59 GMT

I have historical data in azure blob that is immutable, so I don't need to update it every time I need to update the report. That's why I want to implement incremental refresh, taking only the last 7 days, for example. But the big difficulty here is structuring the incremental refresh in the best way, and I'm not succeeding. I believe that the Azure Blob bases don't support query folding, and this may be considerably impacting the incremental update and I don't know what to do about it 😥😭

Re: Incremental Refresh Azure Blob

lbendlin — Wed, 13 Mar 2024 13:11:07 GMT

I think it is a design conflict. Azure Blobs are not supposed to be re-stored somewhere else.

Do you have a performance issue during regular refresh?