topic Re: incremental Refresh sends weirds range calls in Developer

incremental Refresh sends weirds range calls

LiranRom — Sun, 06 Aug 2023 13:33:26 GMT

I have a connector that first calls the Navigation part and form there it calls the methods fetch data ( I've added 2 parameters RangeStart and RangeEnd ) - to get my data from the service, in this range

this lead to successfull E2E working code.
meaning: I have connector and I set in power BI desktop 2 Parameters RangeStart and RangeEnd and when I call my connecot with those parameters - its put the parameters as GTE and LTE and fetch data accordigly - works great.
Here are my parameters, and calling the connector

when I refresh on desktop I got those parameters on my server: same as the initial values of RangeStart and RangeEnd I set. which is as expected: "gte"=>"2023-01-01", "lte"=>"2024-01-01"

Then I set refresh incremental setting on the model - I wish to archive 2 years and refresh only last day.

then, I've created a report and publish it to Power BI service.

and refresh manually (on demand) the dateset

The weirds was the GTE and LTE values, when I monitor on our server, it makes no scence:
1. "gte"=>"2021-01-01", "lte"=>"2022-01-01"

2. "gte"=>"2023-08-02", "lte"=>"2023-08-03"

but today is August 6 2023 (so why it fetch year 2021? and why 3 days ago?)

can you please try to explain?

Thanks

Re: incremental Refresh sends weirds range calls

LiranRom — Sun, 06 Aug 2023 13:12:47 GMT

@lbendlin - can you please advice?

Re: incremental Refresh sends weirds range calls

lbendlin — Mon, 07 Aug 2023 05:19:32 GMT

You must not use GTE and LTE. Only one of the parameters can be inclusive. Use GTE and LT.

Every time you make meta data changes to your query you will reset the partitions. It takes one more refresh to get back into the desired partition structure. If you don't like that, use ALM Toolkit and XMLA partition refreshes.

Re: incremental Refresh sends weirds range calls

LiranRom — Mon, 07 Aug 2023 08:38:47 GMT

@lbendlin Thanks for your feedbacks, but I don't understand how the mechanism works.

let me clarify something:
I have Power BI Pro plan

in the connector code - we fetch data from server by Web.Contents using RelativePath and Query.

we're using the RangeStart and RangeEnd (from PowerBI) in the Query.

RangeStart -> used as GTE param sent to server

RangeEnd -> used as TLE param sent to server

On the server its allow only GTE and TLE, not GT and TL. Why is it important?

I still not understand how the incremental refresh mechanism work, how its takes the setting I've enter in Power BI Desktop (2 years archive, 1 day before refresh) and using it when I perform refersh on Power BI Service

my expectation was -

first refresh (today = August 8 2023) => gets 2 years ago from toady. and last day -

meaning:

one call with

RangeStart = August 7 2021

RangeEnd = August 8 2023 (today)

one call with

last day =>RangeStart = August 7 2023; RangeEnd = August 8 2023 (today)

next refresh (today) => get only the last day - one call with RangeStart = August 7 2023; RangeEnd = August 8 2023 (today)

basically LTE - is not a must. I can call to server without sending it. and it will calculated it as today. which its ok for all calls.

can you please explain more, so I can understand?

Re: incremental Refresh sends weirds range calls

lbendlin — Mon, 07 Aug 2023 10:45:53 GMT

RangeStart -> used as GTE param sent to server RangeEnd -> used as TLE param sent to server On the server its allow only GTE and TLE, not GT and TL. Why is it important?

If both RangeStart and RangeEnd are inclusive then you risk duplicating data in adjacent partitions.

how its takes the setting I've enter in Power BI Desktop (2 years archive, 1 day before refresh) and using it when I perform refersh on Power BI Service

1. You specified 2 years for archive. That means the Power BI service will create partitions for each calendar year in that range, meaning this can be up to three partitions, not just the two you think you specified.

2. You specified 1 day for refresh. That means a couple of things. First of all, you will see up to 32 daily partitions. Then (because of 1.) you will see that the Power BI Service is consolidating these daily partitions into monthly, quarterly and yearly partitions (all in calendar terms). This is done for you automatically. And secondly if you use an XMLA query tool you can see the last refresh date for each partition so you can confirm that you your newest daily partition is indeed the only one that got touched (except for consolidation days).

Re: incremental Refresh sends weirds range calls

LiranRom — Mon, 07 Aug 2023 13:36:56 GMT

@lbendlin Thanks

Can I investigate the partitions of dataset with Pro license?

Re: incremental Refresh sends weirds range calls

lbendlin — Mon, 07 Aug 2023 13:57:43 GMT

I believe so, but cannot say for sure. Try it via DAX Studio or SSMS.

Re: incremental Refresh sends weirds range calls

LiranRom — Thu, 10 Aug 2023 13:38:48 GMT

I've upgrade my licence to Premium Per User licence, but still can't get into the workspase settings to get the url so I could connect to with external tools like XMLA to see the partitions

what am I missing?

Re: incremental Refresh sends weirds range calls

lbendlin — Tue, 15 Aug 2023 12:24:33 GMT

I don't think that works for "My Workspace". Try with a proper shared workspace.

Re: incremental Refresh sends weirds range calls

LiranRom — Thu, 17 Aug 2023 06:28:07 GMT

you are correct. I needed to create new workspace to have the option to get the URL
Thanks