Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hi eveyone!
Details of issue: i need to get some data from the CRM based on MS Dynamics 365 through the API (to Power BI or Power Query). The database has millions of rows and I don’t need all of them (anyway these is lack of memory on the PC, when I’m trying to get all the data). There are several problems:
1) A single way to find only rows which I need – IDs of transactions from another table. There are about 500 unique IDs and every ID has from couple of hundred to 25 thousand rows (i know, it's bad approach, but still don't find more sofisticated columns to filter it).
2) For all request I need to use paging (through @odata.nextLink) because most of the data in IDs is over 5000 rows (API limitation).
3) And the main issue – creating of script running looping URL, which iterate over the all data and returns only the ones I need. I'm tried to find solution with this link and this video. So finally, it works when i make checking of invoked function, but it still doesn't work as a final solution and returns error.
How it looks:
- firstly, i created function (Query3cb), which includes pagination:
= (param2 as text) as table =>
let
initReq = Json.Document(Web.Contents("https://site.api.crm4.dynamics.com/",
[RelativePath="api/data/v9.2/some_payments"&(param2)]
)),
nextUrl = initReq[#"@odata.nextLink"],
initValue= initReq[value],
gather=(data as list, url)=>
let
newReq=Json.Document(Web.Contents(url)),
newNextUrl = newReq[#"@odata.nextLink"],
newData= newReq[value],
data=List.Combine({data,newData}),
Converttotable = Record.ToTable(newReq),
Pivot_Columns = Table.Pivot(Converttotable, List.Distinct(Converttotable[Name]), "Name", "Value", List.Count),
Column_Names=Table.ColumnNames(Pivot_Columns),
Contains_Column=List.Contains(Column_Names,"@odata.nextLink"),
check = if Contains_Column = true then @gather(data, newNextUrl) else data
in
check,
Converttotable = Record.ToTable(initReq),
Pivot_Columns = Table.Pivot(Converttotable, List.Distinct(Converttotable[Name]), "Name", "Value", List.Count),
Column_Names=Table.ColumnNames(Pivot_Columns),
Constain_Column=List.Contains(Column_Names,"@odata.nextLink"),
outputList= if Constain_Column= true then @gather(initValue,nextUrl) else initValue,
expand=Table.FromRecords(outputList)
in
expand
- then i have the list of unique IDs and trying to use invoke custom function:
- and as result it returns error:
(i can send all error code if it could help)
- BUT, when i'm trying to apply this function directrly:
everything works fine (invoked function):
So i have no idea, what's wrong. May be error not from my side but from server?
Thanks in advance for any help and tips!
User | Count |
---|---|
73 | |
70 | |
38 | |
25 | |
23 |
User | Count |
---|---|
96 | |
93 | |
50 | |
43 | |
42 |