Forum Discussion
Consuming XML web service
- 1 year ago
Finally, I solved it by generating a Power Query loop to fetch all pages automatically, and later adding a counter for consulting only the last page and finally combining them. Here is the complete code:
let BaseUrl = "https://xxxxxxxxxxxxxxxxxx", // Función para obtener la página de datos GetPage = (pageNumber) => let // Realiza la solicitud con los parámetros adecuados Response = Web.Contents(BaseUrl, [Query=[page=Number.ToText(pageNumber)]]), // Convierte la respuesta XML en una tabla Data = Xml.Tables(Response), // La primera tabla contiene los parámetros de la consulta (paginación y demás) RequestInfo = Data{0}[Table], ExpandedFilter = Table.ExpandTableColumn(RequestInfo, "filter", {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}, {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}), // La segunda tabla contiene los datos DataTable = Data{1}[Table], Table1 = DataTable[Table], // Expande la tabla de datos ExpandedDataTable = Table.SelectColumns(ExpandedFilter, {"page", "more_pages"}), // Devuelve los datos junto con la información de la paginación PageData = [Data = DataTable, Pagination = ExpandedDataTable] in PageData, // Generar la lista de todas las páginas de datos AllData = List.Generate(// Estado inicial con la primera página () => [Page = Number.FromText(GetPage(1)[Pagination][page]{0}), Data = GetPage(1)[Data], MorePages = GetPage(1)[Pagination][more_pages]{0}], // Condición de continuidad: continuar mientras MorePages sea 1 (hay más páginas) each Number.FromText([MorePages]) = 1, each // Actualización del estado: aumentar la página y obtener los datos correspondientes let NextPage = [Page] + 1, PageData = GetPage(NextPage), NewMorePages = PageData[Pagination][more_pages]{0} in [Page = NextPage, Data = PageData[Data], MorePages = NewMorePages], // Extraer los datos de cada página each [Data] ), // Para capturar el núnmero de páginas, execptuando la última Contador = List.Count(AllData), // Combinar las tablas de datos de cada página en una tabla ExpandedData = Table.Combine(AllData), // Combinar y Expandir todas las tablas de registros Table1 = Table.Combine(ExpandedData[Table]), //Utilizo el contador para acceder a la última página LastPage = [Data = GetPage(Contador + 1)[Data], MorePages = GetPage(Contador + 1)[Pagination][more_pages]{0}], AllData2 = LastPage[Data], Table2 = AllData2{0}[Table], // Combinar las tablas de la primera parte con la tabla de la ultima página CombinedData = Table.Combine({Table1, Table2}), in CombinedData
Hi YCastano,
Could you please confirm if this issue has been resolved? If it has, kindly mark the helpful reply and "Accept as Solution". This will assist other community members in resolving similar problems more quickly.
Thank you,
Pavan.
Hi @v-pbandela-msft thank you.
I think it doesn't work, it seems that it stays in loop because it stays thinking and does not generate the list of tables, or gives me List error. I had also tried to put the (1 or 0) and the same thing happened, I think it stays in a loop.
- Anonymous1 year agoNot applicable
Hi YCastano,
Thank you for your patience. I understand that the query seems to be stuck in an infinite loop. This may be happening due to how MorePages is being checked within List.Generate.
Try the following adjustments to prevent the loop:
1. Instead of checking MorePages = 1, use MorePages <> 0 to ensure it stops when there are no more pages.
2. Use try ... otherwise 0 to prevent errors when accessing more_pages.
here is the updated Codelet
BaseUrl = "https://xxxxxxxxxx",// Function to retrieve a page of data
GetPage = (pageNumber) =>
let
Response = Web.Contents(BaseUrl, [Query=[page=Number.ToText(pageNumber), max_page_size="1000"]]),
Data = Xml.Tables(Response),
RequestInfo = Data{0}[Table],
ExpandedFilter = Table.ExpandTableColumn(RequestInfo, "filter",
{"effective_date", "changed_on_or_after", "changed_before", "change_type",
"deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"},
{"effective_date", "changed_on_or_after", "changed_before", "change_type",
"deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}
),
DataTable = Data{1}[Table],
ExpandedDataTable = Table.SelectColumns(ExpandedFilter, {"page", "max_page_size", "actual_page_size", "more_pages"}),
PageData = [Data = DataTable, Pagination = ExpandedDataTable]
in
PageData,// Generate a list of all pages of data
AllData = List.Generate(
() =>
let
FirstPageData = GetPage(1)
in
[Page = 1, Data = FirstPageData[Data], MorePages = try FirstPageData[Pagination][more_pages]{0} otherwise 0],
// **Stopping condition: Continue only if MorePages is not 0**
each [MorePages] <> 0,
each
let
NextPage = [Page] + 1,
PageData = GetPage(NextPage),
NewMorePages = try PageData[Pagination][more_pages]{0} otherwise 0
in
[Page = NextPage, Data = PageData[Data], MorePages = NewMorePages],
each [Data] // Extract data from each page
),// Combine data from all pages
ExpandedData = Table.Combine(AllData),
CombinedData = Table.Combine(ExpandedData[Table]),// Remove empty records
Results = Table.SelectRows(CombinedData, each ([unidadorg] <> ""))
in
ResultsPlease continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.- YCastano1 year agoFrequent Visitor
Hi Anonymous, With the value [MorePages] <> 0, I still have the last page not included, since the last page has the value [MorePages] = 0.