Forum Discussion
Consuming XML web service
- 1 year ago
Finally, I solved it by generating a Power Query loop to fetch all pages automatically, and later adding a counter for consulting only the last page and finally combining them. Here is the complete code:
let BaseUrl = "https://xxxxxxxxxxxxxxxxxx", // Función para obtener la página de datos GetPage = (pageNumber) => let // Realiza la solicitud con los parámetros adecuados Response = Web.Contents(BaseUrl, [Query=[page=Number.ToText(pageNumber)]]), // Convierte la respuesta XML en una tabla Data = Xml.Tables(Response), // La primera tabla contiene los parámetros de la consulta (paginación y demás) RequestInfo = Data{0}[Table], ExpandedFilter = Table.ExpandTableColumn(RequestInfo, "filter", {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}, {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}), // La segunda tabla contiene los datos DataTable = Data{1}[Table], Table1 = DataTable[Table], // Expande la tabla de datos ExpandedDataTable = Table.SelectColumns(ExpandedFilter, {"page", "more_pages"}), // Devuelve los datos junto con la información de la paginación PageData = [Data = DataTable, Pagination = ExpandedDataTable] in PageData, // Generar la lista de todas las páginas de datos AllData = List.Generate(// Estado inicial con la primera página () => [Page = Number.FromText(GetPage(1)[Pagination][page]{0}), Data = GetPage(1)[Data], MorePages = GetPage(1)[Pagination][more_pages]{0}], // Condición de continuidad: continuar mientras MorePages sea 1 (hay más páginas) each Number.FromText([MorePages]) = 1, each // Actualización del estado: aumentar la página y obtener los datos correspondientes let NextPage = [Page] + 1, PageData = GetPage(NextPage), NewMorePages = PageData[Pagination][more_pages]{0} in [Page = NextPage, Data = PageData[Data], MorePages = NewMorePages], // Extraer los datos de cada página each [Data] ), // Para capturar el núnmero de páginas, execptuando la última Contador = List.Count(AllData), // Combinar las tablas de datos de cada página en una tabla ExpandedData = Table.Combine(AllData), // Combinar y Expandir todas las tablas de registros Table1 = Table.Combine(ExpandedData[Table]), //Utilizo el contador para acceder a la última página LastPage = [Data = GetPage(Contador + 1)[Data], MorePages = GetPage(Contador + 1)[Pagination][more_pages]{0}], AllData2 = LastPage[Data], Table2 = AllData2{0}[Table], // Combinar las tablas de la primera parte con la tabla de la ultima página CombinedData = Table.Combine({Table1, Table2}), in CombinedData
Hi YCastano,
Thank you for reaching out in Microsoft Community Forum.
Thank you Fowmy for the helpful response.
Please Modify the List.Generate function so that the loop includes the last page even when more_pages = 0.
Here is the modified M code;
let
BaseUrl = "https://xxxxxxxxxx",
// Function to retrieve a page of data
GetPage = (pageNumber) =>
let
Response = Web.Contents(BaseUrl, [Query=[page=Number.ToText(pageNumber), max_page_size="1000"]]),
Data = Xml.Tables(Response),
RequestInfo = Data{0}[Table],
ExpandedFilter = Table.ExpandTableColumn(RequestInfo, "filter", {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}, {"effective_date", "changed_on_or_after", "changed_before", "change_type", "deleted", "only_deleted", "page", "max_page_size", "actual_page_size", "more_pages"}),
DataTable = Data{1}[Table],
Table1 = DataTable[Table],
ExpandedDataTable = Table.SelectColumns(ExpandedFilter, {"page", "max_page_size", "actual_page_size", "more_pages"}),
PageData = [Data = DataTable, Pagination = ExpandedDataTable]
in
PageData,
// Generate a list of all pages of data
AllData = List.Generate(
() =>
let
FirstPageData = GetPage(1)
in
[Page = 1, Data = FirstPageData[Data], MorePages = FirstPageData[Pagination][more_pages]{0}],
// **Fix: Change stopping condition to ensure last page is included**
each [MorePages] <> null, // Continue as long as we retrieve data
each
let
NextPage = [Page] + 1,
PageData = GetPage(NextPage),
NewMorePages = PageData[Pagination][more_pages]{0}
in
[Page = NextPage, Data = PageData[Data], MorePages = NewMorePages],
each [Data] // Extract data from each page
),
// Combine data from all pages
ExpandedData = Table.Combine(AllData),
CombinedData = Table.Combine(ExpandedData[Table]),
// Remove empty records
Results = Table.SelectRows(CombinedData, each ([unidadorg] <> ""))
in
Results
Modified the stopping condition in List.Generate to continue looping as long as data is retrieved, ensuring the last page is included.
Please continue using Microsoft community forum.
If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.
Regards,
Pavan.
Hi Anonymous thank you.
I think it doesn't work, it seems that it stays in loop because it stays thinking and does not generate the list of tables. I had also tried to put the (1 or 0) and the same thing happened, I think it stays in a loop.