Forum Discussion

Ivancito112's avatar
Ivancito112
Regular Visitor
1 year ago
Solved

Avoid LIMIT 1000 check per update

Dear Friends,

I need to remove the LIMIT 1000 Power BI check when updating the report.

It happens that a couple of months ago I discovered that by using a view in BigQuery I could avoid the LIMIT 1000 check, this worked for more than 3 months without problems, but a week ago the same report that didn't do the LIMIT 1000 now does it.

This change was done with incremental refresh where I set 5 days of update, and as I said months ago I was running the query 5 times, one per day, per update, but now the check was added.

 

Query with limit 1000:

select `hostname`,
    `ep_page_location`,
    `Mercado`,
    cast(`event_date` as TIMESTAMP) as `C1`,
    `Navegador`,
    `categoria_dispositivo`,
    `subcategoria_performance`,
    `channel_group`,
    `campana`,
    `fuente_medio`,
    `mercado_campana_performance`,
    `plataforma_performance`,
    `nivel_1`,
    `nivel_2`,
    `nivel_3`,
    `nivel_4`,
    `medium_fixed`,
    `source_fixed`,
    `sesiones`,
    `usuarios`,
    `Page_view`,
    `sesiones_EsPagina_Salida`,
    `sesiones_EsPagina_Entrada`,
    `sesiones_NoPagina_Salida_Entrada`,
    `usuarios_EsPagina_Salida`,
    `usuarios_EsPagina_Entrada`,
    `usuarios_NoPagina_Salida_Entrada`,
    `Page_view_EsPagina_Salida`,
    `Page_view_EsPagina_Entrada`,
    `Page_view_NoPagina_Salida_Entrada`
from `sandbox-bigquery-345619`.`DWH_CORPORATIVO`.`v_reporte_trafico_general`
LIMIT 1000 OFFSET 0

 

Query with incremental refresh:

select `hostname`,
    `ep_page_location`,
    `Mercado`,
    `C1`,
    `Navegador`,
    `categoria_dispositivo`,
    `subcategoria_performance`,
    `channel_group`,
    `campana`,
    `fuente_medio`,
    `mercado_campana_performance`,
    `plataforma_performance`,
    `nivel_1`,
    `nivel_2`,
    `nivel_3`,
    `nivel_4`,
    `medium_fixed`,
    `source_fixed`,
    `sesiones`,
    `usuarios`,
    `Page_view`,
    `sesiones_EsPagina_Salida`,
    `sesiones_EsPagina_Entrada`,
    `sesiones_NoPagina_Salida_Entrada`,
    `usuarios_EsPagina_Salida`,
    `usuarios_EsPagina_Entrada`,
    `usuarios_NoPagina_Salida_Entrada`,
    `Page_view_EsPagina_Salida`,
    `Page_view_EsPagina_Entrada`,
    `Page_view_NoPagina_Salida_Entrada`
from 
(
    select `hostname`,
        `ep_page_location`,
        `Mercado`,
        `Navegador`,
        `categoria_dispositivo`,
        `subcategoria_performance`,
        `channel_group`,
        `campana`,
        `fuente_medio`,
        `mercado_campana_performance`,
        `plataforma_performance`,
        `nivel_1`,
        `nivel_2`,
        `nivel_3`,
        `nivel_4`,
        `medium_fixed`,
        `source_fixed`,
        `sesiones`,
        `usuarios`,
        `Page_view`,
        `sesiones_EsPagina_Salida`,
        `sesiones_EsPagina_Entrada`,
        `sesiones_NoPagina_Salida_Entrada`,
        `usuarios_EsPagina_Salida`,
        `usuarios_EsPagina_Entrada`,
        `usuarios_NoPagina_Salida_Entrada`,
        `Page_view_EsPagina_Salida`,
        `Page_view_EsPagina_Entrada`,
        `Page_view_NoPagina_Salida_Entrada`,
        cast(`event_date` as TIMESTAMP) as `C1`
    from `sandbox-bigquery-345619`.`DWH_CORPORATIVO`.`v_reporte_trafico_general`
) as `ITBL`
where `C1` >= TIMESTAMP('2025-01-05 00:00:00') and `C1` < TIMESTAMP('2025-01-06 00:00:00')

 

I did a lot of tests during these days to understand and I found out that a colleague of mine with exactly the same report, exactly the same permissions in BigQuery (from what the administrator told me) does not do the LIMIT 1000 check.

I have already read documentation and forums looking for an option about this, without success.

 

Please, any information or help would be useful.

 

I remain attentive to your expert comments.

Have a nice week.

  • To address the LIMIT 1000 issue in Power BI when querying BigQuery:

    1. Check Power BI Settings: Ensure you’re not using DirectQuery or a limited preview mode, as these can add implicit LIMIT clauses. Use Import Mode or adjust DirectQuery settings to avoid query truncation.

    2. Incremental Refresh: Confirm the incremental refresh configuration is consistent with your colleague's setup. Differences in filters or partitions can cause unexpected behavior.

    3. BigQuery Connector Version: Update the BigQuery connector in Power BI to the latest version to ensure compatibility and avoid unexpected limitations.

    4. View Configuration: Verify the BigQuery view (v_reporte_trafico_general) doesn’t have a hardcoded LIMIT and matches your colleague's version.

    5. Permissions and Caching: Even with identical permissions, query caching or metadata differences could lead to variations. Clear Power BI and BigQuery caches to test.

     

2 Replies

  • To address the LIMIT 1000 issue in Power BI when querying BigQuery:

    1. Check Power BI Settings: Ensure you’re not using DirectQuery or a limited preview mode, as these can add implicit LIMIT clauses. Use Import Mode or adjust DirectQuery settings to avoid query truncation.

    2. Incremental Refresh: Confirm the incremental refresh configuration is consistent with your colleague's setup. Differences in filters or partitions can cause unexpected behavior.

    3. BigQuery Connector Version: Update the BigQuery connector in Power BI to the latest version to ensure compatibility and avoid unexpected limitations.

    4. View Configuration: Verify the BigQuery view (v_reporte_trafico_general) doesn’t have a hardcoded LIMIT and matches your colleague's version.

    5. Permissions and Caching: Even with identical permissions, query caching or metadata differences could lead to variations. Clear Power BI and BigQuery caches to test.

     

  • Hi Ivancito112 ,

     

    The issue with the LIMIT 1000 being applied to your Power BI query likely stems from either Power BI's internal query generation processes or changes in your configuration or BigQuery settings. Power BI often applies a LIMIT clause during query previews in the Query Editor to optimize performance when loading data for editing, but this behavior does not usually impact the full refresh unless additional query transformations inadvertently include it. In the case of incremental refresh, Power BI adds filters to segment data into partitions. If these configurations are not set up correctly, the system might unintentionally apply a query limitation, like a LIMIT clause.

    It's also possible that differences between your Power BI setup and your colleague's—such as the connector version, Query Editor steps, or even Power BI Desktop versions—could be contributing to this discrepancy. The issue might also arise from changes on the BigQuery side, such as project-level settings or permissions that enforce query constraints. Reviewing the job history in BigQuery can help identify any such changes or discrepancies.

    To resolve this, it’s important to verify that no LIMIT clause exists in the Query Editor steps, particularly in the Source step, and ensure the incremental refresh settings (like RangeStart and RangeEnd) are properly configured. Running your query directly in BigQuery's console can also help confirm whether the LIMIT is being applied by Power BI or by BigQuery itself. Comparing your Power BI setup with your colleague's, including the applied transformations, incremental refresh configurations, and credentials, can reveal potential differences causing the issue. If necessary, you can use a custom SQL query in Power BI to bypass Power BI's query building and directly execute the intended query. Ensuring that Power BI Desktop and the BigQuery connector are updated to their latest versions is also a good practice to avoid any inconsistencies caused by outdated software.

    If the problem persists, contacting your BigQuery administrator to confirm there are no new constraints or permissions affecting your account could provide additional insights. In the long term, using a custom SQL query or testing DirectQuery mode in Power BI may offer a more stable solution to avoid similar issues in the future. These steps should help address the problem and prevent further limitations from being applied unexpectedly.

     

    Best regards,