Forum Discussion
SELECT TOP 1000 [$Table] - Causing poor performance for Incremental Load
Hi, PJChip
To avoid using' SELECT TOP 1000 [$Table]....' by default, you can write your own SQL code to get the data you want.
Import data from a database using native database query
Query Folding And Writing Your Own SQL Queries In Power Query/Power BI/Excel Get & Transform
Query folding for native SQL in Power BI
Best Regards,
Community Support Team _ Eason
PJChip did you get any help with this? I am facing the same issue when querying delta tables through Power BI. I have a view in Synapse that is based on three Delta tables. When I give the query a where clause it correctly filters the data from deltatables. However Power BI service sends these top 1000 queries before sending the actual where clause containing query:
For each partition in Power BI Tabular model it seems to first send this Top 1000 query
select top 1000
[$Table].[AccountEntryID] as [AccountEntryID],
[$Table].[ACCOUNTINGPERIOD] as [ACCOUNTINGPERIOD],
[$Table].[ACCOUNTINGYEAR] as [ACCOUNTINGYEAR],
[$Table].[ACCOUNTINGMONTH] as [ACCOUNTINGMONTH],
[$Table].[ACCOUNTINGDATE] as [ACCOUNTINGDATE],
[$Table].[CREATEDDATETIME] as [CREATEDDATETIME],
[$Table].[JOURNALNUMBER] as [JOURNALNUMBER],
[$Table].[SUBLEDGERVOUCHER] as [SUBLEDGERVOUCHER],
[$Table].[POSTINGLAYER] as [POSTINGLAYER],
[$Table].[ACCOUNTINGCURRENCYAMOUNT] as [ACCOUNTINGCURRENCYAMOUNT],
[$Table].[TRANSACTIONCURRENCYCODE] as [TRANSACTIONCURRENCYCODE],
[$Table].[REPORTINGCURRENCYAMOUNT] as [REPORTINGCURRENCYAMOUNT],
[$Table].[POSTINGTYPE] as [POSTINGTYPE],
[$Table].[LEDGERDIMENSION] as [LEDGERDIMENSION],
[$Table].[LEDGERACCOUNT] as [LEDGERACCOUNT],
[$Table].[TEXT] as [TEXT],
[$Table].[ISCREDIT] as [ISCREDIT],
[$Table].[Company] as [Company],
[$Table].[MainAccount_ID] as [MainAccount_ID],
[$Table].[BusinessUnit_ID] as [BusinessUnit_ID],
[$Table].[Department_ID] as [Department_ID],
[$Table].[CostCenter_ID] as [CostCenter_ID],
[$Table].[ItemGroup_ID] as [ItemGroup_ID],
[$Table].[Project_ID] as [Project_ID],
[$Table].[Property_ID] as [Property_ID],
[$Table].[Customer_ID] as [Customer_ID],
[$Table].[ACCOUNTINGDATEKEY] as [ACCOUNTINGDATEKEY]
from [f_GeneralLedger_DELTA] as [$Table]After that it sends the query with where clause:
select top 1000
[_].[AccountEntryID],
[_].[ACCOUNTINGPERIOD],
[_].[ACCOUNTINGYEAR],
[_].[ACCOUNTINGMONTH],
[_].[ACCOUNTINGDATE],
[_].[CREATEDDATETIME],
[_].[JOURNALNUMBER],
[_].[SUBLEDGERVOUCHER],
[_].[POSTINGLAYER],
[_].[ACCOUNTINGCURRENCYAMOUNT],
[_].[TRANSACTIONCURRENCYCODE],
[_].[REPORTINGCURRENCYAMOUNT],
[_].[POSTINGTYPE],
[_].[LEDGERDIMENSION],
[_].[LEDGERACCOUNT],
[_].[TEXT],
[_].[ISCREDIT],
[_].[Company],
[_].[MainAccount_ID],
[_].[BusinessUnit_ID],
[_].[Department_ID],
[_].[CostCenter_ID],
[_].[ItemGroup_ID],
[_].[Project_ID],
[_].[Property_ID],
[_].[Customer_ID],
[_].[ACCOUNTINGDATEKEY]
from [f_GeneralLedger_DELTA] as [_]
where [_].[CREATEDDATETIME] >= convert(datetime2, '2015-01-01 00:00:00') and [_].[CREATEDDATETIME] <= convert(datetime2, '2016-01-01 00:00:00')The top 1000 query forces Delta tables Parquet files to be loaded in full to Synapse on-demand cluster. Where clauses only loads required data from the parquet files. This pretty much destroys one point of using the incremental dataset, which is of course not spending so much resources in Synapse on-demand....