Wilsonjr's avatar
Wilsonjr
Frequent Visitor
3 years ago
Status:
Investigating

Azure Data Explorer Set notruncation not working.

I'm working on creating a dataflow that pulls  data from our Azure Log Analytics using the Azure Data Explorer connection. I have set notrunction written in the query, and it will only pull data until it hits the 500,000 row limit. 

That said, the article says that set notruncation is supposed to allow all the records to pull, in this case it should be over 1 million rows being pulled if not more.

Copy and transform data in Azure Data Explorer - Azure Data Factory & Azure Synapse | Microsoft Learn

 

Example of query for one month, Note hostname is subbed with actual name instead of ******

 

 

set notruncation;
Syslog
| where HostName == "*******" and SeverityLevel == "info" and EventTime >=ago(30d)
| extend url = extract("GET\\s.*HTTP", 0, SyslogMessage)
| extend client = extract("/Client:\\w+", 0, url)
| extend IPCountry = extract("IPCountry: \\w+", 0, SyslogMessage)
| extend isLoggedOut = extract("isLoggedOut=\\w", 0, SyslogMessage)
| extend IpAddress = extract("(([0-9]{1,3})\\.([0-9]{1,3})\\.([0-9]{1,3})\\.(([0-9]{1,3})))", 1, SyslogMessage)
| project
    url,
    client,
    IPCountry,
    isLoggedOut,
    IpAddress,
    EventTime;

 

 

Let me know if you need any more information,

Thanks,

James

 

Data Modeling

3 Comments

  • Anonymous's avatar
    Anonymous
    Not applicable
    3 years ago

    Hi Wilsonjr 

    By default, Azure Data Explorer source has a size limit of 500,000 records or 64 MB . So what is your question? Or what kind of results do you want to get ?

     

    Best Regards,
    Community Support Team _ Ailsa Tao

  • Wilsonjr's avatar
    Wilsonjr
    Frequent Visitor
    3 years ago

    Anonymous , yes by default, but if you use the set notruncation before the query it should allow more then 500,000 rows. But when I set it, it does not bring in more then 500,000 rows. Is there some setting I am missing that is capping it at 500,000?

Completed Ideas