Microsoft 365 Copilot - API misuse / automation abuse from AI agent

Copilot Api Misuse High Failure Rate

Query

let MinCalls = 200;
let ErrorRateThreshold = 0.30;
CopilotActivity
| where TimeGenerated > ago(1h)
| extend
    ToolName = tostring(LLMEventData.ToolName),
    HttpStatus = toint(LLMEventData.HttpStatus),
    ApiTarget = tostring(LLMEventData.ApiTarget)
| where isnotempty(ToolName) or isnotempty(ApiTarget)
| summarize
    TotalCalls = count(),
    ClientErrors = countif(HttpStatus between (400 .. 499)),
    ServerErrors = countif(HttpStatus between (500 .. 599)),
    Tools = make_set(ToolName, 32),
    ApiTargets = make_set(ApiTarget, 32),
    ClientIPs = make_set(SrcIpAddr, 16),
    FirstSeen = min(TimeGenerated),
    LastSeen = max(TimeGenerated)
    by AgentId, AgentName, ActorName, TenantId
| extend ErrorRate = todouble(ClientErrors + ServerErrors) / todouble(TotalCalls)
| where TotalCalls >= MinCalls and ErrorRate >= ErrorRateThreshold
| extend SrcIpAddr = tostring(ClientIPs[0])

Explanation

This query is designed to detect unusual behavior from a Microsoft 365 Copilot agent, which might indicate misuse or abuse of APIs. Here's a simple breakdown:

Purpose: The query identifies if a Microsoft 365 Copilot agent is making an unusually high number of API or tool calls within a one-hour period, with a significant portion of those calls resulting in errors. This could suggest a problem like a runaway loop, automation abuse, or misuse by a compromised agent.
Criteria:
- The agent must make at least 200 calls within the hour.
- At least 30% of these calls must result in client (4xx) or server (5xx) errors.
Data Source: It uses data from the CopilotActivity data type, provided by the MicrosoftCopilot connector.
Process:
- It checks activities from the last hour.
- It calculates the total number of calls, counts the errors, and identifies the tools and APIs used.
- It computes the error rate as a percentage of total calls.
- It filters for agents that meet the criteria of high call volume and error rate.
Output: If an agent meets these conditions, it triggers an alert with a medium severity level.
Additional Details:
- The query is scheduled to run every hour.
- It maps relevant data to entities like CloudApplication, Account, and IP for further analysis.
- It creates incidents for detected issues, with settings for grouping related alerts.

This setup helps in monitoring and managing potential issues with AI-driven automation in Microsoft 365 environments.

Details

David Alonso

Released: May 20, 2026

Tables

CopilotActivity

Keywords

MicrosoftCopilotApiToolAgentTenantCloudApplicationAccountIP

Operators

letagotostringtointisnotemptysummarizecountcountifbetweenmake_setminmaxbytodoubleextendwhere

Severity

Medium

Tactics

Impact

MITRE Techniques

T1499

Frequency: PT1H

Period: PT1H

Actions

GitHub

KQL Search