Foundry - Automated red-team / fuzzing pacing

Foundry Red Team Pacing Anomaly

Query

let window = 1h;
let burstWindow = 10m;
let base =
    AppDependencies
    | where TimeGenerated > ago(window)
    | where isnotempty(Properties["gen_ai.agent.name"])
    | extend
        Agent  = tostring(Properties["gen_ai.agent.name"]),
        ConvId = tostring(Properties["gen_ai.conversation.id"]),
        FilterRaw = tostring(Properties["microsoft.foundry.content_filter.results"])
    | extend FilterArr = todynamic(FilterRaw)
    | extend
        Cfr0 = todynamic(FilterArr[0].content_filter_results),
        Cfr1 = todynamic(FilterArr[1].content_filter_results);
let convFanOut =
    base
    | summarize Convs = dcount(ConvId)
        by Agent, Bucket = bin(TimeGenerated, burstWindow)
    | where Convs >= 10
    | extend Signal = "ConversationFanOut", Value = todouble(Convs);
let guardrailHammer =
    base
    | extend Tripped = isnotempty(FilterRaw) and (
          tobool(Cfr0.jailbreak.detected) or tobool(Cfr0.jailbreak.filtered)
          or tobool(Cfr1.jailbreak.detected) or tobool(Cfr1.jailbreak.filtered)
          or tobool(Cfr0.prompt_shield.detected) or tobool(Cfr0.prompt_shield.filtered)
          or tobool(Cfr1.prompt_shield.detected) or tobool(Cfr1.prompt_shield.filtered)
          or tobool(Cfr0.indirect_attack.detected) or tobool(Cfr0.indirect_attack.filtered)
          or tobool(Cfr1.indirect_attack.detected) or tobool(Cfr1.indirect_attack.filtered))
    | summarize Trips = countif(Tripped)
        by Agent, Bucket = bin(TimeGenerated, window)
    | where Trips >= 5
    | extend Signal = "GuardrailHammer", Value = todouble(Trips);
let sustainedRate =
    base
    | summarize SpansPerMin = count()
        by Agent, Minute = bin(TimeGenerated, 1m)
    | where SpansPerMin >= 30
    | summarize HighRateMinutes = count(), MaxRate = max(SpansPerMin)
        by Agent
    | where HighRateMinutes >= 5
    | extend Signal = "SustainedHighRate", Value = todouble(MaxRate), Bucket = bin(now(), 1m);
union convFanOut, guardrailHammer, sustainedRate
| summarize
    Signals       = make_set(Signal, 4),
    SignalDetails = make_bag(pack(Signal, Value)),
    FirstSeen     = min(Bucket),
    LastSeen      = max(Bucket)
    by Agent
| extend SignalCount = array_length(Signals)
| extend Confidence = iff(SignalCount >= 2, "High", "Medium")
| extend AccountName = iff(isempty(Agent), "unknown-agent", Agent)
| project LastSeen, AccountName, Agent, Signals, SignalDetails, SignalCount, Confidence, FirstSeen
| order by SignalCount desc, LastSeen desc

Explanation

This query is designed to detect unusual patterns of activity that may indicate automated abuse of a system using AI agents. It focuses on identifying the "pacing fingerprint" left by automated tools used for red-teaming or fuzzing, which are techniques for testing the security of systems.

Here's a simple breakdown of what the query does:

Data Source: It analyzes data from an application monitoring system (Application Insights), specifically looking at dependencies and certain properties related to AI agents.
Time Frame: The query examines data within a one-hour window.
Detection Criteria: It uses three main signals to detect potential automated abuse:
- ConversationFanOut: Detects if an agent opens 10 or more distinct conversation IDs within a 10-minute period, suggesting parallel operations.
- GuardrailHammer: Checks if an agent triggers 5 or more content-filter alerts (like jailbreak or prompt-shield trips) within an hour.
- SustainedHighRate: Identifies if an agent maintains a high rate of activity (30 or more actions per minute) for at least 5 minutes.
Alert Generation: If any one of these signals is detected, an alert is raised. If two or more signals are detected, it strongly indicates automated abuse.
Output: The query outputs details about the detected signals, including the agent involved, the number of signals detected, and the confidence level of the detection (medium or high).
Incident Management: The query is configured to create incidents in a security monitoring system, grouping alerts by the agent involved.

Overall, this query helps identify and alert on suspicious automated activities that deviate from normal usage patterns, potentially indicating security threats.