INCIDENT REPORT // IR-2026-001

CPU-Agent-01
Bulk Work Item Modification

After installing Phase 3, the agent autonomously added hundreds of claim tags and failure comments across Azure DevOps work items in multiple projects. This document covers the root cause analysis, remediation scripts, and prevention controls.

Severity: MediumStatus: Active RemediationAffected: csc-ddsb / EDCS

WHAT HAPPENED

Incident Summary

100s

of claim tags added to work items across all projects

Every WI

processed received a failure comment attributed to the PAT owner

60s

polling interval — contamination accumulated rapidly across all active items

The agent CPU-Agent-01 was configured with a PAT token issued under the personal account Minasyan, Levon (He Him) (MPBSDP). The default WIQL query contained no project filter, causing the agent to retrieve and attempt to process every active Task and User Story across all projects visible to the PAT.

The concurrency mechanism in WorkItemCoordinator writes a claim tag to System.Tags in the format agent:CPU-Agent-01:<timestamp>:<expiry> for every work item it attempts to process. A separate bug in the workflow condition matching caused the Code Review Workflow to run on every item regardless of tags, and a null template variable caused every workflow execution to fail with the error "No fields specified for update" — which was written back as a comment on each work item.

ROOT CAUSE ANALYSIS

Four Contributing Causes

RC-01CRITICAL

Overly Broad WIQL Query

The default WorkItemQueryWiql contained no project filter and no type filter. When executed against the Azure DevOps REST API, it returned active work items from all projects visible to the PAT — not just the intended EDCS project.

-- BROKEN: No project scope — returns ALL active items across ALL projects
SELECT [System.Id], [System.Title], [System.State]
FROM WorkItems
WHERE [System.State] = 'Active'
ORDER BY [System.CreatedDate] ASC

-- FIXED: Scoped to single project + opt-in tag required
SELECT [System.Id], [System.Title], [System.State]
FROM WorkItems
WHERE [System.TeamProject] = @project
  AND [System.State] = 'Active'
  AND [System.Tags] CONTAINS 'CPU-Agent-Eligible'
ORDER BY [System.CreatedDate] ASC

RC-02HIGH

Workflow Condition Tag Matching Bug

The Code Review Workflow condition "System.Tags": "CodeReview" used exact string equality. Azure DevOps stores tags as semicolon-separated strings (e.g., "Bug; Sprint-12; CodeReview"), so the condition never matched — causing the engine to fall back to the default workflow and run on every Task and User Story.

// BROKEN: Exact match fails for semicolon-separated tag strings
if (!string.Equals(actualValue, expectedValue, StringComparison.OrdinalIgnoreCase))
    return false;

// FIXED: Contains check for System.Tags field
if (fieldName.Equals("System.Tags", StringComparison.OrdinalIgnoreCase))
{
    var tagList = actualValue?.Split(';').Select(t => t.Trim())
                  ?? Enumerable.Empty<string>();
    if (!tagList.Contains(expectedValue, StringComparer.OrdinalIgnoreCase))
        return false;
}
else
{
    if (!string.Equals(actualValue, expectedValue, StringComparison.OrdinalIgnoreCase))
        return false;
}

RC-03HIGH

agentName Template Variable Not Populated

The {{agentName}} template variable in workflow step parameters resolved to null because WorkflowContext was not pre-populated with agent metadata. This caused Step 2 (Update Work Item State) to submit an empty fields dictionary, triggering the Azure DevOps API error: "No fields specified for update".

// FIXED: Populate agent metadata in WorkflowContext before execution
var context = new WorkflowContext
{
    WorkItemId = workItem.Id,
    Variables = new Dictionary<string, string>
    {
        ["agentName"] = _agentConfig.Name,   // "CPU-Agent-01"
        ["agentId"]   = _agentConfig.Name,
        ["project"]   = _azureDevOpsConfig.ProjectName
    }
};

RC-04MEDIUM

Personal Account PAT Used for Automation

The PAT was issued under the personal account Minasyan, Levon (He Him) (MPBSDP). All automated writes appeared in the Azure DevOps UI as changes made by that person — indistinguishable from manual edits. A dedicated service account must be used for all agent operations.

// Required: Create a dedicated service account
// [email protected]
// PAT scopes (minimum required):
//   - Work Items: Read & Write
//   - Code: Read
//   - Test Management: Read & Write
//   - Project and Team: Read
// PAT expiry: 90 days maximum
// Rotation: Automated via PATRotationService

IMPACT ASSESSMENT

Identify Affected Work Items

Run the following WIQL query in Azure DevOps (Queries UI or REST API) to find all work items that received agent claim tags. This gives you the full blast radius before running any cleanup scripts.

WIQL — Find All Contaminated Work Items

SELECT [System.Id], [System.Title], [System.Tags], [System.ChangedDate], [System.ChangedBy]
FROM WorkItems
WHERE [System.Tags] CONTAINS 'agent:CPU-Agent-01'
ORDER BY [System.ChangedDate] DESC

PowerShell — Query Azure DevOps Audit Log

$org = "csc-ddsb"
$pat = "<your-pat>"
$headers = @{
    Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat")))"
}

$from = (Get-Date).AddDays(-7).ToString("yyyy-MM-ddTHH:mm:ssZ")
$url = "https://auditservice.dev.azure.com/$org/_apis/audit/auditlog?startTime=$from&api-version=7.1-preview.1"
Invoke-RestMethod -Uri $url -Headers $headers |
  ConvertTo-Json -Depth 5 |
  Select-String "CPU-Agent|WorkItemUpdate"

REMEDIATION

Step-by-Step Cleanup

Stop the Agent Process

P0 — Immediate

Stop-Service -Name 'CPU-Agent-01' -Force
# Or: Get-Process -Name 'Phase3.AgentHost' | Stop-Process -Force

Revoke the Personal PAT

P0 — Immediate

# Azure DevOps → User Settings → Personal Access Tokens → Revoke
# Issue a new PAT under a dedicated service account

Run Combined Remediation Script (Dry Run First)

P1 — Today

.\Remediate-AgentWorkItems.ps1 -Pat 'your-pat' -DryRun
# Review output — confirms tags and comments targeted before live run
.\Remediate-AgentWorkItems.ps1 -Pat 'your-pat'

Delete Marked Comments (Remove-AgentComments.ps1)

P1 — After Step 4

.\Remove-AgentComments.ps1 -Pat 'your-pat' -DryRun
# Review output — only comments containing 'please disregard this comment' are targeted
.\Remove-AgentComments.ps1 -Pat 'your-pat'

Apply Code Fixes and Restart with Dry-Run Mode

P1 — Before Restart

# Fix WIQL query, tag matching, and agentName population
# Set DryRunMode: true in appsettings.json
# Set MaxWorkItemsPerRun: 10
dotnet run --project Phase3.AgentHost

Tag Script Safety Guarantee

The tag cleanup script uses a prefix filter: it only removes tags matching ^agent:CPU-Agent-01:. Tags such as Bug, Sprint-12, CodeReview, or any other human-authored tag are never touched. The dry-run output shows the exact before/after tag string for every work item — review it before running live.

Two-Pass Comment Remediation

Pass 1 — Remediate (Remediate-AgentWorkItems.ps1): Removes agent claim tags and marks each agent comment in-place, prepending please disregard this comment to the text. The script targets only comments whose text contains the fingerprint [CPU Agent — it will never modify comments written by humans. After this pass, a human reviewer can inspect the marked comments in Azure DevOps and remove the phrase from any comment they wish to preserve before proceeding to Pass 2.

Pass 2 — Delete (Remove-AgentComments.ps1): Permanently deletes every comment still containing please disregard this comment. Any comment from which a human removed the phrase is automatically skipped.

Quick Start Launcher (Run-Remediation.ps1)

Start here. This single launcher script presents an interactive menu, prompts for your PAT and optional project filter, and executes the correct remediation script with the right parameters. It includes an execution policy bypass so you can run it directly without changing system policy. Download all three .ps1 files to the same folder, then right-click Run-Remediation.ps1 and select Run with PowerShell, or launch from a terminal:

powershell -ExecutionPolicy Bypass -File .\Run-Remediation.ps1

Run-Remediation.ps1

# Run-Remediation.ps1 -- Interactive launcher for CPU-Agent-01 incident remediation
# Usage: powershell -ExecutionPolicy Bypass -File .Run-Remediation.ps1
# Requires: Remediate-AgentWorkItems.ps1 and Remove-AgentComments.ps1 in the same folder.

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Definition

function Show-Banner {
    Write-Host ""
    Write-Host "  ========================================================" -ForegroundColor Cyan
    Write-Host "  CPU-Agent-01 Incident Remediation Launcher" -ForegroundColor Cyan
    Write-Host "  AUTONOMOUS.ML // IR-2026-001" -ForegroundColor DarkCyan
    Write-Host "  ========================================================" -ForegroundColor Cyan
    Write-Host ""
}

function Show-Menu {
    Write-Host "  Select an operation:" -ForegroundColor White
    Write-Host ""
    Write-Host "    [1] Remediate -- DRY RUN   (tags + comments, preview only)" -ForegroundColor Yellow
    Write-Host "    [2] Remediate -- LIVE       (tags + comments, apply changes)" -ForegroundColor Red
    Write-Host "    [3] Remove Comments -- DRY RUN   (delete marked comments, preview only)" -ForegroundColor Yellow
    Write-Host "    [4] Remove Comments -- LIVE       (delete marked comments, apply changes)" -ForegroundColor Red
    Write-Host "    [Q] Quit" -ForegroundColor Gray
    Write-Host ""
}

Show-Banner
Show-Menu

$choice = Read-Host "  Enter choice (1-4 or Q)"
if ($choice -eq 'Q' -or $choice -eq 'q') {
    Write-Host "`n  Cancelled." -ForegroundColor Gray
    exit 0
}
if ($choice -notin @('1','2','3','4')) {
    Write-Host "`n  Invalid selection. Exiting." -ForegroundColor Red
    exit 1
}

# Prompt for PAT (masked input)
Write-Host ""
$secPat = Read-Host "  Enter your Azure DevOps PAT" -AsSecureString
$bstr   = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secPat)
$Pat    = [Runtime.InteropServices.Marshal]::PtrToStringAuto($bstr)
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr)

if ([string]::IsNullOrWhiteSpace($Pat)) {
    Write-Host "  PAT cannot be empty. Exiting." -ForegroundColor Red
    exit 1
}

# Prompt for optional project filter
Write-Host ""
Write-Host "  Enter project name(s) to filter, comma-separated (e.g. EDCS,GALLERY)" -ForegroundColor White
Write-Host "  Leave blank to scan ALL projects." -ForegroundColor Gray
$projectInput = Read-Host "  Projects"
$projectArgs = @()
if (-not [string]::IsNullOrWhiteSpace($projectInput)) {
    $projectArgs = @('-Projects') + @($projectInput -split 's*,s*')
}

# Determine script and flags
switch ($choice) {
    '1' {
        $scriptFile = 'Remediate-AgentWorkItems.ps1'
        $dryRun = $true
        $label = 'Remediate (DRY RUN)'
    }
    '2' {
        $scriptFile = 'Remediate-AgentWorkItems.ps1'
        $dryRun = $false
        $label = 'Remediate (LIVE)'
    }
    '3' {
        $scriptFile = 'Remove-AgentComments.ps1'
        $dryRun = $true
        $label = 'Remove Comments (DRY RUN)'
    }
    '4' {
        $scriptFile = 'Remove-AgentComments.ps1'
        $dryRun = $false
        $label = 'Remove Comments (LIVE)'
    }
}

$scriptPath = Join-Path $scriptDir $scriptFile
if (-not (Test-Path $scriptPath)) {
    Write-Host "`n  ERROR: $scriptFile not found in $scriptDir" -ForegroundColor Red
    Write-Host "  Download all .ps1 files to the same folder and try again." -ForegroundColor Yellow
    exit 1
}

# Confirmation for LIVE runs
if (-not $dryRun) {
    Write-Host ""
    Write-Host "  *** WARNING: You selected LIVE mode. Changes will be applied! ***" -ForegroundColor Red
    Write-Host "  Script : $scriptFile" -ForegroundColor White
    if ($projectArgs.Count -gt 0) {
        Write-Host "  Projects: $($projectArgs[1..$projectArgs.Length] -join ', ')" -ForegroundColor White
    } else {
        Write-Host "  Projects: ALL" -ForegroundColor White
    }
    Write-Host ""
    $confirm = Read-Host "  Type YES to proceed"
    if ($confirm -ne 'YES') {
        Write-Host "`n  Cancelled." -ForegroundColor Gray
        exit 0
    }
}

# Build arguments and execute
Write-Host ""
Write-Host "  Starting: $label" -ForegroundColor Cyan
Write-Host "  ========================================================" -ForegroundColor Cyan
Write-Host ""

$scriptArgs = @('-OrgUrl', 'https://dev.azure.com/csc-ddsb', '-Pat', $Pat) + $projectArgs
if ($dryRun) { $scriptArgs += '-DryRun' }

& $scriptPath @scriptArgs

Combined Remediation Script (Remediate-AgentWorkItems.ps1)

This single script replaces the two separate scripts (Remove-AgentTags and Mark-AgentComments). It processes every project in one pass: first removing agent claim tags from work items, then prepending please disregard this comment to any agent-authored comment. A unified CSV report and per-project summary are produced. Run Remove-AgentComments.ps1 (below) afterwards to permanently delete the marked comments.

Remediate-AgentWorkItems.ps1

param(
    [string]$OrgUrl = "https://dev.azure.com/csc-ddsb",
    [string]$Pat,
    [string]$AgentId = "CPU-Agent-01",
    [string]$AgentCommentFingerprint = "[CPU Agent",
    [string]$ReportPath = "Remediate-AgentWorkItems-Report-$(Get-Date -Format 'yyyyMMdd-HHmmss').csv",
    [string[]]$Projects,
    [switch]$DryRun
)

$headers = @{
    Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$Pat")))"
    "Content-Type" = "application/json"
}

$report  = [System.Collections.Generic.List[PSCustomObject]]::new()
$summary = [System.Collections.Generic.List[PSCustomObject]]::new()
$lastFlush = 0

# Step 1: Enumerate all projects in the organisation
Write-Host "`n=== Discovering projects in $OrgUrl ==="
$projectsResult = Invoke-RestMethod -Uri "$OrgUrl/_apis/projects?`$top=500&api-version=7.1" -Headers $headers
$projects = $projectsResult.value
if ($null -ne $Projects) {
    $projects = @($projects | Where-Object { $_.name -in $Projects })
    Write-Host "Filtered to $($projects.Count) project(s): $($projects.name -join ', ')"
} else {
    Write-Host "Found $($projects.Count) project(s) (all): $($projects.name -join ', ')"
}

foreach ($project in $projects) {
    $projectName = $project.name
    Write-Host "`n--- Project: $projectName ---"

    # Step 2a: Find work items via tag prefix (agent:CPU-Agent-01: with trailing colon)
    # NOTE: WIQL CONTAINS matches full tag tokens; the tag format is agent:CPU-Agent-01:<timestamp>
    # We query using History (comments) as the primary discovery path, then also check tags client-side
    $wiqlByComment = @{
        query = "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = '$projectName' AND [System.History] CONTAINS '$AgentCommentFingerprint' AND [System.State] NOT IN ('Closed', 'Resolved', 'Done', 'Removed')"
    } | ConvertTo-Json

    # Step 2b: Also query by the partial tag string -- Azure DevOps CONTAINS on Tags field
    # matches substrings within the full tags string (semicolon-separated), so 'agent:CPU-Agent-01:'
    # (with trailing colon) will match 'agent:CPU-Agent-01:2026-...'
    $wiqlByTag = @{
        query = "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = '$projectName' AND [System.Tags] CONTAINS 'agent:$($AgentId):' AND [System.State] NOT IN ('Closed', 'Resolved', 'Done', 'Removed')"
    } | ConvertTo-Json

    $allIds = @{}
    foreach ($wiqlBody in @($wiqlByComment, $wiqlByTag)) {
        try {
            $wiqlResult = Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/wiql?api-version=7.1" `
                -Method Post -Headers $headers -Body $wiqlBody
            foreach ($wi in $wiqlResult.workItems) { $allIds[$wi.id] = $true }
        } catch {
            Write-Warning "  WIQL query failed for $projectName : $($_.Exception.Message)"
        }
    }

    $workItemIds = @($allIds.Keys)
    Write-Host "  Found $($workItemIds.Count) work item(s) with agent tags or comments."
    $tagsCleaned    = 0
    $commentsMarked = 0

    # Step 3: Process in batches of 200 for tag cleanup
    $batchSize = 200
    for ($i = 0; $i -lt $workItemIds.Count; $i += $batchSize) {
        $batch = $workItemIds[$i..([Math]::Min($i + $batchSize - 1, $workItemIds.Count - 1))]
        $items = Invoke-RestMethod `
            -Uri "$OrgUrl/_apis/wit/workitems?ids=$($batch -join ',')&fields=System.Id,System.Title,System.Tags,System.Rev&api-version=7.1" `
            -Headers $headers

        foreach ($item in $items.value) {
            $id    = $item.id
            $title = $item.fields.'System.Title'
            $tags  = $item.fields.'System.Tags'
            if ([string]::IsNullOrEmpty($tags)) { continue }

            # Client-side filter: only remove tags matching the agent:AgentId: prefix pattern
            $tagList     = $tags -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ }
            $agentTags   = $tagList | Where-Object { $_ -like "agent:$($AgentId):*" }
            $cleanedTags = ($tagList | Where-Object { $_ -notlike "agent:$($AgentId):*" }) -join '; '

            if ($agentTags.Count -eq 0) { continue }

            $status = if ($DryRun) { "DRY RUN" } else { "TAGS CLEANED" }
            $report.Add([PSCustomObject]@{
                Timestamp  = (Get-Date -Format 'yyyy-MM-dd HH:mm:ss')
                Project    = $projectName
                WorkItemId = $id
                Title      = $title
                Operation  = "TagCleanup"
                Detail     = "Removed $($agentTags.Count) agent tag(s): $($agentTags -join ' | ') | After: $cleanedTags"
                Status     = $status
            })

            if ($DryRun) {
                Write-Host "  [DRY RUN] WI $id '$title': would remove $($agentTags.Count) agent tag(s)"
                $tagsCleaned++
                continue
            }

            $patchBody = @(@{ op = "replace"; path = "/fields/System.Tags"; value = $cleanedTags }) | ConvertTo-Json -AsArray
            Invoke-RestMethod -Uri "$OrgUrl/_apis/wit/workitems/$($id)?api-version=7.1" `
                -Method Patch -Headers $headers -Body $patchBody `
                -ContentType "application/json-patch+json" | Out-Null
            Write-Host "  [TAGS] Cleaned $($agentTags.Count) tag(s) from WI $id '$title'"
            $tagsCleaned++
        }
    }

    # Step 4: Scan comments for each work item and mark agent-authored ones
    $wiIndex = 0
    foreach ($id in $workItemIds) {
        $wiIndex++
        $wiDetail = Invoke-RestMethod -Uri "$OrgUrl/_apis/wit/workitems/$($id)?fields=System.Title&api-version=7.1" -Headers $headers
        $title = $wiDetail.fields.'System.Title'
        Write-Host "  Processing WI $id '$title' ($wiIndex of $($workItemIds.Count))..."
        $allComments = [System.Collections.Generic.List[object]]::new()
        $commentsUrl = "$OrgUrl/$projectName/_apis/wit/workitems/$id/comments?`$top=200&api-version=7.1-preview.3"
        $seenIds = [System.Collections.Generic.HashSet[int]]::new()
        $loopDetected = $false
        do {
            $fetchedUrl = $commentsUrl
            $commentsResult = Invoke-RestMethod -Uri $commentsUrl -Headers $headers
            if ($commentsResult.comments) {
                foreach ($c in $commentsResult.comments) {
                    if (-not $seenIds.Add($c.id)) {
                        Write-Warning "  Loop detected on WI $id -- comment $($c.id) already seen. Breaking pagination."
                        $loopDetected = $true
                        break
                    }
                }
                if ($loopDetected) { break }
                $allComments.AddRange($commentsResult.comments)
            }
            $nl = $commentsResult.nextLink
            $commentsUrl = if ($nl -and $nl -ne $fetchedUrl) { $nl } else { $null }
        } while ($commentsUrl)

        $wiMatches = 0
        $wiSkipped = 0
        foreach ($comment in $allComments) {
            $text = $comment.text
            if ($text -notmatch [regex]::Escape($AgentCommentFingerprint)) { continue }
            if ($text -like "*please disregard this comment*") { $wiSkipped++; continue }  # idempotent

            $newText = "please disregard this comment -- this was generated by an automated CPU agent during an incorrectly scoped test run and does not reflect any human decision or review.`n`n---`n`n$text"

            $status = if ($DryRun) { "DRY RUN" } else { "COMMENT MARKED" }
            $report.Add([PSCustomObject]@{
                Timestamp  = (Get-Date -Format 'yyyy-MM-dd HH:mm:ss')
                Project    = $projectName
                WorkItemId = $id
                Title      = $title
                Operation  = "CommentMark"
                Detail     = "CommentId: $($comment.id) | Author: $($comment.createdBy.displayName) | Text: $($text.Substring(0, [Math]::Min(120, $text.Length)))"
                Status     = $status
            })

            $wiMatches++
            if ($DryRun) {
                Write-Host "  [DRY RUN] WI $id '$title' Comment $($comment.id): would mark"
                $commentsMarked++
                continue
            }

            $patchBody = @{ text = $newText } | ConvertTo-Json
            Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/workitems/$id/comments/$($comment.id)?api-version=7.1-preview.3" `
                -Method Patch -Headers $headers -Body $patchBody | Out-Null
            Write-Host "  [COMMENTS] Marked WI $id '$title' Comment $($comment.id)"
            $commentsMarked++
        }
        Write-Host "    WI ${id}: $($allComments.Count) comments fetched, $wiMatches to mark, $wiSkipped already marked"
    }

    # Incremental flush: write CSV every 10,000 records to avoid data loss
    if ($report.Count - $lastFlush -ge 10000) {
        $report | Export-Csv -Path $ReportPath -NoTypeInformation -Encoding UTF8
        Write-Host "  [FLUSH] Saved $($report.Count) record(s) to $ReportPath"
        $lastFlush = $report.Count
    }

    $summary.Add([PSCustomObject]@{
        Project        = $projectName
        WorkItemsFound = $workItemIds.Count
        TagsCleaned    = $tagsCleaned
        CommentsMarked = $commentsMarked
        Status         = if ($DryRun) { "DRY RUN" } elseif (($tagsCleaned + $commentsMarked) -gt 0) { "REMEDIATED" } else { "NO CHANGES" }
    })
}

# Step 5: Print per-project summary
Write-Host "`n=== PROJECT SUMMARY ==="
$summary | Format-Table -AutoSize

# Step 6: Write detailed report
$report | Export-Csv -Path $ReportPath -NoTypeInformation -Encoding UTF8
Write-Host "Remediation complete. $($report.Count) operation(s) across $($projects.Count) project(s)."
Write-Host "Report saved to: $ReportPath"

# Step 7: Post-execution verification (skipped in dry-run mode)
if (-not $DryRun) {
    Write-Host "`n=== POST-EXECUTION VERIFICATION ==="
    $verifyTagsTotal   = 0
    $verifyOldComments = 0
    $verifyNewComments = 0
    $markedCount       = ($report | Where-Object { $_.Operation -eq 'CommentMark' }).Count

    foreach ($project in $projects) {
        $projectName = $project.name
        $vBody = @{ query = "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = '$projectName' AND [System.Tags] CONTAINS 'agent:$($AgentId):' AND [System.State] NOT IN ('Closed', 'Resolved', 'Done', 'Removed')" } | ConvertTo-Json
        try {
            $vResult = Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/wiql?api-version=7.1" `
                -Method Post -Headers $headers -Body $vBody
            $vIds = $vResult.workItems.id
            $verifyTagsTotal += $vIds.Count
            $icon = if ($vIds.Count -eq 0) { "[PASS]" } else { "[FAIL]" }
            Write-Host "  $icon $projectName - $($vIds.Count) tagged work item(s) remaining"

            foreach ($vid in $vIds) {
                $vAllComments = [System.Collections.Generic.List[object]]::new()
                $vCommentsUrl = "$OrgUrl/$projectName/_apis/wit/workitems/$vid/comments?`$top=200&api-version=7.1-preview.3"
                $vSeenIds = [System.Collections.Generic.HashSet[int]]::new()
                $vLoopDetected = $false
                do {
                    $vFetchedUrl = $vCommentsUrl
                    $vcr = Invoke-RestMethod -Uri $vCommentsUrl -Headers $headers
                    if ($vcr.comments) {
                        foreach ($vc2 in $vcr.comments) {
                            if (-not $vSeenIds.Add($vc2.id)) { $vLoopDetected = $true; break }
                        }
                        if ($vLoopDetected) { break }
                        $vAllComments.AddRange($vcr.comments)
                    }
                    $vnl = $vcr.nextLink
                    $vCommentsUrl = if ($vnl -and $vnl -ne $vFetchedUrl) { $vnl } else { $null }
                } while ($vCommentsUrl)
                foreach ($vc in $vAllComments) {
                    if ($vc.text -match [regex]::Escape($AgentCommentFingerprint) -and $vc.text -notmatch [regex]::Escape("please disregard this comment")) { $verifyOldComments++ }
                    if ($vc.text -like "*please disregard this comment*") { $verifyNewComments++ }
                }
            }
        } catch {
            Write-Warning "  [WARN] Could not verify $projectName : $($_.Exception.Message)"
        }
    }

    Write-Host ""
    $passA = $verifyTagsTotal -eq 0
    $passB = $verifyOldComments -eq 0
    $passC = $verifyNewComments -eq $markedCount
    Write-Host "  $(if ($passA) { '[PASS]' } else { '[FAIL]' }) Tagged work items remaining  : $verifyTagsTotal (expected 0)"
    Write-Host "  $(if ($passB) { '[PASS]' } else { '[FAIL]' }) Unmarked agent comments      : $verifyOldComments (expected 0)"
    Write-Host "  $(if ($passC) { '[PASS]' } else { '[FAIL]' }) Newly marked comments present: $verifyNewComments (expected $markedCount)"

    if ($passA -and $passB -and $passC) {
        Write-Host "`nVERIFICATION PASSED - all tags removed and agent comments marked."
    } else {
        Write-Host "`nVERIFICATION FAILED - review items above and re-run if needed."
    }
}

Comment Deletion Script (Remove-AgentComments.ps1)

Run this script after Mark-AgentComments.ps1 has completed. It finds every comment containing please disregard this comment and permanently deletes it via the Azure DevOps REST API DELETE _apis/wit/workitems/{id}/comments/{commentId} endpoint. The two-pass approach (mark then delete) ensures a human has reviewed the candidates before any permanent deletion occurs.

Two-Pass Safety Model

Step 1 (Mark-AgentComments.ps1) prepends please disregard this comment to agent comments. A human reviewer can inspect the results in Azure DevOps before any deletion occurs. Step 2 (this script) only deletes comments that contain that exact phrase — so if a human has removed the prefix from a comment to preserve it, this script will skip it automatically.

Remove-AgentComments.ps1

param(
    [string]$OrgUrl = "https://dev.azure.com/csc-ddsb",
    [string]$Pat,
    [string]$AgentId = "CPU-Agent-01",
    [string]$MarkerPhrase = "please disregard this comment",
    [string]$ReportPath = "Remove-AgentComments-Report-$(Get-Date -Format 'yyyyMMdd-HHmmss').csv",
    [string[]]$Projects,
    [switch]$DryRun
)

$headers = @{
    Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$Pat")))"
    "Content-Type" = "application/json"
}

$report  = [System.Collections.Generic.List[PSCustomObject]]::new()
$summary = [System.Collections.Generic.List[PSCustomObject]]::new()
$lastFlush = 0

# Step 1: Enumerate all projects
Write-Host "`n=== Discovering projects in $OrgUrl ==="
$projectsResult = Invoke-RestMethod -Uri "$OrgUrl/_apis/projects?`$top=500&api-version=7.1" -Headers $headers
$projects = $projectsResult.value
if ($null -ne $Projects) {
    $projects = @($projects | Where-Object { $_.name -in $Projects })
    Write-Host "Filtered to $($projects.Count) project(s): $($projects.name -join ', ')"
} else {
    Write-Host "Found $($projects.Count) project(s) (all): $($projects.name -join ', ')"
}

foreach ($project in $projects) {
    $projectName = $project.name
    Write-Host "`n--- Project: $projectName ---"

    # Step 2: Find work items that have the marker phrase in their history (comments)
    $wiqlBody = @{
        query = "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = '$projectName' AND [System.History] CONTAINS '$MarkerPhrase' AND [System.State] NOT IN ('Closed', 'Resolved', 'Done', 'Removed')"
    } | ConvertTo-Json

    try {
        $wiqlResult = Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/wiql?api-version=7.1" `
            -Method Post -Headers $headers -Body $wiqlBody
    } catch {
        Write-Warning "  Skipping $projectName - WIQL query failed: $($_.Exception.Message)"
        $summary.Add([PSCustomObject]@{ Project = $projectName; WorkItemsFound = 0; CommentsDeleted = 0; Status = "WIQL ERROR" })
        continue
    }

    $workItemIds = $wiqlResult.workItems.id
    Write-Host "  Found $($workItemIds.Count) work item(s) with marked comments."
    $commentsDeleted  = 0
    $preExistingCount = 0

    # Step 3: Delete marked comments
    $wiIndex = 0
    foreach ($id in $workItemIds) {
        $wiIndex++
        $wiDetail = Invoke-RestMethod -Uri "$OrgUrl/_apis/wit/workitems/$($id)?fields=System.Title&api-version=7.1" -Headers $headers
        $title = $wiDetail.fields.'System.Title'
        Write-Host "  Processing WI $id '$title' ($wiIndex of $($workItemIds.Count))..."
        $allComments = [System.Collections.Generic.List[object]]::new()
        $commentsUrl = "$OrgUrl/$projectName/_apis/wit/workitems/$id/comments?`$top=200&api-version=7.1-preview.3"
        $seenIds = [System.Collections.Generic.HashSet[int]]::new()
        $loopDetected = $false
        do {
            $fetchedUrl = $commentsUrl
            $commentsResult = Invoke-RestMethod -Uri $commentsUrl -Headers $headers
            if ($commentsResult.comments) {
                foreach ($c in $commentsResult.comments) {
                    if (-not $seenIds.Add($c.id)) {
                        Write-Warning "  Loop detected on WI $id -- comment $($c.id) already seen. Breaking pagination."
                        $loopDetected = $true
                        break
                    }
                }
                if ($loopDetected) { break }
                $allComments.AddRange($commentsResult.comments)
            }
            $nl = $commentsResult.nextLink
            $commentsUrl = if ($nl -and $nl -ne $fetchedUrl) { $nl } else { $null }
        } while ($commentsUrl)
        $preExistingCount += $allComments.Count

        $wiMatches = 0
        foreach ($comment in $allComments) {
            $text = $comment.text
            if ($text -notmatch [regex]::Escape($MarkerPhrase)) { continue }

            $report.Add([PSCustomObject]@{
                Timestamp  = (Get-Date -Format 'yyyy-MM-dd HH:mm:ss')
                Project    = $projectName
                WorkItemId = $id
                Title      = $title
                CommentId  = $comment.id
                Author     = $comment.createdBy.displayName
                CommentText = $text.Substring(0, [Math]::Min(120, $text.Length))
                Status     = if ($DryRun) { "DRY RUN" } else { "DELETED" }
            })

            if ($DryRun) {
                Write-Host "  [DRY RUN] Would DELETE WI $id '$title' Comment $($comment.id)"
                $commentsDeleted++
                continue
            }

            try {
                Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/workitems/$id/comments/$($comment.id)?api-version=7.1-preview.3" `
                    -Method Delete -Headers $headers | Out-Null
                Write-Host "  [DELETED] WI $id '$title' Comment $($comment.id)"
                $commentsDeleted++
            } catch {
                $report[-1].Status = "ERROR: $($_.Exception.Message)"
                Write-Warning "  Failed to delete WI $id Comment $($comment.id): $($_.Exception.Message)"
            }
            $wiMatches++
        }
        Write-Host "    WI ${id}: $($allComments.Count) comments fetched, $wiMatches matched marker phrase"
    }

    # Incremental flush: write CSV every 10,000 records to avoid data loss
    if ($report.Count - $lastFlush -ge 10000) {
        $report | Export-Csv -Path $ReportPath -NoTypeInformation -Encoding UTF8
        Write-Host "  [FLUSH] Saved $($report.Count) record(s) to $ReportPath"
        $lastFlush = $report.Count
    }

    $summary.Add([PSCustomObject]@{
        Project          = $projectName
        WorkItemsFound   = $workItemIds.Count
        CommentsDeleted  = $commentsDeleted
        PreExistingTotal = $preExistingCount
        Status           = if ($DryRun) { "DRY RUN" } elseif ($commentsDeleted -gt 0) { "DELETED" } else { "NO CHANGES" }
    })
}

# Step 4: Print per-project summary
Write-Host "`n=== PROJECT SUMMARY ==="
$summary | Format-Table -AutoSize

# Step 5: Write detailed report
$report | Export-Csv -Path $ReportPath -NoTypeInformation -Encoding UTF8
$totalDeleted = ($report | Where-Object { $_.Status -eq 'DELETED' }).Count
Write-Host "Deletion complete. $totalDeleted comment(s) permanently removed."
Write-Host "Report saved to: $ReportPath"

# Step 6: Post-execution verification
if (-not $DryRun) {
    Write-Host "`n=== POST-EXECUTION VERIFICATION ==="
    $verifyRemaining = 0
    $verifyFailed    = [System.Collections.Generic.List[string]]::new()

    foreach ($project in $projects) {
        $projectName = $project.name
        $vBody = @{ query = "SELECT [System.Id] FROM WorkItems WHERE [System.TeamProject] = '$projectName' AND [System.History] CONTAINS '$MarkerPhrase' AND [System.State] NOT IN ('Closed', 'Resolved', 'Done', 'Removed')" } | ConvertTo-Json
        try {
            $vResult = Invoke-RestMethod -Uri "$OrgUrl/$projectName/_apis/wit/wiql?api-version=7.1" `
                -Method Post -Headers $headers -Body $vBody
            $vIds = $vResult.workItems.id
            $verifyRemaining += $vIds.Count
            $icon = if ($vIds.Count -eq 0) { "[PASS]" } else { "[FAIL]" }
            Write-Host "  $icon $projectName - $($vIds.Count) marked comment(s) remaining"
            if ($vIds.Count -gt 0) { $verifyFailed.Add($projectName) }
        } catch {
            Write-Warning "  [WARN] Could not verify $projectName"
        }
    }

    Write-Host ""
    $totalPreExisting = ($summary | Measure-Object -Property PreExistingTotal -Sum).Sum
    $passA = $verifyRemaining -eq 0
    $passB = $totalDeleted -lt $totalPreExisting  # safety: deleted < total (did not over-delete)
    Write-Host "  $(if ($passA) { '[PASS]' } else { '[FAIL]' }) Marked comments remaining : $verifyRemaining (expected 0)"
    Write-Host "  $(if ($passB) { '[PASS]' } else { '[WARN]' }) Deleted ($totalDeleted) vs pre-existing total ($totalPreExisting) -- deleted count should be less than total"

    if ($passA -and $passB) {
        Write-Host "`nVERIFICATION PASSED - all marked comments deleted, no over-deletion detected."
    } else {
        Write-Host "`nVERIFICATION FAILED - review items above."
    }
}

PREVENTION

Controls to Prevent Recurrence

PC-01Required

Opt-In Tag Requirement

No work item is processed unless a human has explicitly added the CPU-Agent-Eligible tag. This prevents the agent from touching any work item that has not been deliberately opted in.

PC-02Required

Project Scope Enforcement

The WIQL query must always include [System.TeamProject] = @project. The @project macro resolves to the configured ProjectName value, preventing cross-project contamination.

PC-03Required

Dedicated Service Account PAT

All agent operations must be attributed to a named service account (svc-cpu-agent), not a personal account. This makes agent activity immediately distinguishable in the Azure DevOps UI.

PC-04Required

Dry-Run Mode for First Run

DryRunMode: true must be set in appsettings.json for any new deployment. The agent logs all intended writes without executing them. A human must review and explicitly set DryRunMode: false.

PC-05Recommended

MaxWorkItemsPerRun Blast Radius Cap

Even if the WIQL query returns thousands of items, the agent processes at most N items per run. Start at 10, increase gradually after validating correct behaviour.

PC-06Recommended

Azure DevOps Audit Log Alerting

Configure an Azure Monitor alert to notify the team when the service account makes more than 50 work item updates per hour. This provides early warning of runaway agent behaviour.

PC-07Recommended

Integration Tests Against Isolated Test Project

Before deploying to production, run the agent against a dedicated CPU-Agents-Test Azure DevOps project containing synthetic work items. Validate that only opted-in items are processed and no cross-project contamination occurs.

CONFIGURATION

Required appsettings.json Changes

Apply these changes to appsettings.json before restarting the agent. Set DryRunMode: false only after reviewing the dry-run output and confirming correct behaviour.

{
  "Agent": {
    "Name": "CPU-Agent-01",
    "PollingIntervalSeconds": 300,
    "MaxConcurrentWorkItems": 2,
    "MaxWorkItemsPerRun": 10,
    "DryRunMode": true,
    "WorkItemQueryWiql": "SELECT [System.Id], [System.Title], [System.State], [System.WorkItemType] FROM WorkItems WHERE [System.TeamProject] = @project AND [System.State] = 'Active' AND [System.Tags] CONTAINS 'CPU-Agent-Eligible' ORDER BY [System.CreatedDate] ASC",
    "WorkflowsDirectory": "workflows"
  },
  "AzureDevOps": {
    "OrganizationUrl": "https://dev.azure.com/csc-ddsb",
    "ProjectName": "EDCS",
    "AuthenticationMethod": "PAT",
    "PAT": "<new-scoped-service-account-pat>",
    "CacheToken": true
  }
}

ACTION SUMMARY

All Required Actions

Action	Owner	Priority	Status
Stop agent process	Ops	P0	required
Revoke personal PAT	Ops	P0	required
Run combined remediation script (Remediate-AgentWorkItems.ps1) — dry run then live	Ops	P1	required
Delete marked agent comments (Remove-AgentComments.ps1)	Ops	P1	required
Fix WIQL query to scope to single project	Dev	P1	required
Fix workflow condition tag matching	Dev	P1	required
Populate agentName in workflow context	Dev	P1	required
Issue scoped service account PAT	Ops	P1	required
Add MaxWorkItemsPerRun limit	Dev	P2	recommended
Add dry-run mode	Dev	P2	recommended
Enable Audit Log alerting	Ops	P2	recommended
Run integration tests against test project	Dev	P2	recommended

IDLE_STATE

CPU-Agent-01Bulk Work Item Modification