Add check-workflows skill for monitoring GitHub Actions workflow health (#2738)

.claude/skills/check-workflows/SKILL.md

+---
+name: dyad:check-workflows
+description: Check GitHub Actions workflow runs from the past day, identify severe or consistent failures, and file an issue if actionable problems are found.
+---
+
+# Check Workflows
+
+Check GitHub Actions workflow runs from the past day for severe or consistent failures and file a GitHub issue if actionable problems are found.
+
+## Arguments
+
+- `$ARGUMENTS`: (Optional) Number of hours to look back (default: 24)
+
+## Instructions
+
+### 1. Gather recent workflow runs
+
+Fetch all workflow runs from the past N hours (default 24):
+
+```
+gh run list --limit 100 --json workflowName,status,conclusion,event,headBranch,createdAt,databaseId,url,name
+```
+
+Filter to only runs created within the lookback window. Group runs by workflow name.
+
+### 2. Classify each failure
+
+For each failed run, determine if it is **expected** or **actionable** by checking these rules:
+
+#### Expected failures (IGNORE these):
+
+1. **Nightly Runner Cleanup**: This workflow intentionally reboots self-hosted macOS runners, which kills the runner process mid-job. It will almost always show as "failed" even when working correctly. **Always skip this workflow entirely.**
+
+2. **Cascading failures from CI**: When the main CI workflow fails, these downstream workflows will also fail because they depend on CI artifacts (e.g. `html-report`, blob reports). This is noise, not an independent problem:
+   - Playwright Report Comment (fails with "artifact not found")
+   - Upload to Flakiness.io (fails when no flakiness reports exist)
+   - Merge PR when ready (skipped/fails when CI hasn't passed)
+
+3. **CLA Assistant**: Failures just mean a contributor hasn't signed the CLA yet. This resolves on its own.
+
+4. **Cancelled runs**: Runs cancelled due to concurrency groups (newer push cancels older run) are normal.
+
+5. **`action_required` / `neutral` conclusions**: Standard GitHub behavior for fork PRs or first-time contributors needing manual approval.
+
+6. **CI failures on non-main branches**: Individual PR CI failures are expected — contributors may have formatting issues, lockfile mismatches, test failures, etc. These are the contributor's responsibility.
+
+7. **Claude Deflake E2E**: This workflow is expected to sometimes have long runs or partial failures as it investigates flaky tests.
+
+#### Actionable failures (FLAG these):
+
+1. **Permission errors**: Workflow can't access secrets, missing `GITHUB_TOKEN`, 403/401 errors on API calls that should be authenticated, `Resource not accessible by integration` errors.
+
+2. **Consistent CI failures on main branch**: If the CI workflow fails on 2+ consecutive pushes to main, something is likely broken. Check if different commits are failing for the same reason.
+
+3. **Infrastructure failures**: Self-hosted runners not coming back online (check if Nightly Runner Cleanup's verify steps are failing), runners consistently unavailable, disk space issues.
+
+4. **Repeated rate limiting**: If GitHub API rate limiting is causing the same workflow to fail across multiple runs (not just a one-off).
+
+5. **Action version issues**: Deprecated or broken GitHub Action versions causing failures.
+
+6. **Workflow configuration errors**: YAML syntax errors, invalid inputs, missing required secrets (distinct from permission issues).
+
+7. **Scheduled workflow failures**: If a scheduled/cron workflow (other than Nightly Runner Cleanup) fails consistently, it likely indicates a systemic issue.
+
+### 3. Investigate actionable failures
+
+For each potentially actionable failure, get more details:
+
+```
+gh run view <run_id> --log-failed 2>/dev/null | head -100
+```
+
+Look for:
+
+- The specific error message
+- Whether the failure is in a setup step (infrastructure) vs. a test/build step (code)
+- Whether the same failure appears across multiple runs
+
+### 4. Determine severity
+
+After investigation, categorize actionable failures:
+
+- **SEVERE**: Permission errors, infrastructure down, main branch consistently broken, workflow configuration errors
+- **MODERATE**: Repeated rate limiting, deprecated action warnings, intermittent infrastructure issues
+- **LOW**: One-off transient failures that resolved on retry
+
+Only proceed to file an issue if there are SEVERE or MODERATE findings.
+
+### 5. Check for existing issues
+
+Before creating a new issue, check if there's already an open issue about workflow problems:
+
+```
+gh issue list --label "workflow-health" --state open --json number,title,body
+```
+
+If an existing issue covers the same problems, do not create a duplicate. Instead, add a comment to the existing issue with the latest findings.
+
+### 6. File a GitHub issue
+
+If there are actionable findings (SEVERE or MODERATE), create a GitHub issue:
+
+```
+gh issue create --title "Workflow issues: <X>, <Y>, and <Z>" --label "workflow-health" --body "$(cat <<'EOF'
+## Workflow Health Report
+
+**Period:** <start_time> to <end_time>
+**Total runs checked:** <N>
+**Failures found:** <N actionable> actionable, <N expected> expected (ignored)
+
+## Issues Found
+
+### <Issue 1 Title>
+- **Workflow:** <workflow name>
+- **Severity:** SEVERE / MODERATE
+- **Failed runs:**
+  - [Run #<id>](<url>) — <date>
+  - [Run #<id>](<url>) — <date>
+- **Error:** <brief error description>
+- **Suggested fix:** <how to resolve>
+
+### <Issue 2 Title>
+...
+
+## Expected Failures (Ignored)
+<Brief summary of expected failures that were skipped and why>
+
+---
+*This issue was automatically created by the daily workflow health check.*
+EOF
+)"
+```
+
+The issue title should list the specific problems found (e.g., "Workflow issues: CI permissions error, flakiness upload rate-limited"). Keep it concise but descriptive.
+
+### 7. Report results
+
+Summarize:
+
+- How many workflow runs were checked
+- How many were expected failures (and which categories)
+- How many were actionable (and what was found)
+- Whether an issue was filed (with link) or if everything looks healthy
+- If no actionable issues were found, report "All workflows healthy" and do not create an issue

.github/workflows/claude-check-workflows.yml

+name: Claude Check Workflows
+
+on:
+  schedule:
+    # Daily at 13:00 UTC (5 AM PST / 6 AM PDT)
+    - cron: "0 13 * * *"
+  workflow_dispatch:
+    inputs:
+      hours:
+        description: "Number of hours to look back for workflow runs"
+        required: false
+        default: "24"
+        type: string
+
+jobs:
+  check-workflows:
+    environment: ai-bots
+    runs-on:
+      - self-hosted
+      - macOS
+      - ARM64
+    permissions:
+      issues: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Check workflow health
+        uses: anthropics/claude-code-action@v1
+        env:
+          GH_TOKEN: ${{ secrets.DEFLAKE_E2E_PAT_GITHUB_TOKEN }}
+          CLAUDE_CODE_MAX_OUTPUT_TOKENS: 48000
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+          github_token: ${{ secrets.DEFLAKE_E2E_PAT_GITHUB_TOKEN }}
+          claude_args: --model claude-sonnet-4-5-20250929
+          prompt: |
+            /dyad:check-workflows ${{ inputs.hours || '24' }}
+      - name: Cleanup (self-hosted macOS)
+        if: always()
+        run: bash scripts/ci-cleanup-macos.sh