ci: add nightly disk cleanup for self-hosted runner ci1 (#2620)

.github/workflows/README.md

@@ -40,7 +40,7 @@ flowchart TD
 ## Workflows
 
 | File                         | Name                           | Description                                                                                          | Trigger                                                                                                              | Output labels                                                                                                                                                                                       |
-| --------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ---------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `bugbot-trigger.yml`         | `BugBot Trigger`               | Posts `@BugBot run` on eligible PRs so BugBot starts a review.                                       | `pull_request_target` on `opened/synchronize/ready_for_review/reopened`; or `workflow_dispatch` with `pr_number`.    | None.                                                                                                                                                                                               |
 | `cancel-ci-after-merge.yml`  | `Cancel CI after merge`        | Cancels still-running or queued `CI` runs for a PR commit after merge.                               | `pull_request` on `closed` (only when merged).                                                                       | None.                                                                                                                                                                                               |
 | `ci.yml`                     | `CI`                           | Runs presubmit checks, type checks, unit tests, build, and Playwright E2E/report merge.              | `push` to `main`; `pull_request` on `opened/synchronize/reopened/closed`; or `workflow_dispatch` with `pr_number`.   | None.                                                                                                                                                                                               |
@@ -54,7 +54,21 @@ flowchart TD
 | `draft-stale-prs.yml`        | `Draft stale PRs`              | Converts inactive open PRs to draft after 7 days without meaningful activity.                        | Daily cron (`0 0 * * *`) or `workflow_dispatch`.                                                                     | None.                                                                                                                                                                                               |
 | `label-rebase-prs.yml`       | `Label PRs needing rebase`     | Finds conflicting open PRs from allowed authors and flags them for rebase.                           | `push` to `main`.                                                                                                    | Adds `cc:rebase` when eligible PR is conflicted (`mergeable_state == dirty`) and not already in rebase states.                                                                                      |
 | `merge-pr.yml`               | `Merge PR when ready`          | Auto-merges eligible PRs after successful CI when all checks pass.                                   | `workflow_run` for `CI` on `completed` (successful PR-triggered CI only).                                            | None (reads `merge-when-ready`, does not set labels).                                                                                                                                               |
+| `nightly-runner-cleanup.yml` | `Nightly Runner Cleanup`       | Safely frees disk space on self-hosted runner ci1 (caches, npm, runner \_work).                      | Daily cron (4 AM PST); or `workflow_dispatch`.                                                                       | None.                                                                                                                                                                                               |
 | `playwright-comment.yml`     | `Playwright Report Comment`    | Posts a Playwright summary comment on the PR tied to a completed CI run.                             | `workflow_run` for `CI` on `completed`.                                                                              | None.                                                                                                                                                                                               |
 | `pr-review-responder.yml`    | `PR Review Responder`          | Runs Claude fix loops for trusted PRs, retriggers checks/reviews, and advances request-state labels. | `workflow_run` for `CI` on `completed`.                                                                              | `cc:request`/`cc:request:N` -> `cc:pending`; then `cc:request:N+1` on pushed commits, `cc:done` on clean finish, `cc:failed` on failure; may add `needs-human:review-issue` when retries exhausted. |
 | `pr-status-labeler.yml`      | `PR Status Labeler`            | Applies human-attention labels based on CI outcome and review freshness/verdict.                     | `workflow_run` for `CI` on `completed`.                                                                              | Swaps between `needs-human:final-check` (clean + passing) and `needs-human:review-issue` (failing/stale/missing/issueful review).                                                                   |
 | `release.yml`                | `Release app`                  | Manually builds and publishes signed release artifacts across platforms, then verifies assets.       | `workflow_dispatch`.                                                                                                 | None.                                                                                                                                                                                               |
+
+## Nightly Runner Cleanup
+
+The `nightly-runner-cleanup.yml` workflow runs at 4:00 AM PST on self-hosted macOS runner `ci1` to reclaim disk space. It only runs when `RUNNER_NAME == ci1`; other runners skip cleanup.
+
+**Validation (manual run):**
+
+1. Go to Actions → Nightly Runner Cleanup → Run workflow.
+2. Confirm the run completes successfully and logs show "Running cleanup on runner: ci1".
+3. Check logs for "Disk before" and "Disk after" to verify space reclaimed.
+4. On non-ci1 runners, logs should show "Skipping cleanup" and exit successfully.
+
+**Expected behavior:** Deletes only allowlisted paths (npm cache, Playwright browsers, runner \_work dirs older than 2 days, Library/Caches subdirs). Never removes runner binaries, config, or user data outside caches.

.github/workflows/nightly-runner-cleanup.yml

+# Nightly disk cleanup for self-hosted macOS CI runners (e.g. ci1).
+# Runs at 4:00 AM PST to prevent disk exhaustion from CI workloads.
+# Safe to run manually via workflow_dispatch for testing.
+
+name: Nightly Runner Cleanup
+
+on:
+  schedule:
+    # 4:00 AM PST = 12:00 UTC (PST is UTC-8)
+    - cron: "0 12 * * *"
+  workflow_dispatch:
+
+jobs:
+  cleanup:
+    runs-on: [self-hosted, macOS, ARM64]
+    steps:
+      - name: Guard — run only on ci1
+        id: guard
+        run: |
+          if [ "$RUNNER_NAME" = "ci1" ]; then
+            echo "run_cleanup=true" >> "$GITHUB_OUTPUT"
+            echo "Running cleanup on runner: $RUNNER_NAME"
+          else
+            echo "run_cleanup=false" >> "$GITHUB_OUTPUT"
+            echo "Skipping cleanup: runner is '$RUNNER_NAME', expected ci1"
+          fi
+
+      - name: Checkout (for cleanup script)
+        if: steps.guard.outputs.run_cleanup == 'true'
+        uses: actions/checkout@v4
+
+      - name: Nightly disk cleanup
+        if: steps.guard.outputs.run_cleanup == 'true'
+        env:
+          CI_NIGHTLY_CLEANUP: "1"
+        run: bash scripts/ci-cleanup-macos.sh

scripts/ci-cleanup-macos.sh

@@ -4,6 +4,9 @@
 # Intended to run as a post-job step in GitHub Actions workflows that use
 # self-hosted macOS ARM64 runners. Safe to run multiple times (idempotent).
 #
+# Set CI_NIGHTLY_CLEANUP=1 for nightly runs to also clean host-level caches
+# (Library/Caches subdirs, runner _work stale dirs). Only allowlisted paths.
+#
 # What it cleans:
 #   1. Build outputs           (out/, out-macos.tar)
 #   2. Blob reports            (blob-report/)
@@ -11,10 +14,16 @@
 #   4. Old Playwright browsers (keeps only the current version)
 #   5. npm cache artifacts     (_cacache, _logs)
 #   6. Old runner diagnostics  (_diag/*.log older than 7 days)
+#   7. [Nightly only] ~/Library/Caches subdirs, runner _work (older than 2 days)
 
 set -euo pipefail
 
 echo "=== CI Cleanup (macOS self-hosted) ==="
+if [ "${CI_NIGHTLY_CLEANUP:-0}" = "1" ]; then
+  echo "Mode: nightly (host-level + workspace)"
+else
+  echo "Mode: per-job (workspace only)"
+fi
 df -h / | tail -1 | awk '{print "Disk before cleanup: "$4" available ("$5" used)"}'
 
 bytes_before=$(df -k / | tail -1 | awk '{print $4}')
@@ -115,6 +124,40 @@ if [ -d "$RUNNER_DIR/_diag" ]; then
   fi
 fi
 
+# ---------------------------------------------------------------------------
+# 7. [Nightly only] Host-level caches and stale runner _work
+#    Only when CI_NIGHTLY_CLEANUP=1. Allowlisted paths only.
+# ---------------------------------------------------------------------------
+if [ "${CI_NIGHTLY_CLEANUP:-0}" = "1" ]; then
+  CACHES="$HOME/Library/Caches"
+  # Allowlisted subdirs (never wipe entire Caches)
+  for subdir in Homebrew com.apple.dt.Xcode; do
+    d="$CACHES/$subdir"
+    if [ -d "$d" ]; then
+      size=$(du -sh "$d" 2>/dev/null | cut -f1 || echo "?")
+      echo "Removing cache: $d (${size})"
+      rm -rf "$d"
+    fi
+  done
+  for d in "$CACHES"/org.llvm.clang*; do
+    [ -d "$d" ] || continue
+    size=$(du -sh "$d" 2>/dev/null | cut -f1 || echo "?")
+    echo "Removing cache: $d (${size})"
+    rm -rf "$d"
+  done
+
+  # Runner _work: remove stale job workspaces older than 2 days
+  if [ -d "$RUNNER_DIR/_work" ]; then
+    stale=$(find "$RUNNER_DIR/_work" -mindepth 1 -maxdepth 1 -type d -mtime +2 2>/dev/null || true)
+    if [ -n "$stale" ]; then
+      echo "Removing stale _work dirs (older than 2 days):"
+      echo "$stale"
+      echo "$stale" | while IFS= read -r dir; do rm -rf "$dir"; done
+
+    fi
+  fi
+fi
+
 # ---------------------------------------------------------------------------
 # Summary
 # ---------------------------------------------------------------------------