• Will Chen's avatar
    Improve test failure reporting with snapshot detection (#2484) · ea3b676f
    Will Chen 提交于
    ## Summary
    Enhanced the Playwright test summary generation to better handle
    snapshot failures and provide more actionable test commands in PR
    comments.
    
    ## Key Changes
    - **Added snapshot failure detection**: New `isSnapshotFailure()`
    function that identifies snapshot-related test failures by checking
    error messages for common snapshot comparison keywords
    - **Simplified test command output**: Removed the separate
    `generateCommands()` function and consolidated to a single command per
    test
    - **Smart command generation**: Automatically appends
    `--update-snapshots` flag only for snapshot failures, reducing confusion
    about when to use it
    - **Improved readability**: 
    - Reformatted test commands into a single code block with better
    organization
    - Added error preview (first 120 chars) to help developers understand
    failure context
    - Added `export PLAYWRIGHT_HTML_OPEN=never` to prevent browser windows
    from opening during test runs
    - Changed section description from "run or update snapshots" to "re-run
    failing tests locally" for clarity
    
    ## Implementation Details
    - The snapshot detection checks for 10 different snapshot-related error
    message patterns (case-insensitive)
    - Test commands are now grouped in a single bash code block for easier
    copy-pasting
    - Error messages are truncated to 120 characters to keep the comment
    concise while still providing useful context
    
    https://claude.ai/code/session_014DzYzvnt4559rZEGLwbVm5
    <!-- devin-review-badge-begin -->
    
    ---
    
    <a href="https://app.devin.ai/review/dyad-sh/dyad/pull/2484">
      <picture>
    <source media="(prefers-color-scheme: dark)"
    srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
    <img
    src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
    alt="Open with Devin">
      </picture>
    </a>
    <!-- devin-review-badge-end -->
    
    <!-- This is an auto-generated description by cubic. -->
    ---
    ## Summary by cubic
    Improved Playwright PR test failure reporting by detecting snapshot
    mismatches and generating clearer, single-run commands. Makes it faster
    and safer to re-run failures locally and reduces confusion around
    snapshot updates.
    
    - **New Features**
    - Detect snapshot failures from error text and auto-append
    `--update-snapshots` only when needed.
    - Consolidate all failing test commands into one bash block with
    `PLAYWRIGHT_HTML_OPEN=never` set once.
    - Add a short error preview (first 120 chars) as a comment above each
    command.
    - Sanitize spec paths and escape test names/error previews to prevent
    injection.
    
    <sup>Written for commit 0515c892605dc303c60d1dbae4a2c0cd3a537481.
    Summary will update on new commits.</sup>
    
    <!-- End of auto-generated description by cubic. -->
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Low Risk**
    > Low risk: changes only affect CI-generated PR comments and local rerun
    command formatting, with added input sanitization reducing injection
    risk.
    > 
    > **Overview**
    > Improves the Playwright PR comment generator to **detect
    snapshot-related failures** and tailor rerun commands accordingly.
    > 
    > The macOS “Test Commands” section is reformatted into a single
    copy-pasteable bash block (including `PLAYWRIGHT_HTML_OPEN=never`), adds
    a short error preview per test, and appends `--update-snapshots` *only*
    when a failure looks snapshot-related. Command generation now also
    sanitizes the spec path and removes the separate run/update command
    helper.
    > 
    > <sup>Written by [Cursor
    Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
    0515c892605dc303c60d1dbae4a2c0cd3a537481. This will update automatically
    on new commits. Configure
    [here](https://cursor.com/dashboard?tab=bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    
    ---------
    Co-authored-by: 's avatarClaude <noreply@anthropic.com>
    ea3b676f
名称
最后提交
最后更新
..
README.md 正在载入提交数据...
clear_console_logs.py 正在载入提交数据...
generate-playwright-summary.js 正在载入提交数据...
tsconfig.json 正在载入提交数据...
verify-release-assets.js 正在载入提交数据...