Skip to main content

Testing 3D Applications with Playwright on GPU

Reference: Testing 3D Applications with Playwright on GPU

Developing medical-grade web applications necessitates robust testing practices. End-to-end (E2E) testing is a critical component of the test suite, and Playwright is utilized to automate these tests. Playwright's native browser events and support for orchestrated test execution make it a suitable choice for testing 3D applications rendered on canvas.

During the expansion of the test suite, challenges were encountered: tests performed efficiently on local ARM MacBooks but exhibited slower and less stable behavior on CI. Testing web-based 3D applications is a specialized topic, requiring investigation and experimentation to identify effective solutions.

Ultimately, enabling hardware acceleration for all major browsers in CI significantly improved runtime and stability. The following outlines the key findings and solutions.

Summary​

  • GPU Utilization: A performant GPU on the CI runner can substantially enhance test speed and stability, albeit with increased costs.
  • Chromium on Linux: Specific flags are required to enable hardware acceleration in the new headless mode (introduced in Playwright v1.49).
  • Firefox: Hardware acceleration is enabled by default but requires Xvfb for functionality in headless mode.
  • WebKit: GPU acceleration is not supported in headless mode and must be executed in headed mode.

Command for Running Playwright​

# Xvfb facilitates WebGL in Firefox Headless
# --headed is necessary for WebKit, as GPU acceleration is unsupported in headless mode
xvfb-run yarn exec playwright test --headed

Browser Configuration in playwright.config.ts​

import { defineConfig, devices } from "@playwright/test";

const chromiumGpuOnLinuxFlags = [
  "--use-angle=vulkan",
  "--enable-features=Vulkan",
  "--disable-vulkan-surface",
  "--enable-unsafe-webgpu",
];

export default defineConfig({
  use: {
    trace: "retain-on-failure",
  },
  projects: [
    {
      name: "chromium",
      use: {
        ...devices["Desktop Chrome"],
        channel: "chromium",
        launchOptions: {
          args: [
            "--no-sandbox",
            ...(process.platform === "linux" ? chromiumGpuOnLinuxFlags : []),
          ],
        },
      },
    },
    {
      name: "firefox",
      use: { ...devices["Desktop Firefox"] },
    },
    {
      name: "webkit",
      use: { ...devices["Desktop Safari"] },
    },
  ],
});

Implementation Details​

Chromium​

Chromium's new headless mode supports GPU acceleration, but specific flags must be configured. Guidance from the Puppeteer project and the Supercharge Web AI Testing article was followed to enable WebGPU.

Firefox​

For Firefox, Xvfb is required to enable hardware acceleration in headless mode. Incorporating xvfb-run into CI commands resolved this issue.

WebKit​

WebKit does not support GPU acceleration in headless mode. Running it in headed mode addressed this limitation, though it introduced certain quirks with Firefox, which were mitigated by enabling Playwright traces.

Results​

Enabling GPU acceleration reduced the test suite runtime from 24 minutes on a default Azure runner to 3.5 minutes on a self-hosted AWS GPU runner. However, tests continue to execute faster locally on an M3 MacBook Air.

Cost Analysis​

The g5.xlarge GPU runner incurs a cost of $1.06/hour. To manage expenses, auto-scaling was implemented. Cost considerations should be taken into account when setting up GPU runners for similar projects.

Conclusion​

Testing 3D applications with Playwright on GPU presents challenges but is achievable with appropriate configurations. While the solution involves several workarounds, it has significantly improved the CI pipeline's performance. This documentation aims to assist others testing Three.js or similar applications.

Future advancements in GPU-accelerated testing are anticipated to simplify and streamline this process.