Skip to content

Add tests for WebAgg using Playwright - #23540

Draft
QuLogic wants to merge 8 commits into
matplotlib:mainfrom
QuLogic:test-webagg
Draft

Add tests for WebAgg using Playwright#23540
QuLogic wants to merge 8 commits into
matplotlib:mainfrom
QuLogic:test-webagg

Conversation

@QuLogic

@QuLogic QuLogic commented Aug 3, 2022

Copy link
Copy Markdown
Member

PR Summary

This may make WebAgg the most tested backend...

I'm not sure what to do about the screenshot test, as I really want to confirm that the figure is sent to the browser, but browsers differ in how they show the resize grip, which causes te comparison to fail.

PR Checklist

Tests and Styling

  • Has pytest style unit tests (and pytest passes).
  • Is Flake 8 compliant (install flake8-docstrings and run flake8 --docstring-convention=all).

Documentation

  • [n/a] New features are documented, with examples if plot related.
  • [n/a] New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
  • [n/a] API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).
  • [n/a] Documentation is sphinx and numpydoc compliant (the docs should build without error).

@timhoffm

timhoffm commented Aug 3, 2022

Copy link
Copy Markdown
Member

I'm not sure what to do about the screenshot test, as I really want to confirm that the figure is sent to the browser, but browsers differ in how they show the resize grip, which causes te comparison to fail.

Can you mask the resize grip by drawing a white rectangle on top using imagemagic?

@tacaswell

Copy link
Copy Markdown
Member
result = next(iter(done)).result()
E       playwright._impl._api_types.Error: Executable doesn't exist at /home/vsts/.cache/ms-playwright/chromium-1015/chrome-linux/chrome
E       ╔════════════════════════════════════════════════════════════╗
E       ║ Looks like Playwright was just installed or updated.       ║
E       ║ Please run the following command to download new browsers: ║
E       ║                                                            ║
E       ║     playwright install                                     ║
E       ║                                                            ║
E       ║ <3 Playwright Team                                         ║
E       ╚════════════════════════════════════════════════════════════╝

looks like we need some extra steps in the CI setup?

@QuLogic

QuLogic commented Aug 3, 2022

Copy link
Copy Markdown
Member Author

I'm not sure what to do about the screenshot test, as I really want to confirm that the figure is sent to the browser, but browsers differ in how they show the resize grip, which causes te comparison to fail.

Can you mask the resize grip by drawing a white rectangle on top using imagemagic?

Actually, I forgot we have access to which browser is running the test, so I could just have two test images... I'll see what the diff looks like.

@QuLogic

QuLogic commented Oct 5, 2022

Copy link
Copy Markdown
Member Author

So this is nearly working on GitHub Actions. The current failure is on macOS. For some reason, the canvas gets a border and it's black on Linux, but blue on macOS. There's also some slight difference in the resize grip size.

AppVeyor only fails one test, so it's likely a similar deal.

Azure passes on Linux, but fails on macOS and Windows, but on several more tests. I'm not sure why that is yet.

Comment thread .github/workflows/tests.yml Fixed
@QuLogic
QuLogic force-pushed the test-webagg branch 2 times, most recently from fa75d73 to 8b208f8 Compare August 11, 2026 05:55
@QuLogic

QuLogic commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Current status:

  1. Firefox renders the resize grip differently across platforms; I've added a simple edit with Pillow to smooth out those differences.
  2. WebKit seems to be broken on Ubuntu (it generates an empty screenshot, and resizing fails), but the build that Playwright ships requires very old libraries, so I can't test it on Fedora.
  3. WebKit fails on Windows, for interactive tests. The screenshot does appear to pass.
  4. The macOS 14 build seems to get stuck for currently unknown reasons.

On all other platforms, Chromium and Firefox pass all tests.

@ianthomas23

Copy link
Copy Markdown
Member

On web-based projects it is common to do functional tests across the whole matrix of OS and web browser, but only take screenshots on one (linux chrome) to minimise the churn in test images caused by regular browser updates. It should be less of a problem here as the screenshot images are really those generated by Agg, but it is an option.

It might be worth adding some real plotted data to the screenshots, maybe just a line plot, to prove it works.

@QuLogic

QuLogic commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

We did have a case where things broke on a specific browser #18142, so I would like to check all of them if possible. That was of course due to something that was eventually standardized and not likely to be something to worry about in the future, but who knows what else we might use later that is inconsistent.

I did limit the screenshot to just the canvas part that should come from us, to avoid most browser-specific UI (other than the resize grip which has been edited out.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI: Run cibuildwheel Run wheel building tests on a PR topic: testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants