WebKit2GTK+ regression testing under Weston

WebKit2GTK+ utilizes the WebKit‘s massive layout test suite to test for any regressions that might happen with the enormous amount of commits flying into the tree. The whole suite contains over 30,000 tests of various types that are run in a custom and tightly controlled environment. The Wayland display protocol support can be of use here as well, so that’s another part to which I’ve dedicated attention during my GSoC 2013 project.

 

That environment is automatically set up through a Jhbuild moduleset, located in the WebKit2GTK+ trunk, when building developer builds. This way the fonts and icons used in tests are the same across different systems. To further avoid the default systems’ affects on rendering of the Web content, Xvfb is used. Xvfb works well because an actual display and input devices are not required, so it also fits nicely with the CI builders that must run the whole suite without an actual display and input devices being present in the system. Multiple instances of Xvfb are launched and the whole test suite is run in parallel to speed up the testing step.

 

Xvfb has its shortcomings as well though, as it makes it very hard to test proper rendering of the WebGL content or the accelerated compositing output. Mesa’s offscreen rendering might be a viable solution here, and there were attempts at this but no real results.

 

In comes Weston. The reference Wayland compositor fits perfectly here. Multiple instances of it can be launched in parallel, each providing proper hardware acceleration, opening the path to testing WebGL, CSS shaders, accelerated compositing output and additional features. To put all these instances under control in any developer’s environment, they can all be grouped under an additional Weston compositor which the developer can then dismiss and focus on other stuff. The obvious downside is that real hardware is required, mainly a proper GPU.

 

Screenshot of three Weston instances grouped under the fourth one that’s running under GNOME Shell.

 

A bit on the implementation – WebKit relies on a large Python framework, dubbed webkitpy, for a large set of modern-day tools (a parallel test runner, patch processing utilities, IRC bots and separate Web applications for test management) used by the developers. There are also older scripts, written in Perl, that are still used today, but are not really important in this context. In the webkitpy’s test runner infrastructure, ports can define their own set of rules when it comes to the environment in which the tests should be run. This basically means that the GTK port must define the DISPLAY environment variable and also properly set up the Xvfb instance. The same applies for the WAYLAND_DISPLAY environment variable and Weston, with the so-called WestonDriver class being chosen over the XvfbDriver class when running the test suite under a Wayland display.

 

This blog post really covers the very specifics (and only a part of that) of running the layout tests under WebKit2GTK+ (or, again, WebKit1GTK+, even if that framework is moving towards deprecation) and how the Wayland support was added for that scope of the project. I wanted to outline the benefits of using Weston as the for-testing display and the neat hack of keeping all the parallel instances (of which there can be a lot) under control, and I hope I’ve managed just that.