Final remarks on GSoC 2013

September 23, 2013

This is the last post in the last-minute and short-spanned reporting on my GSoC 2013 project. I’ll only write some final thoughts and a bit about what’s still missing when it comes to support for the Wayland display protocol in WebKit2GTK+.


Over the previous posts I’ve described three main objectives I’ve managed to complete. I added a Wayland build target that can be enabled both on its own or in parallel with the X11 target, implemented support for running the large layout test suite under the Weston compositor, and added support for running WebGL content under Wayland displays.


The work was fun, challenging and I also learned a lot. I’m glad I took up this specific task, as I’m pretty much devoted at this moment to the WebKit project and, specifically, the GTK port. A great group of people is producing fascinating work, and that’s valuable to me as it’s something I can look at, examine and learn from it.


Thanks again to Martin Robinson for the mentoring, giving great advice and doing helpful code reviews. He’s immensely smart and gladly shares his knowledge and experience, which I greatly appreciate. Thanks also to everyone else involved in the WebKit2GTK+ project that chipped in with advice.


While the work I’ve done did improve the WebKit2GTK+ experience under Wayland and I did manage to achieve all the goals I’ve set for the GSoC period, there are still some features that need further attention, mainly the accelerated compositing. I’ve put forth some details about this in the Wayland target report blog, outlining the problem and the most likely solution. The plugin support also needs work, and there are most likely other bugs that need polishing before the experience is optimal.


I better get started.


This part of the belated reporting on my GSoC 2013 project covers support for running WebGL content through WebKit2GTK+ under Wayland displays. This was by far the most fun I’ve had during the summer as it was quite challenging, and I’ve also learned much.


The WebGL implementation uses a 3D rendering-specific class that is basically a wrapper around OpenGL ES function calls but also sets up the textures and framebuffers appropriately. That class further relies on an additional class that wraps the OpenGL context as it can be provided by the port-specific code that tries to do its best to create a valid context instance, depending on the underlying system and, of course, the display protocol in use.


A bit on how the context is acquired under X displays. The implementation relies on GLX, and tries to create either a window-, pbuffer- or pixmap-based context. A shared X11 display is used for all operations. A context can only be produced through GLX if running under X displays and if the X11 display support is actually enabled at build-time.


For Wayland the GLX-based context of course won’t be of use – instead, EGL-based context will have to be created. As a part of my work I’ve had to refactor a good part of the current code to provide support for three different code paths that differ in the EGL display that must be used and the context creation process. The first is the default one that relies on the EGL_DEFAULT_DISPLAY and tries to create either a window- or pbuffer-based surface that’s then used to create the context. The second is the X11-specific one that again relies on the shared X11 display and additionally also tries to provide a pixmap-based surface for the context. It’s quite similar to the GLX-based acquiring of the context but ultimately uses the EGL interface.


The new, third path to creating the context is Wayland-specific. The shared display is acquired from the wl_display instance that is provided by the default GdkDisplay (which is guaranteed to be of the GdkWaylandDisplay type, since we’re only choosing this code path when running under Wayland). The Wayland platform of the EGL interface unfortunately only implements window-based surfaces, but that’s not really an option here – a hack that works is creating a dummy 1x1px window, but there’s a better solution in the form of the EGL_KHR_surfaceless_context extension. This allows for creating a context without any surface, and using that context works just as fine. The extension is currently only implemented in Mesa, but you’re pretty much depending on that project if you’re running Wayland at this point.


Rendering of the simple WebGL demo in MiniBrowser under Weston.


The work displayed here is still ongoing but rapidly approaching a finished state. Because of that the WebGL support under Wayland is missing from the forthcoming WebKit2GTK+ 2.2.0, but I hope I’ll be able to convince the maintainers to include it in the 2.2.1 release.

WebKit2GTK+ utilizes the WebKit‘s massive layout test suite to test for any regressions that might happen with the enormous amount of commits flying into the tree. The whole suite contains over 30,000 tests of various types that are run in a custom and tightly controlled environment. The Wayland display protocol support can be of use here as well, so that’s another part to which I’ve dedicated attention during my GSoC 2013 project.


That environment is automatically set up through a Jhbuild moduleset, located in the WebKit2GTK+ trunk, when building developer builds. This way the fonts and icons used in tests are the same across different systems. To further avoid the default systems’ affects on rendering of the Web content, Xvfb is used. Xvfb works well because an actual display and input devices are not required, so it also fits nicely with the CI builders that must run the whole suite without an actual display and input devices being present in the system. Multiple instances of Xvfb are launched and the whole test suite is run in parallel to speed up the testing step.


Xvfb has its shortcomings as well though, as it makes it very hard to test proper rendering of the WebGL content or the accelerated compositing output. Mesa’s offscreen rendering might be a viable solution here, and there were attempts at this but no real results.


In comes Weston. The reference Wayland compositor fits perfectly here. Multiple instances of it can be launched in parallel, each providing proper hardware acceleration, opening the path to testing WebGL, CSS shaders, accelerated compositing output and additional features. To put all these instances under control in any developer’s environment, they can all be grouped under an additional Weston compositor which the developer can then dismiss and focus on other stuff. The obvious downside is that real hardware is required, mainly a proper GPU.


GNOME Shell > Weston > Weston x 3

Screenshot of three Weston instances grouped under the fourth one that’s running under GNOME Shell.


A bit on the implementation – WebKit relies on a large Python framework, dubbed webkitpy, for a large set of modern-day tools (a parallel test runner, patch processing utilities, IRC bots and separate Web applications for test management) used by the developers. There are also older scripts, written in Perl, that are still used today, but are not really important in this context. In the webkitpy’s test runner infrastructure, ports can define their own set of rules when it comes to the environment in which the tests should be run. This basically means that the GTK port must define the DISPLAY environment variable and also properly set up the Xvfb instance. The same applies for the WAYLAND_DISPLAY environment variable and Weston, with the so-called WestonDriver class being chosen over the XvfbDriver class when running the test suite under a Wayland display.


This blog post really covers the very specifics (and only a part of that) of running the layout tests under WebKit2GTK+ (or, again, WebKit1GTK+, even if that framework is moving towards deprecation) and how the Wayland support was added for that scope of the project. I wanted to outline the benefits of using Weston as the for-testing display and the neat hack of keeping all the parallel instances (of which there can be a lot) under control, and I hope I’ve managed just that.

In WebKit2GTK+, a number of targets can be chosen at configure-time – X11, Win32, Quartz, DirectFB. As a result of my work for the GSoC 2013 project, the Wayland build target is now also available.



The obligatory demo screenshot of the MiniBrowser running under Weston

Different targets require different dependencies and also determine which platform-specific set of files must be built. In WebKit2GTK+, the target is chosen when running the configure script, passing the `–with-target` option on the command line. The new Wayland target can be selected along with the X11 target (by executing `./configure –with-target=x11,wayland`) or can be the only chosen target (by executing `./configure –with-target=wayland`). Choosing the Wayland target via the `–with-target` option will be available in the forthcoming 2.2 stable release series of WebKit2GTK+ (and WebKit1GTK+ too, though that technical name is not really used). For future stable releases, I plan to provide multiple `–enable-*-target` configuration options as the current approach involves some nasty hacks to support builds that enable both the X11 and Wayland targets in parallel. The new flags should be coming to trunk relatively soon (if the approach is accepted).

Wayland is under rapid development and with other projects lagging behind with the support for it, WebKit2GTK+ is unfortunately no exception. In order for the basic functionality to work, WebKit2GTK+ under Wayland disables accelerated compositing at run-time (and also at build-time if Wayland is the only chosen target), and disables Netscape plugins entirely.

Accelerated compositing for WebKit2GTK+ under Wayland is quite a hard cookie to crumble due to the multi-process nature of the WebKit2 infrastructure upon which WebKit2GTK+, the GTK port, sits. The crux of the issue is the fact that EGL surfaces cannot be shared between processes. I won’t go too deep into technicalities at this point, but you can brush up on the problem by reading this blog post by Iago Toral. Note that the basic painting of the Web content is not an issue as in this case using shared memory between the two processes is possible.

Plugins, the bane of the Web, are not yet working with WebKit2GTK+ under Wayland due to, again, the multiple process architecture. The Plugin process that’s provided by WebKit2GTK+ is linked against GTK+ 2.0 so plugins like Flash can still be used. At least in the case of the Flash plugin, Adobe concluded any further development apart from security updates, meaning the plugin will never progress towards GTK+ 3.0. I’d be the first to join the revolution and completely disable the Plugin process, but this probably won’t be possible anytime soon. Given that, the Plugin process will have to use GTK+ 2.0 as long as required. With Wayland only supported in the latest releases of GTK+ 3.0, this means that the plugins will most likely have to be run through XWayland.

Even with accelerated compositing and plugins not yet supported, WebKit2GTK+ under Wayland is still functioning quite well. Video playback works well, WebGL support is on its way, and input is properly processed. A great deal of functionality is already assured because of great work being done to support Wayland in the GTK+ toolkit itself.

I’ve been accepted into this year’s Google Summer of Code to work on adding support for running WebKit2GTK+ under the Wayland display protocol. While the work started back in late May and with the hard pencils-down date looming just around the corner, this is actually the very first blog post on the project.


Turns out, I suck at reporting. Don’t worry, I’ve been working on the project throughout the summer and achieved solid results, but I failed completely on providing timely updates on the status and progress of my work. I apologize for that.


WebKit2GTK+, the GTK port of the WebKit2 web engine infrastructure, needed a fair amount of attention so it could properly run under Wayland displays. With the whole GNOME ecosystem moving towards the new cool display protocol, it quickly made sense what to propose as the GSoC project earlier this year. The project got accepted, and Martin Robinson kindly volunteered to be my mentor, and I’m very thankful for all his hints, pointers and reviews.


On Monday the GSoC work period finishes and the final evaluations follow. This being written on Thursday evening, I’m left with three more days I can spend on documenting the project’s results and shortcomings, filling up the project’s Wiki page. Therefor, for the next three days, I’ll write three more posts, presenting three main goals of the project that I’ve managed to achieve (more or less successfully). On Monday, I’ll post the final conclusions and also some information on what bits are still needed to have a complete, smooth experience of WebKit2GTK+ under Wayland displays.

The final report

August 19, 2012

So, the hard-pencils-down date is now just around the corner, and there were only a few small things left to be done.

Nothing radical happened since the last report just a few days ago. A couple of you left some comments that I haven’t yet had time to get to, but I sure will, hopefully soon.

The code for the GNOME Settings Daemon plugin and GNOME Control Center panel was refined a bit and documented where I felt it was required, and pushed into the GitHub forks (GSDGCC) I’ve been using during this period. To mark the end point of the GSoC coding period I’ve created the GSOC_PENCILS_DOWN tags in both repositories (GSDGCC) – there might still be some commits due to the merging of both components into their corresponding repositories at Two bugs were created to cover that effort, one for the GSD plugin and another for the GCC panel.

The design proposal was also updated to reflect the latest intention of the panel functionality. The design itself will most likely be quite important when merging the panel into the main trunk and enabling it.

Keeping it short this time, and covering pretty much everything I had in mind. You’re welcome to check out the project page to see more about how I spent this summer. But do note that the timeline there was at the end not really followed as closely as one would like.

The GSoC period is swiftly coming to its end, with the ‘soft pencils-down’ date already passed. While I’m not yet completely finished with my project I am very close, though, with only some small bits of functionality left to implement and with a quick walk through all the code to sort out the code styling issues and possible memory leaks.

What’s new

Quite some time has passed since the last report and the work has progressed quite a lot in that time. The Gamepad API support in WebKitGTK+ was already complete and pretty polished so the only thing remaining was the implementation of the GNOME Settings Daemon plugin and GNOME System Settings panel. The former’s task would be to recognize supported devices as they are connected and apply to them the correct buttons and axes mappings and previously-stored calibration data. The latter would be used by the users to see if their device is supported and, if the device was well enough supported by the Linux kernel itself, the user could calibrate the device him- or herself.

What changed from the original idea

The most significant change is that originally, I was thinking of giving the users the complete control over the buttons and axes mappings. That would mean that each user would go through an annoying and long process of manually configuring the device. But such processes immediately require the user to understand more about his device and the general gamepad/joystick button and axis layout than it is either required or productive for everyday life. Rather than that the decision was made to spare the user all the hassle around the configuration and rather use preset buttons and axes mappings for specific devices.

This would of course mean that some users might connect a device that the system does not yet recognize as supported. The user could still interact with the device (the system would not block the device’s usage) but could also request that device is awarded the supported status with appropriate mappings assigned to its unique ID. Rather than waiting 6 months for the new release, the data about all the supported devices is stored on the Web and is cached locally and updated whenever a new device is given the supported status and the list of supported devices is expanded. While the Web-located list of supported devices is already working well in the current daemon plugin, the ‘support request’ is, at this moment, only an idea that needs more discussion about approach and implementation.

The internals

The GNOME Settings Daemon plugin was already described in quite some detail. The update of the supported devices list is done when a mismatch in the checksums of the local and Web-located list occurs. The checksums are compared when the daemon starts, and that’s normally when the session begins (at login).  The devices list is written in JSON, so the plugin uses JSON-GLib to parse the list. The device is updated with proper mappings and current calibration through the Linux kernel driver. As much as convenient, these drivers are not quite the holy grail, but more on this later.

The GNOME System Settings panel, as already told, shows the connected devices, their support status, and provides calibration dialog if the device can handle it. The current design is my own, so it’s nothing extra (hopefully in either extreme, good or bad :>), but it has the functionality that it should provide. Here are the two screenshots, first shows the device listing, the second the calibration dialog.

What’s left

The functionality of both the GNOME Settings Daemon plugin and the GNOME System Settings panel is there, all that remains to do in the time left in the GSoC period (three days :>) is to polish the code, document it and start the effort of merging back into the main repositories. The work was done in the mirror repositories I’ve set up on GitHub (the project page has more info on this), with all the work contained in the master branches.

I’ll open two separate bugs for each project in which I’ll upload the merge patches up for review. My mentor Carlos has already volunteered to do some reviewing while the two projects’ maintainers will have a thorough look as well, of course. The design proposal will be updated as well, based on the current panel design. GNOME designers will be invited to have a look, comment on the various approaches and present any design ideas in sketch form.

Hopefully, with a proper design and a couple of devices being supported, the maintainers will give the thumbs-up to enable both the daemon plugin and the control panel in trunk, so they’d make it into GNOME 3.6. If not, there will be enough time to polish both components to near-perfection for the 3.8 release. And talking of polishing things, the device drivers in the Linux kernel need some of that too.

The devices agony

Truth be told, this is partly a rant. And it’s only based on two devices that don’t work properly. I’ve invested into a Wii Remote and a Sony Playstation DualShock3 gamepad to see how they behave with GNOME. They’re both Bluetooth devices, with the DualShock being also connectable through a mini USB cable. Unfortunately, while I’ve had to put quite some effort into connecting the Wiimote, the DualShock stayed reluctant to connect via Bluetooth. But even when connected, the Wiimote wasn’t presented well as a joystick device (the /dev/input/js* file was there, but pretty much useless), neither was the accelerometer propagated as such through GUdev.

The DualShock, when connected through the USB cable, was picked up as a joystick device by the joydev driver, the generic joystick device driver in the kernel. But you’d think it’s quite a special device when it popped out with 27 axes and (if I remember correctly) 18 buttons. A normal gamepad has 6 axes (two of those are the D-pads) and 12 buttons. The DualShock has these neat pressure-sensitive buttons, but the sensor data is exposed as an axis instead of the button value being regulated between 0 and 1 based on that pressure. Also exported as device axes is the accelerometer data, which could (well, should) be exposed as what it is – an accelerometer. There are also a couple of axes that I wasn’t able to locate at all, so I guess that’s sort of a bonus.

While the inability to connect via Bluetooth probably falls under Bluez, it’s obvious DualShock should have a custom driver to handle all its features. There’s a Wiimote-specific driver but I’m not entirely sure of its status in the Linux 3.2 kernel (used when testing), this driver looks OK in Git master, exporting accelerometer data as well, but I haven’t had time yet to test it properly. The Xbox controllers have a specific driver, so they should work in theory but I haven’t tested them really (well, I should buy one first). But it’s the DualShock annoyance that bothers me the most, the device is expensive and advanced as well, but with the current support it turns out to be nothing more than a toy. The device driver wasn’t a planned goal of this GSoC project but it might as well be a late collateral damage.


This blog post has gotten a bit lengthy, so let’s end it. I’ll post another, final report on Sunday just to explain what was left undone and what was actually completed, along with the status of the merge, and where this project could still improve outside of the GSoC scope.

Biweekly report #4

July 17, 2012

Weeks #7 and #8 contained the mid-term evaluation, which I passed (yay!). This event also signaling the mid-point of the project, there are still changes in the approach of the gamepad configuration panel implementation in GNOME Control Center.

The current idea

Currently the design proposal suggests the user could reconfigure the device’s buttons and axes if he or she finds them incorrectly configured (mapped). The mappings would then be saved via GSettings and loaded every time the device would connect to the system. Ideally the user would be able to report to the GNOME Bugzilla that he or she had to go through the configuration process and add to that report the data about the device and its new configuration.

These configurations would then be stored in some kind of a device list that would group the devices based on their axes and buttons layout. If the user would connect an unrecognized device, he or she could search through the different types of devices to find a layout that would suit his or hers device and report that to the GNOME Bugzilla. That device would then be added to the appropriate layout group.

Axis calibration would also be available for those more advanced users.

‘Stupid configuration … WTH?! … That’s it, I’m done with this!’

That’s exactly what some user that’s not that deep into technology and gamepad reconfiguration might think when he or she is forced into something not only time-consuming, but also completely unnecessary.

The new idea is to maintain a list of ‘supported devices’, that is a list of all the axes and buttons mappings for devices that have been tested and should work out-of-the-box. These mappings would then be applied to corresponding devices when they are connected.

In the case of an unsupported device the proper mapping would be reported to the GNOME Bugzilla and the list of supported devices expanded (the first part still needs some planning and work). That would basically mean that no reconfiguration would be either required or available for any gamepad device. The axis calibration would still be possible.

TODO in the second half of the GSOC period

With the Gamepad API support in WebKitGTK+ pretty much complete (by the way, the 1.9.5 release is out and you can test the feature there), the only thing to do is the GNOME Control Center panel implementation. The GNOME Settings Daemon plugin might become redundant in implementing the current idea, but that’s still to see.

The design proposal will be updated and represented to the GNOME designers, hopefully discussed and started being implemented. I’ll try to do the implementation in the tip-of-tree (jhbuild here I come) instead of doing it in the 3.4 branch and the porting it forward.

Biweekly report #3

July 4, 2012

Weeks #5 and #6 were quite fun and productive. Here’s what was happening.

Support for Gamepad API landed

The support for Gamepad API for the Gtk port has landed in WebKit. The feature is currently disabled at compile-time due to the specification still being a work in progress. This might change in the future if API for enabling experimental features at run-time is introduced for WebKitGTK+. Currently there’s no release version (stable or unstable) available that includes the support, but the next unstable release (1.9.5) will have it as well as the next stable release series. Build instructions for the impatient are found on the project’s page in the Wiki (will be updated soon).

GNOME Settings Daemon plugin almost done

The GNOME Settings Daemon plugin is pretty much finished. The code (available on GitHub) is based on the 3.4 branch and will go through a review by my mentor, Carlos, and will eventually be rebased on the master branch. The plugin’s task is to apply any previously saved configurations to the device on its connection. Currently the axes and buttons mappings are applied, but the axes calibration data will be applied as well (support for this will be added parallel with support in the Control Center panel). Currently saving the configuration data is only available through the command line, using the gsettings application’s capabilities. This is extremely hacky approach though, and not a pleasant user experience.

Next two weeks

The next two weeks will be mostly focused on implementing the panel in GNOME Control Center. The code is available on GitHub. Also in this timespan is the mid-term evaluation.

Biweekly report #2

June 19, 2012

Weeks #3 and #4 have flown past, and again more could have been done. Student finals are currently getting in the way of productivity. Studying is no fun when you could be spending the time on a really cool project, and unfortunately I have a few more exams in the next two weeks, meaning I won’t be able to commit much more time to getting things done compared to these last two weeks.

Shifting priorities

It’s because of that I’ve decided to shuffle the expected timeline a bit, focusing for the next two weeks on getting the Gamepad API support finally landed in WebKit and implement the GNOME Settings Daemon plugin for storing the configured gamepad data. The GNOME System Settings panel will have to wait a bit, but hopefully I’ll still make some progress on its design in the next two weeks.

Review of the last two weeks

Patch for adding Gamepad API support was updated and now requires only some minor details to be polished before it can be landed.

Basic infrastructure implementations for the GNOME Settings Daemon plugin and GNOME System Settings panel have been created. Currently, the implementations are based on the ‘gnome-3-4’ branches – this makes it simple for pretty much anyone with the latest stable GNOME system to try them out. It also aids me as I don’t have to deal with the mess that can result when using JHBuild. This is of course just temporary, the implementations will move onto a 3.6 codebase once there are some decent GNOME 3.5 releases (I’m currently bound to Ubuntu and therefor would wait for some later 12.10 Alpha releases). Again, this is simply just to avoid the JHBuild – both Settings Daemon and System Settings (the Control Center) are large enough projects to require some extraordinary dependencies, making a successful compilation a matter of luck.

Both repositories are available on GitHub – GNOME Settings Daemon and GNOME System Settings.