Goodbye Mozilla

Today is effectively my last day at Mozilla, before I start at Impossible on Monday. I’ve been here for 6 years and a bit and it’s been quite an experience. I think it’s worth reflecting on, so here we go; Fair warning, if you have no interest in me or Mozilla, this is going to make pretty boring reading.

I started on June 6th 2011, several months before the (then new, since moved) London office opened. Although my skills lay (lie?) in user interface implementation, I was hired mainly for my graphics and systems knowledge. Mozilla was in the region of 500 or so employees then I think, and it was an interesting time. I’d been working on the code-base for several years prior at Intel, on a headless backend that we used to build a Clutter-based browser for Moblin netbooks. I wasn’t completely unfamiliar with the code-base, but it still took a long time to get to grips with. We’re talking several million lines of code with several years of legacy, in a language I still consider myself to be pretty novice at (C++).

I started on the mobile platform team, and I would consider this to be my most enjoyable time at the company. The mobile platform team was a multi-discipline team that did general low-level platform work for the mobile (Android and Meego) browser. When we started, the browser was based on XUL and was multi-process. Mobile was often the breeding ground for new technologies that would later go on to desktop. It wasn’t long before we started developing a new browser based on a native Android UI, removing XUL and relegating Gecko to page rendering. At the time this felt like a disappointing move. The reason the XUL-based browser wasn’t quite satisfactory was mainly due to performance issues, and as a platform guy, I wanted to see those issues fixed, rather than worked around. In retrospect, this was absolutely the right decision and lead to what I’d still consider to be one of Android’s best browsers.

Despite performance issues being one of the major driving forces for making this move, we did a lot of platform work at the time too. As well as being multi-process, the XUL browser had a compositor system for rendering the page, but this wasn’t easily portable. We ended up rewriting this, first almost entirely in Java (which was interesting), then with the rendering part of the compositor in native code. The input handling remained in Java for several years (pretty much until FirefoxOS, where we rewrote that part in native code, then later, switched Android over).

Most of my work during this period was based around improving performance (both perceived and real) and fluidity of the browser. Benoit Girard had written an excellent tiled rendering framework that I polished and got working with mobile. On top of that, I worked on progressive rendering and low precision rendering, which combined are probably the largest body of original work I’ve contributed to the Mozilla code-base. Neither of them are really active in the code-base at the moment, which shows how good a job I didn’t do maintaining them, I suppose.

Although most of my work was graphics-focused on the platform team, I also got to to do some layout work. I worked on some over-invalidation issues before Matt Woodrow’s DLBI work landed (which nullified that, but I think that work existed in at least one release). I also worked a lot on fixed position elements staying fixed to the correct positions during scrolling and zooming, another piece of work I was quite proud of (and probably my second-biggest contribution). There was also the opportunity for some UI work, when it intersected with platform. I implemented Firefox for Android’s dynamic toolbar, and made sure it interacted well with fixed position elements (some of this work has unfortunately been undone with the move from the partially Java-based input manager to the native one). During this period, I was also regularly attending and presenting at FOSDEM.

I would consider my time on the mobile platform team a pretty happy and productive time. Unfortunately for me, those of us with graphics specialities on the mobile platform team were taken off that team and put on the graphics team. I think this was the start in a steady decline in my engagement with the company. At the time this move was made, Mozilla was apparently trying to consolidate teams around products, and this was the exact opposite happening. The move was never really explained to me and I know I wasn’t the only one that wasn’t happy about it. The graphics team was very different to the mobile platform team and I don’t feel I fit in as well. It felt more boisterous and less democratic than the mobile platform team, and as someone that generally shies away from arguments and just wants to get work done, it was hard not to feel sidelined slightly. I was also quite disappointed that people didn’t seem particular familiar with the graphics work I had already been doing and that I was tasked, at least initially, with working on some very different (and very boring) desktop Linux work, rather than my speciality of mobile.

I think my time on the graphics team was pretty unproductive, with the exception of the work I did on b2g, improving tiled rendering and getting graphics memory-mapped tiles working. This was particularly hard as the interface was basically undocumented, and its implementation details could vary wildly depending on the graphics driver. Though I made a huge contribution to this work, you won’t see me credited in the tree unfortunately. I’m still a little bit sore about that. It wasn’t long after this that I requested to move to the FirefoxOS systems front-end team. I’d been doing some work there already and I’d long wanted to go back to doing UI. It felt like I either needed a dramatic change or I needed to leave. I’m glad I didn’t leave at this point.

Working on FirefoxOS was a blast. We had lots of new, very talented people, a clear and worthwhile mission, and a new code-base to work with. I worked mainly on the home-screen, first with performance improvements, then with added features (app-grouping being the major one), then with a hugely controversial and probably mismanaged (on my part, not my manager – who was excellent) rewrite. The rewrite was good and fixed many of the performance problems of what it was replacing, but unfortunately also removed features, at least initially. Turns out people really liked the app-grouping feature.

I really enjoyed my time working on FirefoxOS, and getting a nice clean break from platform work, but it was always bitter-sweet. Everyone working on the project was very enthusiastic to see it through and do a good job, but it never felt like upper management’s focus was in the correct place. We spent far too much time kowtowing to the desires of phone carriers and trying to copy Android and not nearly enough time on basic features and polish. Up until around v2.0 and maybe even 2.2, the experience of using FirefoxOS was very rough. Unfortunately, as soon as it started to show some promise and as soon as we had freedom from carriers to actually do what we set out to do in the first place, the project was cancelled, in favour of the whole Connected Devices IoT debacle.

If there was anything that killed morale for me more than my unfortunate time on the graphics team, and more than having FirefoxOS prematurely cancelled, it would have to be the Connected Devices experience. I appreciate it as an opportunity to work on random semi-interesting things for a year or so, and to get some entrepreneurship training, but the mismanagement of that whole situation was pretty epic. To take a group of hundreds of UI-focused engineers and tell them that, with very little help, they should organised themselves into small teams and create IoT products still strikes me as an idea so crazy that it definitely won’t work. Certainly not the way we did it anyway. The idea, I think, was that we’d be running several internal start-ups and we’d hopefully get some marketable products out of it. What business a not-for-profit company, based primarily on doing open-source, web-based engineering has making physical, commercial products is questionable, but it failed long before that could be considered.

The process involved coming up with an idea, presenting it and getting approval to run with it. You would then repeat this approval process at various stages during development. It was, however, very hard to get approval for enough resources (both time and people) to finesse an idea long enough to make it obviously a good or bad idea. That aside, I found it very demoralising to not have the opportunity to write code that people could use. I did manage it a few times, in spite of what was happening, but none of this work I would consider myself particularly proud of. Lots of very talented people left during this period, and then at the end of it, everyone else was laid off. Not a good time.

Luckily for me and the team I was on, we were moved under the umbrella of Emerging Technologies before the lay-offs happened, and this also allowed us to refocus away from trying to make an under-featured and pointless shopping-list assistant and back onto the underlying speech-recognition technology. This brings us almost to present day now.

The DeepSpeech speech recognition project is an extremely worthwhile project, with a clear mission, great promise and interesting underlying technology. So why would I leave? Well, I’ve practically ended up on this team by a series of accidents and random happenstance. It’s been very interesting so far, I’ve learnt a lot and I think I’ve made a reasonable contribution to the code-base. I also rewrote python_speech_features in C for a pretty large performance boost, which I’m pretty pleased with. But at the end of the day, it doesn’t feel like this team will miss me. I too often spend my time finding work to do, and to be honest, I’m just not interested enough in the subject matter to make that work long-term. Most of my time on this project has been spent pushing to open it up and make it more transparent to people outside of the company. I’ve added model exporting, better default behaviour, a client library, a native client, Python bindings (+ example client) and most recently, Node.js bindings (+ example client). We’re starting to get noticed and starting to get external contributions, but I worry that we still aren’t transparent enough and still aren’t truly treating this as the open-source project it is and should be. I hope the team can push further towards this direction without me. I think it’ll be one to watch.

Next week, I start working at a new job doing a new thing. It’s odd to say goodbye to Mozilla after 6 years. It’s not easy, but many of my peers and colleagues have already made the jump, so it feels like the right time. One of the big reasons I’m moving, and moving to Impossible specifically, is that I want to get back to doing impressive work again. This is the largest regret I have about my time at Mozilla. I used to blog regularly when I worked at OpenedHand and Intel, because I was excited about the work we were doing and I thought it was impressive. This wasn’t just youthful exuberance (he says, realising how ridiculous that sounds at 32), I still consider much of the work we did to be impressive, even now. I want to be doing things like that again, and it feels like Impossible is a great opportunity to make that happen. Wish me luck!

Machine Learning Speech Recognition

Keeping up my yearly blogging cadence, it’s about time I wrote to let people know what I’ve been up to for the last year or so at Mozilla. People keeping up would have heard of the sad news regarding the Connected Devices team here. While I’m sad for my colleagues and quite disappointed in how this transition period has been handled as a whole, thankfully this hasn’t adversely affected the Vaani project. We recently moved to the Emerging Technologies team and have refocused on the technical side of things, a side that I think most would agree is far more interesting, and also far more suited to Mozilla and our core competence.

Project DeepSpeech

So, out with Project Vaani, and in with Project DeepSpeech (name will likely change…) – Project DeepSpeech is a machine learning speech-to-text engine based on the Baidu Deep Speech research paper. We use a particular layer configuration and initial parameters to train a neural network to translate from processed audio data to English text. You can see roughly how we’re progressing with that here. We’re aiming for a 10% Word Error Rate (WER) on English speech at the moment.

You may ask, why bother? Google and others provide state-of-the-art speech-to-text in multiple languages, and in many cases you can use it for free. There are multiple problems with existing solutions, however. First and foremost, most are not open-source/free software (at least none that could rival the error rate of Google). Secondly, you cannot use these solutions offline. Third, you cannot use these solutions for free in a commercial product. The reason a viable free software alternative hasn’t arisen is mostly down to the cost and restrictions around training data. This makes the project a great fit for Mozilla as not only can we use some of our resources to overcome those costs, but we can also use the power of our community and our expertise in open source to provide access to training data that can be used openly. We’re tackling this issue from multiple sides, some of which you should start hearing about Real Soon Now™.

The whole team has made contributions to the main code. In particular, I’ve been concentrating on exporting our models and writing clients so that the trained model can be used in a generic fashion. This lets us test and demo the project more easily, and also provides a lower barrier for entry for people that want to try out the project and perhaps make contributions. One of the great advantages of using TensorFlow is how relatively easy it makes it to both understand and change the make-up of the network. On the other hand, one of the great disadvantages of TensorFlow is that it’s an absolute beast to build and integrates very poorly with other open-source software projects. I’ve been trying to overcome this by writing straight-forward documentation, and hopefully in the future we’ll be able to distribute binaries and trained models for multiple platforms.

Getting Involved

We’re still at a fairly early stage at the moment, which means there are many ways to get involved if you feel so inclined. The first thing to do, in any case, is to just check out the project and get it working. There are instructions provided in READMEs to get it going, and fairly extensive instructions on the TensorFlow site on installing TensorFlow. It can take a while to install all the dependencies correctly, but at least you only have to do it once! Once you have it installed, there are a number of scripts for training different models. You’ll need a powerful GPU(s) with CUDA support (think GTX 1080 or Titan X), a lot of disk space and a lot of time to train with the larger datasets. You can, however, limit the number of samples, or use the single-sample dataset (LDC93S1) to test simple code changes or behaviour.

One of the fairly intractable problems about machine learning speech recognition (and machine learning in general) is that you need lots of CPU/GPU time to do training. This becomes a problem when there are so many initial variables to tweak that can have dramatic effects on the outcome. If you have the resources, this is an area that you can very easily help with. What kind of results do you get when you tweak dropout slightly? Or layer sizes? Or distributions? What about when you add or remove layers? We have fairly powerful hardware at our disposal, and we still don’t have conclusive results about the affects of many of the initial variables. Any testing is appreciated! The Deep Speech 2 paper is a great place to start for ideas if you’re already experienced in this field. Note that we already have a work-in-progress branch implementing some of these ideas.

Let’s say you don’t have those resources (and very few do), what else can you do? Well, you can still test changes on the LDC93S1 dataset, which consists of a single sample. You won’t be able to effectively tweak initial parameters (as unsurprisingly, a dataset of a single sample does not represent the behaviour of a dataset with many thousands of samples), but you will be able to test optimisations. For example, we’re experimenting with model quantisation, which will likely be one of multiple optimisations necessary to make trained models usable on mobile platforms. It doesn’t particularly matter how effective the model is, as long as it produces consistent results before and after quantisation. Any optimisation that can be made to reduce the size or the processor requirement of training and using the model is very valuable. Even small optimisations can save lots of time when you start talking about days worth of training.

Our clients are also in a fairly early state, and this is another place where contribution doesn’t require expensive hardware. We have two clients at the moment. One written in Python that takes advantage of TensorFlow serving, and a second that uses TensorFlow’s native C++ API. This second client is the beginnings of what we hope to be able to run on embedded hardware, but it’s very early days right now.

And Finally

Imagine a future where state-of-the-art speech-to-text is available, for free (in cost and liberty), on even low-powered devices. It’s already looking like speech is going to be the next frontier of human-computer interaction, and currently it’s a space completely tied up by entities like Google, Amazon, Microsoft and IBM. Putting this power into everyone’s hands could be hugely transformative, and it’s great to be working towards this goal, even in a relatively modest capacity. This is the vision, and I look forward to helping make it a reality.

State of Embedding in Gecko

Following up from my last post, I’ve had some time to research and assess the current state of embedding Gecko. This post will serve as a (likely incomplete) assessment of where we are today, and what I think the sensible path forward would be. Please note that these are my personal opinions and not those of Mozilla. Mozilla are gracious enough to employ me, but I don’t yet get to decide on our direction 😉

The TLDR; there are no first-class Gecko embedding solutions as of writing.

EmbedLite (aka IPCLite)

EmbedLite is an interesting solution for embedding Gecko that relies on e10s (Electrolysis, Gecko’s out-of-process feature code-name) and OMTC (Off-Main-Thread Compositing). From what I can tell, the embedding app creates a new platform-specific compositor object that attaches to a window, and with e10s, a separate process is spawned to handle the brunt of the work (rendering the site, running JS, handling events, etc.). The existing widget API is exposed via IPC, which allows you to synthesise events, handle navigation, etc. This builds using the xulrunner application target, which unfortunately no longer exists. This project was last synced with Gecko on April 2nd 2015 (the day before my birthday!).

The most interesting thing about this project is how much code it reuses in the tree, and how little modification is required to support it (almost none – most of the changes are entirely reasonable, even outside of an embedding context). That we haven’t supported this effort seems insane to me, especially as it’s been shipping for a while as the basis for the browser in the (now defunct?) Jolla smartphone.

Building this was a pain, on Fedora 22 I was not able to get the desktop Qt build to compile, even after some effort, but I was able to compile the desktop Gtk build (trivial patches required). Unfortunately, there’s no support code provided for the Gtk version and I don’t think it’s worth the time me implementing that, given that this is essentially a dead project. A huge shame that we missed this opportunity, this would have been a good base for a lightweight, relatively easily maintained embedding solution. The quality of the work done on this seems quite high to me, after a brief examination.

Spidernode

Spidernode is a port of Node.js that uses Gecko’s ‘spidermonkey’ JavaScript engine instead of Chrome’s V8. Not really a Gecko embedding solution, but certainly something worth exploring as a way to enable more people to use Mozilla technology. Being a much smaller project, of much more limited scope, I had no issues building and testing this.

Node.js using spidermonkey ought to provide some interesting advantages over a V8-based Node. Namely, modern language features, asm.js (though I suppose this will soon be supplanted by WebAssembly) and speed. Spidernode is unfortunately unmaintained since early 2012, but I thought it would be interesting to do a simple performance test. Using the (very flawed) technique detailed here, I ran a few quick tests to compare an old copy of Node I had installed (~0.12), current stable Node (4.3.2) and this very old (~0.5) Spidermonkey-based Node. Spidermonkey-based Node was consistently over 3x faster than both old and current (which varied very little in performance). I don’t think you can really draw any conclusions than this, other than that it’s an avenue worth exploring.

Many new projects are prototyped (and indeed, fully developed) in Node.js these days; particularly Internet-Of-Things projects. If there’s the potential for these projects to run faster, unchanged, this seems like a worthy project to me. Even forgetting about the advantages of better language support. It’s sad to me that we’re experimenting with IoT projects here at Mozilla and so many of these experiments don’t promote our technology at all. This may be an irrational response, however.

GeckoView

GeckoView is the only currently maintained embedding solution for Gecko, and is Android-only. GeckoView is an Android project, split out of Firefox for Android and using the same interfaces with Gecko. It provides an embeddable widget that can be used instead of the system-provided WebView. This is not a first-class project from what I can tell, there are many bugs and many missing features, as its use outside of Firefox for Android is not considered a priority. Due to this dependency, however, one would assume that at least GeckoView will see updates for the foreseeable future.

I’d experimented with this in the past, specifically with this project that uses GeckoView with Cordova. I found then that the experience wasn’t great, due to the huge size of the GeckoView library and the numerous bugs, but this was a while ago and YMMV. Some of those bugs were down to GeckoView not using the shared APZC, a bug which has since been fixed, at least for Nightly builds. The situation may be better now than it was then.

The Future

This post is built on the premise that embedding Gecko is a worthwhile pursuit. Others may disagree about this. I’ll point to my previous post to list some of the numerous opportunities we missed, partly because we don’t have an embedding story, but I’m going to conjecture as to what some of our next missed opportunities might be.

IoT is generating a lot of buzz at the moment. I’m dubious that there’s much decent consumer use of IoT, at least that people will get excited about as opposed to property developers, but if I could predict trends, I’d have likely retired rich already. Let’s assume that consumer IoT will take off, beyond internet-connected thermostats (which are actually pretty great) and metered utility boxes (which I would quite like). These devices are mostly bespoke hardware running random bits and bobs, but an emerging trend seems to be Node.js usage. It might be important for Mozilla to provide an easily deployed out-of-the-box solution here. As our market share diminishes, so does our test-bed and contribution base for our (currently rather excellent) JavaScript engine. While we don’t have an issue here at the moment, if we find that a huge influx of diverse, resource-constrained devices starts running V8 and only V8, we may eventually find it hard to compete. It could easily be argued that it isn’t important for our solution to be based on our technology, but I would argue that if we have to start employing a considerable amount of people with no knowledge of our platform, our platform will suffer. By providing a licensed out-of-the-box solution, we could also enforce that any client-side interface remain network-accessible and cross-browser compatible.

A less tenuous example, let’s talk about VR. VR is also looking like it might finally break out into the mid/high-end consumer realm this year, with heavy investment from Facebook (via Oculus), Valve/HTC (SteamVR/Vive), Sony (Playstation VR), Microsoft (HoloLens), Samsung (GearVR) and others. Mozilla are rightly investing in WebVR, but I think the real end-goal for VR is an integrated device with no tether (certainly Microsoft and Samsung seem to agree with me here). So there may well be a new class of device on the horizon, with new kinds of browsers and ways of experiencing and integrating the web. Can we afford to not let people experiment with our technology here? I love Mozilla, but I have serious doubts that the next big thing in VR is going to come from us. That there’s no supported way of embedding Gecko worries me for future classes of device like this.

In-vehicle information/entertainment systems are possibly something that will become more of the norm, now that similar devices have become such commodity. Interestingly, the current big desktop and mobile players have very little presence here, and (mostly awful) bespoke solutions are rife. Again, can we afford to make our technology inaccessible to the people that are experimenting in this area? Is having just a good desktop browser enough? Can we really say that’s going to remain how people access the internet for the next 10 years? Probably, but I wouldn’t want to bet everything on that.

A plan

If we want an embedding solution, I think the best way to go about it is to start from Firefox for Android. Due to the way Android used to require its applications to interface with native code, Firefox for Android is already organised in such a way that it is basically an embedding API (thus GeckoView). From this point, I think we should make some of the interfaces slightly more generic and remove the JNI dependency from the Gecko-side of the code. Firefox for Android would be the main consumer of this API and would guarantee that it’s maintained. We should allow for it to be built on Linux, Mac and Windows and provide the absolute minimum harness necessary to allow for it to be tested. We would make no guarantees about API or ABI. Externally to the Gecko tree, I would suggest that we start, and that the community maintain, a CEF-compatible library, at least at the API level, that would be a Tier-3 project, much like Firefox OS now is. This, to me, seems like the minimal-effort and most useful way of allowing embeddable Gecko.

In addition, I think we should spend some effort in maintaining a fork of Node.js LTS that uses spidermonkey. If we can promise modern language features and better performance, I expect there’s a user-base that would be interested in this. If there isn’t, fair enough, but I don’t think current experiments have had enough backing to ascertain this.

I think that both of these projects are important, so that we can enable people outside of Mozilla to innovate using our technology, and by osmosis, become educated about our mission and hopefully spread our ideals. Other organisations will do their utmost to establish a monopoly in any new emerging market, and I think it’s a shame that we have such a powerful and comprehensive technology platform and we aren’t enabling other people to use it in more diverse situations.

This post is some insightful further reading on roughly the same topic.

The case for an embeddable Gecko

Strap yourself in, this is a long post. It should be easy to skim, but the history may be interesting to some. I would like to make the point that, for a web rendering engine, being embeddable is a huge opportunity, how Gecko not being easily embeddable has meant we’ve missed several opportunities over the last few years, and how it would still be advantageous to make Gecko embeddable.

What?

Embedding Gecko means making it easy to use Gecko as a rendering engine in an arbitrary 3rd party application on any supported platform, and maintaining that support. An embeddable Gecko should make very few constraints on the embedding application and should not include unnecessary resources.

Examples

  • A 3rd party browser with a native UI
  • A game’s embedded user manual
  • OAuth authentication UI
  • A web application
  • ???

Why?

It’s hard to predict what the next technology trend will be, but there’s is a strong likelihood it’ll involve the web, and there’s a possibility it may not come from a company/group/individual with an existing web rendering engine or particular allegiance. It’s important for the health of the web and for Mozilla’s continued existence that there be multiple implementations of web standards, and that there be real competition and a balanced share of users of the various available engines.

Many technologies have emerged over the last decade or so that have incorporated web rendering or web technologies that could have leveraged Gecko;

(2007) iPhone: Instead of using an existing engine, Apple forked KHTML in 2002 and eventually created WebKit. They did investigate Gecko as an alternative, but forking another engine with a cleaner code-base ended up being a more viable route. Several rival companies were also interested in and investing in embeddable Gecko (primarily Nokia and Intel). WebKit would go on to be one of the core pieces of the first iPhone release, which included a better mobile browser than had ever been seen previously.

(2008) Chrome: Google released a WebKit-based browser that would eventually go on to eat a large part of Firefox’s user base. Chrome was initially praised for its speed and light-weightedness, but much of that was down to its multi-process architecture, something made possible by WebKit having a well thought-out embedding capability and API.

(2008) Android: Android used WebKit for its built-in browser and later for its built-in web-view. In recent times, it has switched to Chromium, showing they aren’t adverse to switching the platform to a different/better technology, and that a better embedding story can benefit a platform (Android’s built in web view can now be updated outside of the main OS, and this may well partly be thanks to Chromium’s embedding architecture). Given the quality of Android’s initial WebKit browser and WebView (which was, frankly, awful until later revisions of Android Honeycomb, and arguably remained awful until they switched to Chromium), it’s not much of a leap to think they may have considered Gecko were it easily available.

(2009) WebOS: Nothing came of this in the end, but it perhaps signalled the direction of things to come. WebOS survived and went on to be the core of LG’s Smart TV, one of the very few real competitors in that market. Perhaps if Gecko was readily available at this point, we would have had a large head start on FirefoxOS?

(2009) Samsung Smart TV: Also available in various other guises since 2007, Samsung’s Smart TV is certainly the most popular smart TV platform currently available. It appears Samsung built this from scratch in-house, but it includes many open-source projects. It’s highly likely that they would have considered a Gecko-based browser if it were possible and available.

(2011) PhantomJS: PhantomJS is a headless, scriptable browser, useful for testing site behaviour and performance. It’s used by several large companies, including Twitter, LinkedIn and Netflix. Had Gecko been more easily embeddable, such a product may well have been based on Gecko and the benefits of that would be many sites that use PhantomJS for testing perhaps having better rendering and performance characteristics on Gecko-based browsers. The demand for a Gecko-based alternative is high enough that a similar project, SlimerJS, based on Gecko was developed and released in 2013. Due to Gecko’s embedding deficiencies though, SlimerJS is not truly headless.

(2011) WIMM One: The first truly capable smart-watch, which generated a large buzz when initially released. WIMM was based on a highly-customised version of Android, and ran software that was compatible with Android, iOS and BlackBerryOS. Although it never progressed past the development kit stage, WIMM was bought by Google in 2012. It is highly likely that WIMM’s work forms the base of the Android Wear platform, released in 2014. Had something like WebOS been open, available and based on Gecko, it’s not outside the realm of possibility that this could have been Gecko based.

(2013) Blink: Google decide to fork WebKit to better build for their own uses. Blink/Chromium quickly becomes the favoured rendering engine for embedding. Google were not afraid to introduce possible incompatibility with WebKit, but also realised that embedding is an important feature to maintain.

(2014) Android Wear: Android specialised to run on watch hardware. Smart watches have yet to take off, and possibly never will (though Pebble seem to be doing alright, and every major consumer tech product company has launched one), but this is yet another area where Gecko/Mozilla have no presence. FirefoxOS may have lead us to have an easy presence in this area, but has now been largely discontinued.

(2014) Atom/Electron: Github open-sources and makes available its web-based text editor, which it built on a home-grown platform of Node.JS and Chromium, which it later called Electron. Since then, several large and very successful projects have been built on top of it, including Slack and Visual Studio Code. It’s highly likely that such diverse use of Chromium feeds back into its testing and development, making it a more robust and performant engine, and importantly, more widely used.

(2016) Brave: Former Mozilla co-founder and CTO heads a company that makes a new browser with the selling point of blocking ads and tracking by default, and doing as much as possible to protect user privacy and agency without breaking the web. Said browser is based off of Chromium, and on iOS, is a fork of Mozilla’s own WebKit-based Firefox browser. Brendan says they started based off of Gecko, but switched because it wasn’t capable of doing what they needed (due to an immature embedding API).

Current state of affairs

Chromium and V8 represent the state-of-the-art embeddable web rendering engine and JavaScript engine and have wide and varied use across many platforms. This helps reenforce Chrome’s behaviour as the de-facto standard and gradually eats away at the market share of competing engines.

WebKit is the only viable alternative for an embeddable web rendering engine and is still quite commonly used, but is generally viewed as a less up-to-date and less performant engine vs. Chromium/Blink.

Spidermonkey is generally considered to be a very nice JavaScript engine with great support for new EcmaScript features and generally great performance, but due to a rapidly changing API/ABI, doesn’t challenge V8 in terms of its use in embedded environments. Node.js is likely the largest user of embeddable V8, and is favoured even by Mozilla employees for JavaScript-based systems development.

Gecko has limited embedding capability that is not well-documented, not well-maintained and not heavily invested in. I say this with the utmost respect for those who are working on it; this is an observation and a criticism of Mozilla’s priorities as an organisation. We have at various points in history had embedding APIs/capabilities, but we have either dropped them (gtkmozembed) or let them bit-rot (IPCLite). We do currently have an embedding widget for Android that is very limited in capability when compared to the default system WebView.

Plea

It’s not too late. It’s incredibly hard to predict where technology is going, year-to-year. It was hard to predict, prior to the iPhone, that Nokia would so spectacularly fall from the top of the market. It was hard to predict when Android was released that it would ever overtake iOS, or even more surprisingly, rival it in quality (hard, but not impossible). It was hard to predict that WebOS would form the basis of a major competing Smart TV several years later. I think the examples of our missed opportunities are also good evidence that opening yourself up to as much opportunity as possible is a good indicator of future success.

If we want to form the basis of the next big thing, it’s not enough to be experimenting in new areas. We need to enable other people to experiment in new areas using our technology. Even the largest of companies have difficulty predicting the future, or taking charge of it. This is why it’s important that we make easily-embeddable Gecko a reality, and I plead with the powers that be that we make this higher priority than it has been in the past.

Firefox for Android in 2013

Lucas Rocha and I gave a talk at FOSDEM over the weekend on Firefox for Android. It went ok, I think we could have rehearsed it a bit better, but it was generally well-received and surprisingly well-attended! I’m sure Lucas will have the slides up soon too. If you were unfortunate enough not to have attended FOSDEM, and doubly unfortunate that you missed our talk (guffaw), we’ll be reiterating it with a bit more detail in the London Mozilla space on February 22nd. We’ll do our best to answer any questions you have about Firefox for Android, but also anything Mozilla-related. If you’re interested in FirefoxOS, there may be a couple of phones knocking about too. Do come along, we’re looking forward to seeing you 🙂

p.s. I’ll be talking on a performance panel at EdgeConf this Saturday. Though it’s fully booked, I think tickets occasionally become available again, so might be worth keeping an eye on. They’ll be much cleverer people than me knocking about, but I’ll be doing my best to answer your platform performance related questions.

Progressive Tile Rendering

So back from layout into graphics again! For the last few weeks, I’ve been working with Benoit Girard on getting progressive tile rendering finished and turned on by default in Firefox for Android. The results so far are very promising! First, a bit of background (feel free to skip to the end if you just want the results).

You may be aware that we use a multi-threaded application model for Firefox for Android. The UI runs in one thread and Gecko, which does the downloading and rendering of the page, runs in another. This is a bit of a simplification, but for all intents and purposes, that’s how it works. We do this so that we can maintain interactive performance – something of paramount important with a touch-screen. We render a larger area than you see on the screen, so that when you scroll, we can respond immediately without having to wait for Gecko to render more. We try to tell Gecko to render the most relevant area next and we hope that it returns in time so that the appearance is seamless.

There are two problems with this as it stands, though. If the work takes too long, you’ll be staring at a blank area (well, this isn’t quite true either, we do a low-resolution render of the entire page and use that as a backing in this worst-case scenario – but that often doesn’t work quite right and is a performance issue in and of itself…) The second problem is that if a page is made up of many layers, or updates large parts of itself as you scroll, uploading that work to the graphics unit can take a significant amount of time. During this time, the page will appear to ‘hang’, as unfortunately, you can’t upload data to the GPU and continue to use it to draw things (this isn’t true in every single case, but again, for our purposes, it is).

Progressive rendering tries to spread this load by breaking up that work into several smaller tiles, and processing them one-by-one, where appropriate. This helps us mitigate those pauses that may happen for particularly complex/animated pages. Alongside this work, we also add the ability for a render to be cancelled. This is good for the situation that a page takes so long to render that by the time it’s finished, what it rendered is no longer useful. Currently, because a render is done all at once, if it takes too long, we can waste precious cycles on irrelevant data. As well as splitting up this work, and allowing it to be cancelled, we also try to do it in the most intelligent order – render areas that the user can see that were previously blank first, and if that area intersects with more than one tile, make sure to do it in the order that maintains visual coherence the best.

A cherry on the top (which is still very much work-in-progress, but I hope to complete it soon), is that splitting this work up into tiles makes it easy to apply nice transitions to make the pathological cases not look so bad. With that said, how’s about some video evidence? Here’s an almost-Nightly (an extra patch or two that haven’t quite hit central), with the screenshot layer disabled so you can see what can happen in a pathological case:

And here’s the same code, with progressive tile rendering turned on and a work-in-progress fading patch applied.

This page is a particularly slow page to render due to the large radial gradient in the background (another issue which will eventually be fixed), so it helps to highlight how this work can help. For a fast-to-render page that we have no problems with, this work doesn’t have such an obvious effect (though scrolling will still be smoother). I hope the results speak for themselves 🙂

]]>

Eurogamer Expo 2012

One of the perks of being a Virgin Media customer (beyond getting my name wrong and constant up-sell harassment) is that I got cheap, early-access Eurogamer Expo tickets! This was my first Eurogamer Expo, though I’m no stranager to ECTS or ATEI/EAG. The setup was quite good – perhaps a bit smaller than I expected, but nice to see a games show that’s actually aimed at gamers. I was always amused at the hoops you had to jump through to get tickets for ECTS and ATEI; more so when you actually visit the events and realise the majority of people there are gamers who have jumped through those same hoops. Good to see that the games industry, finally, after several years, got wise.

There was a fair amount on show. Lots of soon and not-so-soon to be released games, the WiiU, a surprising and pleasing amount of indie content and various bits and bobs. The WiiU was certainly the main attraction, but was managed terribly and was extremely disappointing. While most of the company reps were great and very helpful, a couple of Nintendo’s were oddly aggressive and patronising. I don’t think anyone at Eurogamer needs to be told how to play WiiU mini-games, or have buttons on their controllers pressed for them. The decision to dedicate three entire kiosks in the WiiU section to a video panorama viewer was baffling too. It’s almost as if no one at Nintendo has picked up a smart-phone in the last 5 years or so – this isn’t astounding stuff. Wonderful 101 seemed quite fun, but not as fun as I was expecting. The rest of the WiiU content was very disappointing. Pikmin 3 looking bland and boring was especially upsetting. It’s ironic that playing on the console has secured my decision not to buy it on release. I could easily write about how disappointing the WiiU was for a lot longer, but I just don’t care enough.

What was pleasantly surprising was how good Sony’s presence and content was. Reps were polite and helpful, not getting in the way where they weren’t needed and turning up when they were. Much like a good waiter. They had plenty of kiosks and space, and queues were minimal (not due to lack of interest, mind). Playstation All-stars Battle Royale, though clearly a Smash Bros. rip-off, is actually a very good one. We spent quite a while on it, and it was very enjoyable (possibly more so than Smash Bros. Brawl, but it doesn’t even approach the heights of Melee). The cross-play was especially impressive too, mirroring almost the exact same game frame-for-frame with only minor graphical omissions. Stand-out game of the show had to be When Vikings Attack, though. Incredibly simple concept, but perfect execution and impressive cross-play again. The only disappointment was that it doesn’t have a confirmed release date, but Clever Beans say it will be on PSN before the end of the year. This is definitely day-one purchase material.

Carmageddon definitely deserves a mention. It’s just as much fun as it was all those years ago, and the tablet/smartphone port has been handled perfectly. A shame that there was no demo or footage for the Carmageddon Reincarnation project, but hopefully it made a few more people aware. Also worth mentioning was God of War: Ascension, which although is more of the same, it’s a brilliant same that it’s more of. The multiplayer worked surprisingly well too, though a LAN setup is always going to be more fun than online. There were a few things that I’d have liked to have tried, but queues prevented me – nothing I would deem queue-worthy though. Hitman looked quite impressive, but the whole misogyny thing has put me off. Same goes for Tomb Raider. Dishonoured looked interesting, but not so interesting to queue for. Halo 4 looked like more of the same, though the considerable graphical upgrade certainly doesn’t hurt. Dead or Alive 5 was quite fun, and pleasing to see that they’ve returned to the mechanics of Dead or Alive 2 (clearly the series high). Disappointing amount of guys picking bikini-clad women to fight and leaving the camera aimed at crotch/chest areas; we evened the score a bit by playing as ridiculous-looking guys and aiming at the groin. Yes, I am 12. Disappointed to see that they’ve not included Zack’s weird sports-bra costume. The indie games arcade section is probably worth mentioning in that almost everything in it was terrible and just trading on a quirky look with zero gameplay to back it up. I conclude that there’s still plenty of room for ideas and innovation in the British indie games community.

All in all, a pretty fun event. Slightly disappointing that the industry still hasn’t moved on from the whole booth-babe thing, but it’s definitely far less prevalent than it used to be, so that leaves me with some hope. The graphical standard of console games is astounding, especially given there hasn’t been a hardware refresh in over 5 years. I’ll definitely be returning next year.

]]>

position:fixed in Firefox Mobile

It seems, somehow, for the last few months, I’ve been working on layout. I’m not quite sure how it happened, as anyone who’s spoken to me or follows me on Twitter would know that I have a very healthy fear of the Gecko layout code. I still have that fear, but I’d like to think now that it’s coupled with a tiny amount of understanding; understanding that has, dare I say it, even let me have fun while working on this code!

Those of you that have used browsers on mobile phones (all of you?), especially not the very latest phones, may be familiar with an annoying problem. That is, elements that have position:fixed in their style tend to jump around the place like they’ve had too much coffee. You commonly see this on sites that have a persistent bar at the top or bottom of the page, or floating confirmation notifications, things like this. Brad Frost wrote about this far more eloquently than I could here. This has always annoyed me, especially after learning more about how browsers work. Certainly in Gecko, we have all of the context we need for this not to happen. It also ended up that this problem had been worked on long before I even joined Mozilla last year, so that made it doubly surprising that we suffered from this problem in all releases of Firefox Mobile.

When I first came across this last year, I discovered that the support was already there in the old Firefox Mobile, but disabled by default due to it causing test failures. I was working on other things then, and wasn’t at all acquainted with layout code, so I let it be. Revisiting it for the new, native Firefox Mobile though, these test failures didn’t exist anymore. Enabling this basic support that would let position:fixed elements move and zoom with user input correctly was not too big a deal – just flip an environment variable and write a small amount of support code. This landed in Firefox 15 and is tracked in Bug 607417. Just this is enough for a lot of mobile sites to start using position:fixed (I’m looking at you, Twitter and Facebook!).

This wasn’t enough for me though. Around this time, Android 3.x (Honeycomb) tablets had been around for quite a while and the Galaxy Nexus with Android 4.0 (Ice-cream Sandwich) had just come out, both with even better support for position:fixed. Not to mention the iPhone, which has excellent support. A problem with our implementation in Firefox 15, is that anything fixed to the bottom or right of the screen, or anything that doesn’t anchor to the top-left in any way, may become inaccessible after zooming in. In recent versions of the Android stock browser, not only do these remain accessible, but they zoom very smoothly too. Not wanting to be one-upped by what could be considered our main competition, I started to work on more comprehensive position:fixed support. This work was tracked in Bug 758620.

When zooming in our browser, we don’t change how the page is laid out, but fixed position elements are still rendered relative to the viewport. What we want (at least, for now) is for fixed position elements to lay out with respect to this viewport so that they always remain visible, and to transition smoothly between these states. To achieve this, I changed layout so that fixed position elements are laid out to what we call the scroll-port. When we zoom in, we change the scroll-port, otherwise you wouldn’t be able to scroll to the bottom-right of the page, but this only changes scrolling behaviour and nothing else. This change also made it so that fixed-position children of the document would be relayed out when this scrollport was changed. This fixed the inaccessibility problem, but left nasty-looking transitions when zooming in.

To fix the transitions was quite a bit more comprehensive and lead me down a long path of causing and fixing various layout bugs. When a page is rendered, the DOM tree is parsed into a frame tree, which better represents the layout of the page. This frame tree is then parsed into a display-list, which represents how to draw the page. This display-list is then optimised and parsed/drawn into a layer tree, which is the final representation before we composite it. There’s cleverness to make sure that layers aren’t recreated unnecessarily, but that’s another subject for another time. We wanted to be able to annotate the layer tree so that when compositing, we have enough information to determine how to place fixed-position layers when zooming. This involved creating a new display-list item with the information about how the element is fixed (to the top? left? right? bottom?), and also that would guarantee that the items representing this element would end up on their own layer. Once this was done, code in the compositor was added to leverage this information and draw layers in the right place.

This is an area that a lot of browsers have difficulty with, so it was a fun problem to work on. Try out a couple of my testcases if you’re interested, they expose how different browsers handle this situation, and in the case of a few of them, some bugs 🙂 We’re still not perfect, but we’re better than we were before – and to my feeling, we’re adequate now. This work landed in Firefox 16.

So is there work left to do? Well, unfortunately, yes. I’ve just finished off support for fixed backgrounds and backgrounds with multiple fixed/non-fixed layers, and this will arrive in Firefox 18. This is tracked in Bug 786502. I also think that the best behaviour would be for fixed position elements to layout to whichever is largest of the CSS viewport or the scroll-port, and for scrolling to be within the CSS viewport and push the edges when you reach them. I’m told this is what happens in IE10 on Windows 8, and is similar (but slightly better) to what gets done in Safari on iOS. I think it’s about time for a break from this particular feature for me, though.

]]>

How can Mozilla and Gnome work together?

I’ve been pretty lax on blogging lately, but here’s something that’s troubling me. I haven’t really done any work directly related to Gnome since I started working at Mozilla. Ends up writing browsers is pretty hard, and any recreational programming time I get, I don’t particularly feel inclined to work on Gnome. I have, however, been attending Guadec this week. I haven’t missed one since 2006 and I don’t intend to. What’s troubling me, is that although Mozilla were kind enough to sponsor my presence here (we’re hiring!), Gnome doesn’t seem to be hugely relevant to us anymore. I’d love to be corrected of course, but judging by the amount of effort we’re putting into the Gtk+3 port, themeing and other Linux-related bugs, I’m pretty sure this is the case.

I have some ideas about this, but I’d like to be brief. For now. So, my simple question is, How can Mozilla and Gnome work better together?

[Edit]: Seems my blog’s commenting form is broken. Until it’s fixed, feel free to mail me your comments, I’d love to hear them! (address on the side of the page)

[Edit2]: Comments appear to be working again, but if they fail, do mail me!

]]>

Mobile Platform at the Toronto Firefox Work Week

For those that aren’t already aware, many Mozillians gathered last week in Toronto to Maximise Synergy. Seeing as there have been updates for the UX team and the Firefox/Mobile UI team, I think it’d be worth having a similar update for mobile platform. Some fantastic work is going on in this area, and people deserve kudos 🙂

  • Kartikaya Gupta (aka kats) continues to do sterling work, fixing an inhuman amount of bugs. Most prominently, kats has been working on our display-port ‘strategies’, code that allows us to make the most of our rendering time and, thus, reduce checkerboarding. He’s also been working on our checkerboard measurement, augmenting and refining our tests and test-method to make sure we’re always pushing ourselves. This, alongside much general bug-fixing work.
  • Benoit Girard (aka BenWa) is part of the gfx team, all of whom have been helping us out for the past few months, getting our performance back to a competetive level. Benoit’s tiled rendering patches finally landed, which make a huge difference to performance. Benoit has also been working on some useful remote profiling tools, which he blogs about here.
  • James Willcox (aka snorp) continues to battle Flash to get us the best Flash experience in an Android browser (ironic, right?) snorp’s work to get more accurate positioning of Flash elements landed, along with his work to replace elements with a snapshot during page movement on Froyo/Gingerbread, completing the experience.
  • Brad Lassey (aka blassey) managed to get enough time to do some coding alongside his usual cat-herding responsibilities, and got us a low-res page cache. This means that in the rare situation that our rendering can’t keep up, instead of showing checkerboard, we show a low-resolution rendering of the page. It’s possible that this content can go stale, but we find it provides a much better experience than showing either a blank colour or checkerboard. This works much better than any of us expected it to, and for me at least, was the turning point after which I find our browser much easier to recommend.
  • And me? I managed to get my retained tiles patch landed. Currently, we render the page into system-memory tiles, which we then upload to the GPU. My patch keeps a hold of the GPU tiles and metrics for a while after they’re invalidated. Again, when our renderer can’t keep up, it renders these old tiles into the space instead of rendering nothing. This mostly helps quick changes of direction and zooming out after zooming in. I also spent some time fixing up the low-res cache work, then fixing up my fix-ups, as is my custom 🙂 Hopefully, preliminary fixed-position layer support will also land soon.

Much other work also occured, and sorry for missing anyone out that I surely did. It’s worth mentioning that the gfx team have been a huge help and have been working extra-hard for months now, helping us get to where we are today (and hopefully beyond!) I had a great (and hard) time during my stay and am very much looking forward to our upcoming release 🙂

]]>