Rethinking rsync at Mozilla

Some time ago Mozilla moved away from volunteer servers for delivering installers and updates to our end users, but we still offer two rsync modules

  • mozilla-current – just the latest Firefox and Thunderbird releases
  • mozilla-releases – a wider set of recent releases

Both of these have been unmaintained for some time, so they are out of date (1 year) and huge (500GB) respectively. We still serve quite a lot of traffic through those modules, although there were no complaints when it was down for a few days recently.

I’m interested to hear opinions on whether we should maintain rsync access to release bits. From the logs it’s clear some of former mirrors are still pulling data, which is fine if it’s intentional rather than legacy. There may be other use cases we’re not aware of, so please let us know in the comments or on bug 807543.

Issues with Android Nightly

This is a PSA-style post on a couple of recent issues with Nightly builds of Firefox on Android:

  • Pages aren’t rendered when you haven’t used Firefox for a while, the about:home content ‘sticks’ instead. This is bug 863803, present in builds from April 18th. The workaround is to close and restart Firefox. On ICS or higher you can swipe it away in the recent apps list, otherwise force close the app
  • Nightly doesn’t update any more and is stuck on the build from April 10th (fixed for later builds by bug 860454). If you see ‘(2013-04-10)’ after loading about:firefox, then install a new copy of Nightly from

Nightly updates for Firefox on Tablets

Development continues apace on the native rewrite of Firefox for Android, with the emphasis being on smartphones for the moment. In the meantime the XUL-based builds are recommended for tablets, and from a few days ago they started updating to the next XUL nightly (instead of swapping back to the native one).

The current links for the development channels are Nightly and Aurora. Give them a spin and help us test these builds on your tablet!

New home for build data

For quite a while now Release Engineering has been publishing information about Firefox compile and test jobs in JSON form, for example

The system that has been generating that data has been outgrown by the tasks given to it, so Chris AtLee has been working on moving it from to

If you use this data please update your code to use the new location, as the old one won’t be updated any more.

Broken updater for Mac Nightly builds

There was some accidental breakage in the Mac builds of Nightly last Friday (Sept 16th), which results in a crash when you try to update your build. The revisions & buildIDs for the broken builds are

  • 39b192706927, 20110916030907
  • f4d78560721a, 20110916071142

You can check what you have by loading about:buildconfig, or evaluating navigator.buildID in the Web Console.

The solution is to download the latest Nightly build from

Applications like Thunderbird and SeaMonkey are probably also affected.

Retention for on-change builds

When someone pushes into a mercurial repository like mozilla-central the resulting Firefox builds end up on the ftp server, in a <branch>-<platform> directory (eg win32 optimised, win32 debug). Since early February we’ve been keeping those builds for 30 days, up from 1 day, which makes for much more fine-grained regression searches than using nightlies.

Firefox Mac Universal builds have changed

We now have a universal build that contains 32 and 64-bit binaries for Intel CPUs,  instead of PowerPC & Intel 32-bit. This is for all branches except Firefox 3.5 and 3.6, and is based on all the work Josh Aas et. al. have done getting the 64-bit build working nicely, plus the decision to drop PowerPC support for Firefox 4.0.

The new universal build defaults to running 32-bit on Leopard (10.5), and 64-bit on Snow Leopard (10.6). Anyone who has been getting nightly updates for the old universal and also has an Intel processor will be moved over to the new one.

For those of you paying attention to TinderBoxPushLog there is no longer a B for OS X opt, and the B for OS X64 opt is where the new universal build is happening. We do talos and optimized unittest on the new universal build, running it 32-bit on 10.5 and 64bit on 10.6. There are still two debug builds – OS X debug is 32-bit and the unit tests run on 10.5, while OS X64 debug is a 64-bit build tested on 10.6. The debug mochitest-other for the 32-bit build has been disabled due to a packaging problem, but we still have the 64bit equivalent which is green.

Data for builds in progress

Chris AtLee has blogged about RelEng’s move to Buildbot v0.8.0 to control the many compilation and test slaves that serve the Firefox codebase, and about the database backend which will allow us to better track build progress and results. We’re working on making more of that information available for people to use, for example in TBPL.

As a first step, we’re now publishing data on jobs which are running and waiting for service in JSON format. There are also HTML versions if you’d like to view the information in a table form: running, pending. The JSON is broken down by branch and revision with the following data:

  • buildername – as appears on the top of the tinderbox waterfall column and TBPL display for a build
  • submitted_at – timestamp for when the job was added to the database
  • start_time – timestamp for when the job started running
  • last_heartbeat – timestamp for the buildbot master confirming the job is still running

All the timestamps are seconds since the Unix epoch, and the data is refreshed every 5 minutes.

Note that the data currently covers a subset of all the jobs that run, in particular the unit and talos jobs running on ‘Rev3 …’ builders are not included. We’ll close that gap when the farm of minis migrates to buildbot 0.8.0. In the meantime we have all the compile jobs for mozilla-central and other branches, some unit tests, about half of mobile, and all of try. The graphs for pending builds does track all jobs, if you were wondering.

Let us know what you think.

Nightly updates for Tracemonkey & Electrolysis

For the TraceMonkey and Electrolysis projects we’ve had a problem with nightly updates sending people back to mozilla-central builds. Nightlies produced from April 1st onwards (yes, really!) no longer had that issue but weren’t getting the latest build. Now I’m happy to tell you that updates will be offered to the latest nightly build for those branches, so you can get the newest JavaScript or process separation fixes automatically. It works just like other nightly and release updates, and we can do the same for any project branch that produces nightly builds.

And if you are one of the lost souls who have a Firefox 4.0a1pre build from March – May 2008, when mozilla-central used that version before dropping down to 3.1a1pre, then welcome to 2010 and all the great changes in mozilla-central since 2008.