Building all the major open-source web browsers

Mozilla Firefox, Chromium (the open-source variant of Chrome) and WebKit (the basis for Safari) are all great examples of open-source software. The Qt project has a simple webkit-based web browser in their examples. So that’s at least four different open-source web browsers to choose from.

But what does it take to actually build them? The TL;DR answer is that these are complex pieces of software, each of them with rather idiosyncratic build systems, and that you should consider 100GB of disk space to build all the browsers, a few hours of download, and be prepared to learn lots of new, rather specific tools.

So here is my cheat sheet, in case you want to do the same thing. All the experience related below is on OSX 10.9 “Mavericks” with the latest Xcode installed as of today (just released for OSX 10.10 Yosemite).

WebKit

WebKit is the most standard of the three web browsers. It uses git for version control and make for builds.

WebKit can be fetched using a standard git command. Simple et de bon goût.

git clone git://git.webkit.org/WebKit.git

Building looks quite simple, with one major caveat (see below). To build:

make

This can’t be much simpler. There isn’t even an autoconf step in sight. Now, the caveat is this: it currently does not build on my Mac. I get the following:

ld: warning: directory not found for option '-L/Volumes/Whopper/Work/WebKit/WebKitBuild/Release/usr/local/LLVMForJavaScriptCore/lib'
ld: warning: directory not found for option '-L/usr/local/LLVMForJavaScriptCore/lib'
ld: library not found for -lLLVMX86Disassembler
clang: error: linker command failed with exit code 1 (use -v to see invocation)
** BUILD FAILED **
The following build commands failed:
Ld /Volumes/Whopper/Work/WebKit/WebKitBuild/Release/libllvmForJSC.dylib normal x86_64

The build failure changes if I do a clean build and then a non-parallel build (make without -j option):

** BUILD FAILED **
The following build commands failed:
CompileC /Volumes/Whopper/Work/WebKit/WebKitBuild/DumpRenderTree.build/Release/DumpRenderTree\ (Library).build/Objects-normal/x86_64/ObjCPlugin.o mac/ObjCPlugin.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
make[2]: *** [all] Error 65
make[1]: *** [all] Error 2
make: *** [all] Error 2

Something frustrates me here: I don’t see the actual compile error, unless I’m missing something???

Hmmm, investigating a little more, it looks like it’s necessary to use the build-webkit script. Maybe that will solve my problem? At the very least, it has the benefit of setting a rather reasonable level of parallelism for the build (now my CPUs are all churning).

OK, that did not help, but at least now I have the error message on the terminal (huge chunks cut for clarity):

CompileC MiniBrowserWebProcessPlugIn.m [...] WebKit/Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m -o [...]/Objects-normal/x86_64/MiniBrowserWebProcessPlugIn.o
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m:26:
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.h:26:
In file included from WebKitBuild/Release/WebKit.framework/PrivateHeaders/WKWebProcessPlugIn.h:26:
WebKitBuild/Release/WebKit.framework/Headers/WKFoundation.h:55:20: error: 
      typedef redefinition with different types ('NSUInteger'
      (aka 'unsigned long') vs 'enum NSEventModifierFlags')
typedef NSUInteger NSEventModifierFlags;
                   ^
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m:1:
In file included from Tools/MiniBrowser/mac/Bundle/MiniBrowserBundle_Prefix.pch:27:
[...]
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSResponder.h:9:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSEvent.h:108:32: note: 
      previous definition is here
typedef NS_OPTIONS(NSUInteger, NSEventModifierFlags) {
                               ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObjCRuntime.h:269:52: note: 
      expanded from macro 'NS_OPTIONS'
#define NS_OPTIONS(_type, _name) CF_OPTIONS(_type, _name)
                                                   ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFAvailability.h:175:53: note: 
      expanded from macro 'CF_OPTIONS'
#define CF_OPTIONS(_type, _name) enum _name : _type _name; enum _name : _type
                                                    ^
1 error generated.
** BUILD FAILED **

So apparently, the latest Xcode changed the definition of NS_OPTIONS to use an enum instead of a plain integer type, and redefinitions that don’t use the macro fail. Not too complicated to fix on my side, but I’m surprised it’s not been flagged by build bots.

Removed the offending line, now it works.

Firefox

Firefox uses Mercurial for its source control, and a set of build scripts that seems to be built on good old autoconf and make. However, it uses autoconf 2.13 because its build system uses features that were removed in autoconf 2.5. Fortunately, I use MacPorts, so installing Mercurial and autoconf 2.13 is quite easy. Same thing would be true on Linux, but I guess Windows users would once more have a few extra steps to take.

sudo port install mercurial
sudo port install autoconf213

You get the source code using Mercurial from Mozilla Central:

hg clone https://hg.mozilla.org/mozilla-central/ firefox

I had not used Mercurial in a long time. I find it less informative than Git for large downloads like this. You get something like this:

requesting all changes
adding changesets
adding manifests
adding file changes
added 212327 changesets with 1196756 changes to 183983 files
updating to branch default

There are minutes between each of these lines. The last one, in particular, seems to take forever, if it does something similar to a git checkout…By contrast, Git outputs a percentage indicating progress, which gives you a much better sense that something is actually happening, something like:

Receiving objects:  67% (1766736/2628930), 4.21 GiB | 387.00 KiB/s

To build Firefox, you use a specific script:

./mach build

Believe it or not, Firefox does not build from a directory with spaces in the path:

ddd@Marypuce firefox> ./mach build
 0:37.47 /opt/local/bin/gmake -f client.mk -s
 0:37.63 client.mk:43: *** The mozilla directory cannot be located in a path with spaces..  Stop.
 0:37.75 0 compiler warnings present.

So I had to rename one of my disks just to build Firefox. That and the fact that it refuses to build without autoconf 2.13 gave me a strong whiff of XXth-century… But once this is fixed, the build output is rather clean, showing nice progress and timing information along the way. It scrolls, but other than that, it’s rather good, even displaying a colorised line at the bottom of your terminal showing “TIER: export compile libs tools”… :

 0:18.10 /opt/local/bin/gmake -f client.mk -s
 0:19.31 client.mk:201: /Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0/.mozconfig.mk: No such file or directory
 0:28.01 Clobber not needed.
 0:28.94 Adding client.mk options from :
 0:28.94     MOZ_OBJDIR=/Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0
 0:28.94     OBJDIR=/Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0
 0:29.44 Generating /Volumes/Whopper/Work/firefox/configure using autoconf
 0:36.13 Generating /Volumes/Whopper/Work/firefox/js/src/configure using autoconf

Apparently, you need to explicitly pass the -j12 option to “./mach build” to get parallel builds. But even with that, I’m nowhere near 100% CPU on my machine, so not sure exactly how this option is being used.

That being said, the build fails after 15mn in a rather mysterious way:

14:05.65 warning: interface 'nsISelectionPrivate' is scriptable but derives from non-scriptable 'nsISelection', ../../../dist/idl/nsISelectionPrivate.idl line 31:0
14:05.65 interface nsISelectionPrivate : nsISelection
14:05.65 ^
14:05.68 dom_events.xpt
14:05.93 gmake[6]: *** write jobserver: Broken pipe.  Stop.
14:05.93 gmake[6]: *** Waiting for unfinished jobs....
14:05.94 gmake[6]: *** write jobserver: Broken pipe.  Stop.
14:05.94 Makefile:14: recipe for target 'export' failed
14:05.94 gmake[5]: *** [export] Error 2
14:05.95 /Volumes/Whopper/Work/firefox/config/recurse.mk:82: recipe for target 'xpcom/xpidl/export' failed
14:05.95 gmake[4]: *** [xpcom/xpidl/export] Error 2
14:05.95 gmake[4]: *** write jobserver: Broken pipe.  Stop.
14:05.95 gmake[4]: *** Waiting for unfinished jobs....
14:17.78 gmake[4]: *** write jobserver: Broken pipe.  Stop.
14:17.78 /Volumes/Whopper/Work/firefox/config/recurse.mk:36: recipe for target 'export' failed
14:17.78 gmake[3]: *** [export] Error 2
14:17.78 /Volumes/Whopper/Work/firefox/config/rules.mk:551: recipe for target 'default' failed
14:17.78 gmake[2]: *** [default] Error 2
14:17.78 gmake[2]: INTERNAL: Exiting with 1 jobserver tokens available; should be 8!
14:17.79 /Volumes/Whopper/Work/firefox/client.mk:398: recipe for target 'realbuild' failed
14:17.79 gmake[1]: *** [realbuild] Error 2
14:17.80 client.mk:171: recipe for target 'build' failed
14:17.80 gmake: *** [build] Error 2
14:17.85 16 compiler warnings present.
14:17.85 ccache (direct) hit rate: 83.7%; (preprocessed) hit rate: 0.0%; miss rate: 16.3%
14:17.86 Notification center failed: Install terminal-notifier to get a notification when the build finishes.

OK. Where do I go for there? Google, help! Did not find anything. Restarted the build, which seemed to complete after 36m50s and to tell me I had built Firefox. Yeah, it seems to have built successfully. Next step is to launch it:

./mach run

It takes forever to launch, maybe because it’s a debug build. The built is called “Nightly”. It spouts tons of relatively uninformative messages like:

2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x1003111a0
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff5fbf2380
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x90
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff743f2420
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x3
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff5fbf22f0

Interestingly, the start page for Nightly invites me to visit pages about testing, coding or localising the product. That’s rather useful.

Chromium

Chromium is the most peculiar of the three to build. It has its own non-standard set of development tools both for version control and for building. So the first step is to install the depot-tools:

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

You need to add the depot_tools directory to your path.

To get the source code, you use a three step process, which seems rather complicated to me. The first step fetches the source code, and you can choose between for example chromium, blink, etc, depending on what part of the code you work on. Not sure yet if that means it fetches different branches, different repositories, or something else:

fetch chromium

Remember what I wrote about progress information in Mercurial vs. Git? Well, here, it’s worse. As far as I can tell, ‘fetch’ is built on ‘git’, but it manages to remove the useful, one-liner, no-scrolling information from Git with a multi-line, useless, scrolling “hey, I’m still working” reminder!

[3:41:33] Still working on:
[3:41:33]   src/third_party/WebKit
[3:41:43] Still working on:
[3:41:43]   src/third_party/WebKit
[3:41:53] Still working on:
[3:41:53]   src/third_party/WebKit

And after literally hours of that, you apparently need to checkout master and do a git pull again. At least, until I did that, I was in “detached branch” mode on git, something that’s not very comfortable, because without a branch name to refer to, you are likely to loose changes if you do something wrong. So this is what I did:

cd src
git checkout master
git pull

The last step is something called “gclient sync”, and again, I have at the moment no idea what it does precisely. And this page is not helping me… This is a key problem with all these custom build tools. All I know is that whatever “gclient sync” does, it takes a while to do it (about 9 minutes for me)… I don’t know what the value add is. Is this useful stuff, or just useless crud? If someone cares to comment and explain to me, I’d appreciate that.

gclient sync

For building, Chromium does things its own way, just like source code management. It may have good reasons to do so, but you have to learn the way. So what seems to work for me is:

ninja -C out/Debug

Apparently, out/Debug and out/Release are two directories where a build.ninja file exists. Need to dig deeper into how ninja works, but something that describes itself as the assembly language of makefiles is not exactly to reassure me… Even less so when an incremental “make” of the WebKit starts right away, when an incremental ninja build of Chromium seems to take forever before starting the first build step. Your mileage may vary.

One interesting aspect of the ninja build tool is that it shows progress in a rather intelligent way, without scrolling your terminal contents out of the way. I have yet to see how well it does with build failures. One thing I found curious is that in two subsequent attempts at building Chromium from a “clean” tree, the number of build steps varied wildly (between around 17000 and around 25000). Not sure why.

Qt5

`The final candidate is Qt5, the latest iteration of the Qt project.

Getting the code is simple enough for someone familiar with Git:

git clone git://gitorious.org/qt/qt5.git

However, that’s only the top-level directory. If you want the whole build tree, you also need many submodules. This is achieved with:

cd qt5
./init-repository

This steps takes much longer, as it fetches dozens of different repositories from gitorious. But it’s basically adding the required submodules to the top repository, so it’s operating in a very standard way.

Building Qt5 is also simple enough, since it uses rather standard build procedures (although its “configure” is not based on autoconf, for the better if you ask me):

configure
make

You need to answer a few simple questions during the “configure” step (e.g. if you want an open source or commercial license). But after that, the build progresses swiftly. There is currently a minor issue when building using Yosemite tools on OSX, but that’s easily fixed.

Build time and disk space considerations

Building all these browsers uses considerable resources.

Product	Source fetch time	Source size	Build time	Build size
Qt5	27mn	1.46GB	1h46mn	15.5GB
Firefox	3h41mn	2.1GB	50mn	4.58GB
Chromium	278mn (fetch) + 6m30 (git checkout/pull) + 8m40 (gclient sync)	13.42GB	120mn	56.15GB
WebKit	5h26mn	7.12GB	43mn	19.47GB

I fetched a top-of-tree copy of all the trees using the same Internet connexion. Since they were running in parallel, they were competing for the same resources, which I think makes the test fair. I then measured the size of the code repository right after fetching the code.

I don’t know why it takes so much longer for Firefox to download just a little bit more code than Qt5… Is it that Git is more efficient than Mercurial? Is it that the servers or network were giving me different throughput? Or it may be a matter of how complex the history is. For example, I noticed that in the WebKit, the “resolving deltas” step is quite slow.

Each project was built by itself, running on a 8-CPU machine with make -j12, or whatever the default was for the build with “special” build tools. I think most project built “Debug”, but I need to double check that. The size I report after build is the size as given by the Mac Finder.

Comments

It takes less time to download and build Qt than to download any of the other projects!

At some point, the story behind Chrome switching to Blink was “simplification” and “removing 7000 files and 4.5M lines of code”. Given the current state of the code, that seems hard to believe today. 56GB for a build, that’s a LOT of code.