Building all the major open-source web browsers

Mozilla Firefox, Chromium (the open-source variant of Chrome) and WebKit (the basis for Safari) are all great examples of open-source software. The Qt project has a simple webkit-based web browser in their examples. So that’s at least four different open-source web browsers to choose from.

But what does it take to actually build them? The TL;DR answer is that these are complex pieces of software, each of them with rather idiosyncratic build systems, and that you should consider 100GB of disk space to build all the browsers, a few hours of download, and be prepared to learn lots of new, rather specific tools.

So here is my cheat sheet, in case you want to do the same thing. All the experience related below is on OSX 10.9 “Mavericks” with the latest Xcode installed as of today (just released for OSX 10.10 Yosemite).

WebKit

WebKit is the most standard of the three web browsers. It uses git for version control and make for builds.

WebKit can be fetched using a standard git command. Simple et de bon goût.

git clone git://git.webkit.org/WebKit.git

Building looks quite simple, with one major caveat (see below). To build:

make

This can’t be much simpler. There isn’t even an autoconf step in sight. Now, the caveat is this: it currently does not build on my Mac. I get the following:

ld: warning: directory not found for option '-L/Volumes/Whopper/Work/WebKit/WebKitBuild/Release/usr/local/LLVMForJavaScriptCore/lib'
ld: warning: directory not found for option '-L/usr/local/LLVMForJavaScriptCore/lib'
ld: library not found for -lLLVMX86Disassembler
clang: error: linker command failed with exit code 1 (use -v to see invocation)
** BUILD FAILED **
The following build commands failed:
Ld /Volumes/Whopper/Work/WebKit/WebKitBuild/Release/libllvmForJSC.dylib normal x86_64

The build failure changes if I do a clean build and then a non-parallel build (make without -j option):

** BUILD FAILED **
The following build commands failed:
CompileC /Volumes/Whopper/Work/WebKit/WebKitBuild/DumpRenderTree.build/Release/DumpRenderTree\ (Library).build/Objects-normal/x86_64/ObjCPlugin.o mac/ObjCPlugin.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler
(1 failure)
make[2]: *** [all] Error 65
make[1]: *** [all] Error 2
make: *** [all] Error 2

Something frustrates me here: I don’t see the actual compile error, unless I’m missing something???

Hmmm, investigating a little more, it looks like it’s necessary to use the build-webkit script. Maybe that will solve my problem? At the very least, it has the benefit of setting a rather reasonable level of parallelism for the build (now my CPUs are all churning).

OK, that did not help, but at least now I have the error message on the terminal (huge chunks cut for clarity):

CompileC MiniBrowserWebProcessPlugIn.m [...] WebKit/Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m -o [...]/Objects-normal/x86_64/MiniBrowserWebProcessPlugIn.o
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m:26:
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.h:26:
In file included from WebKitBuild/Release/WebKit.framework/PrivateHeaders/WKWebProcessPlugIn.h:26:
WebKitBuild/Release/WebKit.framework/Headers/WKFoundation.h:55:20: error: 
      typedef redefinition with different types ('NSUInteger'
      (aka 'unsigned long') vs 'enum NSEventModifierFlags')
typedef NSUInteger NSEventModifierFlags;
                   ^
In file included from Tools/MiniBrowser/MiniBrowserWebProcessPlugIn.m:1:
In file included from Tools/MiniBrowser/mac/Bundle/MiniBrowserBundle_Prefix.pch:27:
[...]
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSResponder.h:9:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/AppKit.framework/Headers/NSEvent.h:108:32: note: 
      previous definition is here
typedef NS_OPTIONS(NSUInteger, NSEventModifierFlags) {
                               ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObjCRuntime.h:269:52: note: 
      expanded from macro 'NS_OPTIONS'
#define NS_OPTIONS(_type, _name) CF_OPTIONS(_type, _name)
                                                   ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFAvailability.h:175:53: note: 
      expanded from macro 'CF_OPTIONS'
#define CF_OPTIONS(_type, _name) enum _name : _type _name; enum _name : _type
                                                    ^
1 error generated.
** BUILD FAILED **

So apparently, the latest Xcode changed the definition of NS_OPTIONS to use an enum instead of a plain integer type, and redefinitions that don’t use the macro fail. Not too complicated to fix on my side, but I’m surprised it’s not been flagged by build bots.

Removed the offending line, now it works.

Firefox

Firefox uses Mercurial for its source control, and a set of build scripts that seems to be built on good old autoconf and make. However, it uses autoconf 2.13 because its build system uses features that were removed in autoconf 2.5. Fortunately, I use MacPorts, so installing Mercurial and autoconf 2.13 is quite easy. Same thing would be true on Linux, but I guess Windows users would once more have a few extra steps to take.

sudo port install mercurial
sudo port install autoconf213

You get the source code using Mercurial from Mozilla Central:

hg clone https://hg.mozilla.org/mozilla-central/ firefox

I had not used Mercurial in a long time. I find it less informative than Git for large downloads like this. You get something like this:

requesting all changes
adding changesets
adding manifests
adding file changes
added 212327 changesets with 1196756 changes to 183983 files
updating to branch default

There are minutes between each of these lines. The last one, in particular, seems to take forever, if it does something similar to a git checkout…By contrast, Git outputs a percentage indicating progress, which gives you a much better sense that something is actually happening, something like:

Receiving objects:  67% (1766736/2628930), 4.21 GiB | 387.00 KiB/s

To build Firefox, you use a specific script:

./mach build

Believe it or not, Firefox does not build from a directory with spaces in the path:

ddd@Marypuce firefox> ./mach build
 0:37.47 /opt/local/bin/gmake -f client.mk -s
 0:37.63 client.mk:43: *** The mozilla directory cannot be located in a path with spaces..  Stop.
 0:37.75 0 compiler warnings present.

So I had to rename one of my disks just to build Firefox. That and the fact that it refuses to build without autoconf 2.13 gave me a strong whiff of XXth-century… But once this is fixed, the build output is rather clean, showing nice progress and timing information along the way. It scrolls, but other than that, it’s rather good, even displaying a colorised line at the bottom of your terminal showing “TIER: export compile libs tools”… :

 0:18.10 /opt/local/bin/gmake -f client.mk -s
 0:19.31 client.mk:201: /Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0/.mozconfig.mk: No such file or directory
 0:28.01 Clobber not needed.
 0:28.94 Adding client.mk options from :
 0:28.94     MOZ_OBJDIR=/Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0
 0:28.94     OBJDIR=/Volumes/Whopper/Work/firefox/obj-x86_64-apple-darwin13.4.0
 0:29.44 Generating /Volumes/Whopper/Work/firefox/configure using autoconf
 0:36.13 Generating /Volumes/Whopper/Work/firefox/js/src/configure using autoconf

Apparently, you need to explicitly pass the -j12 option to “./mach build” to get parallel builds. But even with that, I’m nowhere near 100% CPU on my machine, so not sure exactly how this option is being used.

That being said, the build fails after 15mn in a rather mysterious way:

14:05.65 warning: interface 'nsISelectionPrivate' is scriptable but derives from non-scriptable 'nsISelection', ../../../dist/idl/nsISelectionPrivate.idl line 31:0
14:05.65 interface nsISelectionPrivate : nsISelection
14:05.65 ^
14:05.68 dom_events.xpt
14:05.93 gmake[6]: *** write jobserver: Broken pipe.  Stop.
14:05.93 gmake[6]: *** Waiting for unfinished jobs....
14:05.94 gmake[6]: *** write jobserver: Broken pipe.  Stop.
14:05.94 Makefile:14: recipe for target 'export' failed
14:05.94 gmake[5]: *** [export] Error 2
14:05.95 /Volumes/Whopper/Work/firefox/config/recurse.mk:82: recipe for target 'xpcom/xpidl/export' failed
14:05.95 gmake[4]: *** [xpcom/xpidl/export] Error 2
14:05.95 gmake[4]: *** write jobserver: Broken pipe.  Stop.
14:05.95 gmake[4]: *** Waiting for unfinished jobs....
14:17.78 gmake[4]: *** write jobserver: Broken pipe.  Stop.
14:17.78 /Volumes/Whopper/Work/firefox/config/recurse.mk:36: recipe for target 'export' failed
14:17.78 gmake[3]: *** [export] Error 2
14:17.78 /Volumes/Whopper/Work/firefox/config/rules.mk:551: recipe for target 'default' failed
14:17.78 gmake[2]: *** [default] Error 2
14:17.78 gmake[2]: INTERNAL: Exiting with 1 jobserver tokens available; should be 8!
14:17.79 /Volumes/Whopper/Work/firefox/client.mk:398: recipe for target 'realbuild' failed
14:17.79 gmake[1]: *** [realbuild] Error 2
14:17.80 client.mk:171: recipe for target 'build' failed
14:17.80 gmake: *** [build] Error 2
14:17.85 16 compiler warnings present.
14:17.85 ccache (direct) hit rate: 83.7%; (preprocessed) hit rate: 0.0%; miss rate: 16.3%
14:17.86 Notification center failed: Install terminal-notifier to get a notification when the build finishes.

OK. Where do I go for there? Google, help! Did not find anything. Restarted the build, which seemed to complete after 36m50s and to tell me I had built Firefox. Yeah, it seems to have built successfully. Next step is to launch it:

./mach run

It takes forever to launch, maybe because it’s a debug build. The built is called “Nightly”. It spouts tons of relatively uninformative messages like:

2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x1003111a0
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff5fbf2380
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x90
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff743f2420
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x3
2014-10-25 18:18:03: basic_code_modules.cc:88: INFO: No module at 0x7fff5fbf22f0

Interestingly, the start page for Nightly invites me to visit pages about testing, coding or localising the product. That’s rather useful.

Chromium

Chromium is the most peculiar of the three to build. It has its own non-standard set of development tools both for version control and for building. So the first step is to install the depot-tools:

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

You need to add the depot_tools directory to your path.

To get the source code, you use a three step process, which seems rather complicated to me. The first step fetches the source code, and you can choose between for example chromium, blink, etc, depending on what part of the code you work on. Not sure yet if that means it fetches different branches, different repositories, or something else:

fetch chromium

Remember what I wrote about progress information in Mercurial vs. Git? Well, here, it’s worse. As far as I can tell, ‘fetch’ is built on ‘git’, but it manages to remove the useful, one-liner, no-scrolling information from Git with a multi-line, useless, scrolling “hey, I’m still working” reminder!

[3:41:33] Still working on:
[3:41:33]   src/third_party/WebKit
[3:41:43] Still working on:
[3:41:43]   src/third_party/WebKit
[3:41:53] Still working on:
[3:41:53]   src/third_party/WebKit

And after literally hours of that, you apparently need to checkout master and do a git pull again. At least, until I did that, I was in “detached branch” mode on git, something that’s not very comfortable, because without a branch name to refer to, you are likely to loose changes if you do something wrong. So this is what I did:

cd src
git checkout master
git pull

The last step is something called “gclient sync”, and again, I have at the moment no idea what it does precisely. And this page is not helping me… This is a key problem with all these custom build tools. All I know is that whatever “gclient sync” does, it takes a while to do it (about 9 minutes for me)… I don’t know what the value add is. Is this useful stuff, or just useless crud? If someone cares to comment and explain to me, I’d appreciate that.

gclient sync

For building, Chromium does things its own way, just like source code management. It may have good reasons to do so, but you have to learn the way. So what seems to work for me is:

ninja -C out/Debug

Apparently, out/Debug and out/Release are two directories where a build.ninja file exists. Need to dig deeper into how ninja works, but something that describes itself as the assembly language of makefiles is not exactly to reassure me… Even less so when an incremental “make” of the WebKit starts right away, when an incremental ninja build of Chromium seems to take forever before starting the first build step. Your mileage may vary.

One interesting aspect of the ninja build tool is that it shows progress in a rather intelligent way, without scrolling your terminal contents out of the way. I have yet to see how well it does with build failures. One thing I found curious is that in two subsequent attempts at building Chromium from a “clean” tree, the number of build steps varied wildly (between around 17000 and around 25000). Not sure why.

Qt5

`The final candidate is Qt5, the latest iteration of the Qt project.

Getting the code is simple enough for someone familiar with Git:

git clone git://gitorious.org/qt/qt5.git

However, that’s only the top-level directory. If you want the whole build tree, you also need many submodules. This is achieved with:

cd qt5
./init-repository

This steps takes much longer, as it fetches dozens of different repositories from gitorious. But it’s basically adding the required submodules to the top repository, so it’s operating in a very standard way.

Building Qt5 is also simple enough, since it uses rather standard build procedures (although its “configure” is not based on autoconf, for the better if you ask me):

configure
make

You need to answer a few simple questions during the “configure” step (e.g. if you want an open source or commercial license). But after that, the build progresses swiftly. There is currently a minor issue when building using Yosemite tools on OSX, but that’s easily fixed.

Build time and disk space considerations

Building all these browsers uses considerable resources.

Product Source fetch time Source size Build time Build size
Qt5 27mn 1.46GB 1h46mn 15.5GB
Firefox 3h41mn 2.1GB 50mn 4.58GB
Chromium 278mn (fetch)

+ 6m30 (git checkout/pull)

+ 8m40 (gclient sync)

13.42GB 120mn 56.15GB
WebKit 5h26mn 7.12GB 43mn 19.47GB

I fetched a top-of-tree copy of all the trees using the same Internet connexion. Since they were running in parallel, they were competing for the same resources, which I think makes the test fair. I then measured the size of the code repository right after fetching the code.

I don’t know why it takes so much longer for Firefox to download just a little bit more code than Qt5… Is it that Git is more efficient than Mercurial? Is it that the servers or network were giving me different throughput? Or it may be a matter of how complex the history is. For example, I noticed that in the WebKit, the “resolving deltas” step is quite slow.

Each project was built by itself, running on a 8-CPU machine with make -j12, or whatever the default was for the build with “special” build tools. I think most project built “Debug”, but I need to double check that. The size I report after build is the size as given by the Mac Finder.

Comments

It takes less time to download and build Qt than to download any of the other projects!

At some point, the story behind Chrome switching to Blink was “simplification” and “removing 7000 files and 4.5M lines of code”. Given the current state of the code, that seems hard to believe today. 56GB for a build, that’s a LOT of code.

15 thoughts on “Building all the major open-source web browsers

    1. Fixed. This post is a work in progress, as I decided to start over midway because I had done some builds on a Flash drive and then, running out of space, other builds on a separate platter-based disk. So I’m updating things as I go, running into different things. For example, Firefox built successfully yesterday, but today, the first build failed. The second one succeeded. Not sure why.

  1. All I think “gclient sync” does is wrap a bunch of git calls to keep the third_party directory up to date. And run some python hook scripts afterwards. Basically “git pull” for chromium gets the chromium code; “gclient sync” keeps the 80+ third party dependencies up to date (as far as chromium is concerned), as occasionally updates directories in depot_tools.

  2. All I think “gclient sync” does is wrap a bunch of git calls to keep the third_party directory up to date, and run some python hook scripts afterwards.

    Basically “git pull” for chromium gets the chromium code; “gclient sync” keeps the 80+ third party dependencies in the third_party directory up to date (as far as chromium is concerned), as occasionally runs hook scripts to keep other parts of the tree in sync (updates to depot_tools, dir layout, etc.)

  3. All the better to Spy on you my dear…..when you have no idea what the tools do….hehehe, you can be SURE the NSA does! And of course, all the ‘google’ insertions on the sly to report what youve been up to.

  4. With all browsers getting into nightlies and quick release cycles, there may be no one really looking at dropping unused code or build references. That may be one reason for the hugely different source download and build sizes.

    Eric, the reference on NSA is particularly relevant.🙂
    Other than the regular contributors to FOSS, someone who is tinkering with a custom browser build is worthy of an oversight dekko.

  5. Mercurial too outputs percentage and progress when you clone or pull changes
    check out the ‘progress’ extension

  6. I have actually built all but WebKit from tarball in release mode, most recently Chromium 39. It took 73 minutes in a 4GB ramdisk.

  7. Just a small correction: The Mozilla build FAQ says, that you have to use autoconf 2.13 instead of autoconf 2.5x because features were removed, not autoconf 2.5 (that would be weird, because 2.5 is older than 2.13).

  8. Also chromium does not link on i386/i686 hardware with Linux anymore. They statically link everything in one go and need more than 4 gigs of virtual memory for that. There’s a more dynamic linking option included in chromium but that produces broken binaries…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s