Using git to push to Mozilla’s hg repositories

Last night I spent a huge amount of time working with Nicolas Pierron on setting up a two-way git-hg bridge to allow for those of us using git to push straight to try/central/inbound without manually importing patches into a local mercurial clone.

The basic design of the bridge is fairly simple: you have a local hg clone of mozilla-central, which has remote paths set up for try, central and inbound. It is set up as an hg-git hybrid and so the entire history is also visible via git at hg-repo/.hg/git. This is a fairly standard set up as far as hg-git goes.

Then on top of that, there’s a special git repository for each remote path in hg (try, central and inbound) inside .hg/repos. These are all set up with special git hooks such that when you push to the master branch of one of these repositories, they will automatically invoke hg-git, import the commits into hg and then invoke hg to push to the true remote repository on hg.mozilla.org.

Simple, right? Well, the good news is that for the most part, people shouldn’t need to actually set up this system. There is infrastructure in place to make it just look like a multi-user git repository that people can authenticate against and push to. So ultimately we can set this up on, say, git.mozilla.org and to push to try we just push to remote ssh://git.mozilla.org/try.git, or something. Authentication is handled by the system just by using ssh’s ForwardAgent option, so in theory it should be as secure as hg.mozilla.org (but don’t quote me on that!).

Now onto setting it up; first you have to clone mozilla-central from hg:

hg clone ssh://hg.mozilla.org/mozilla-central

Then edit the .hg/hgrc to contain the following:

[paths]
mozilla-central = ssh://hg.mozilla.org/mozilla-central
mozilla-inbound = ssh://hg.mozilla.org/integration/mozilla-inbound
try-pushonly = ssh://hg.mozilla.org/try
[extensions]
hgext.bookmarks =
hggit =

The -pushonly suffix on the try path tells the bridge to not bother pulling from try when synchronising the repositories. The other two will be kept in sync.

The next step is go ahead and use Ehsan’s git-mapfile to short-cut the repository creation process. By default, the bridge will use hg-git to create the embedded git repository, and doing this requires that hg-git processes every single commit in the entire repository, which takes days. The git-mapfile is the map that hg-git uses to determine which hg commit IDs correspond to which git commit IDs, and using Ehsan’s git-mapfile along with a clone of the canonical mozilla-central git repository at git://github.com/mozilla/mozilla-central.git will allow us to create these local repositories in a matter of minutes instead of days.

git clone https://github.com/ehsan/mozilla-history-tools
cp mozilla-history-tools/updates/git-mapfile /path/to/your/hg/clone/.hg/git-mapfile
cd /path/to/your/hg/clone/.hg
git clone --bare git://github.com/mozilla/mozilla-central.git git

This lays the groundwork, but there is still a little more to do. Unfortunately, this git repository contains a huge amount of commit history from the CVS era that isn’t present in the hg repositories, so if you try and push using the bridge, hg-git will see these commits that aren’t in the hg repository and try to import all these CVS commits into hg. To work around this, we can hack the git-mapfile. The basic idea here is to grab a list of all the git commit SHA1s that correspond to CVS commits, then map those in the git-mapfile to dummy hg commits (such as “0000000000000000000000000000000000000000”). Unfortunately, hg-git requires that all the mappings are unique, so we need to generate a unique dummy commit ID for each and every CVS commit in git.

If you’re using Ehsan’s repository, go ahead and just grab my git-mapfile from http://people.mozilla.org/~gwright/git-cvs-mapfile-non-unique for just the CVS commits and pre-pend that to your .hg/git-mapfile.

Now comes the fun part; setting up the bridge itself.

git clone git://github.com/nbp/spidermonkey-dev-tools.git
mkdir /path/to/your/hg/clone/.hg/bridge/
cp spidermonkey-dev-tools/git-hg-bridge/*.sh /path/to/your/hg/clone/.hg/bridge/

Then, you need to add the following to your .hg/git/config’s origin remote to ensure that the branches are set up correctly for inbound and central:

fetch = +refs/heads/master:refs/heads/mozilla-central/master
fetch = +refs/heads/inbound:refs/heads/mozilla-inbound/master

This is because pull.sh expects to find the mozilla-central history at a branch called mozilla-central/master, and the inbound history at mozilla-inbound/master.

Now to create the special push-only repositories. First pull.sh needs to be modified in order to allow for the short circuiting; temporarily remove the “return 0” call on line 112 after “No update needed”, then:

cd /path/to/your/hg/clone/.hg/bridge/
./pull.sh ../..
(Add the return 0 call back to pull.sh now)

This will create three repositories in /path/to/your/hg/clone/.hg/repos that correspond to try, mozilla-central and mozilla-inbound. If you now set these repositories as remotes in your main git working tree such as:

git remote add try /path/to/your/hg/clone/.hg/repos/try

You can just push to try by pushing to the master branch of that remote! The first push will take a while as the push-only repository has no commits in it (this should not be an issue for mozilla-central and mozilla-inbound pushes), but after that they should be nice and fast. Here’s an example:

[george@aluminium mozilla-central]$ git push -f try gwright/skia_rebase:master
Counting objects: 3028016, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (549026/549026), done.
Writing objects: 100% (3028016/3028016), 729.80 MiB | 59.21 MiB/s, done.
Total 3028016 (delta 2452303), reused 3015470 (delta 2440473)
remote: fatal: Not a valid commit name 0000000000000000000000000000000000000000
remote: Will Force update!
remote: Get git repository lock for pushing to the bridge.
remote: To ./.hg/git
remote:  * [new branch]      refs/push/master -> try-pushonly/push
remote: Get mercurial repository lock for pushing from the bridge.
remote: Convert changes to mercurial.
remote: abort: bookmark 'try-pushonly/push' does not exist
remote: importing git objects into hg
remote: pushing to ssh://hg.mozilla.org/try
remote: searching for changes
remote: remote: adding changesets
remote: remote: adding manifests
remote: remote: adding file changes
remote: remote: added 4 changesets with 7 changes to 877 files (+1 heads)
remote: remote: Looks like you used try syntax, going ahead with the push.
remote: remote: If you don't get what you expected, check http://trychooser.pub.build.mozilla.org/ for help with building your trychooser request.
remote: remote: Thanks for helping save resources, you're the best!
remote: remote: You can view the progress of your build at the following URL:
remote: remote:   https://tbpl.mozilla.org/?tree=Try&rev=b9783e130dd6
remote: remote: Trying to insert into pushlog.
remote: remote: Please do not interrupt...
remote: remote: Inserted into the pushlog db successfully.
To /home/george/dev/hg/mozilla-central/.hg/repos/try
 * [new branch]      gwright/skia_rebase -> master

And that’s it! Mad props to Nicolas Pierron for the huge amount of work he put in on building this solution. If you’re in the Mozilla Mountain View office, you can ping him to get access to his git push server, but hopefully Ehsan and I will work on getting an accessible server out there for everyone.

Booting Linux on a Retina MacBook Pro

For those of you who know me, you’ll know I have a soft spot for pixels. The smaller and more plentiful the pixels, the better. So of course, when Apple announced the new MacBook Pro with the insanely high DPI display, I hesitated for a bit, then bought one with the intention of running Linux on it natively.

Now let me prefix this with a statement: Linux on the rMBP can be a sod to get working. It is doable, but don’t expect everything to work right away.

This blog will be part of a series on getting the rMBP working well on Linux, and will only concentrate on getting it to boot via the EFI bootloader.

Getting Linux to boot

I’m using Fedora 17 because @mjg59 is a Red Hat employee, and actively working on getting Linux to play nicely with this machine. So go ahead and grab one of the CD images from http://alt.fedoraproject.org/pub/alt/live-respins/ and dump it to a USB stick, e.g. (assuming you’re doing this on a Linux box and your USB storage device is /dev/sdb):

dd if=F17-Live-DESK-x86_64-20120720.iso of=/dev/sdb bs=1M

Remember to run sync before unplugging the USB device, or you will get bizarre errors and the installer won’t run.

You’ll also need to resize your OS X partition if you want to dual boot; go ahead and resize it in Apple’s Disk Utility (in /Applications/Utilities).

Once that’s done, you can reboot the machine whilst holding down the Alt key and you’ll be presented with a list of devices you can boot from, one of which should be your USB device with the Fedora logo showing. Go ahead and boot off that.

This will bring up the grub prompt; hit ‘e’ to edit the boot configuration and append the following parameters to your kernel arguments:

nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e

That will boot you into a glorious Xorg session running at native panel resolution, and you can install Fedora as normal (sort of).

First off, you’ll need to ensure that there’s a 200MB HFS+ partition which you can install the EFI bootloader to. This should be set to mount at
/boot/efi.

Apart from that, you probably shouldn’t need to do anything special to the partition table, but here’s mine for reference:

Number  Start   End    Size   File system     Name                  Flags
1      20.5kB  210MB  210MB  fat32           EFI system partition  boot
2      210MB   256GB  256GB  hfs+            Customer
3      256GB   257GB  650MB  hfs+            Recovery HD
4      257GB   257GB  210MB  hfs+
5      257GB   258GB  524MB  linux-swap(v1)
6      258GB   500GB  243GB  ext4

Once the installation is done, you may or may not get an error installing the bootloader. If you do, you will need to fire up a terminal and do the following:

sudo chroot /mnt/sysimage
efibootmgr -c
efibootmgr -v

The last line should show Linux’s bootloader is enabled.

You then need to ensure grub’s configuration is set correctly. Go ahead and find out your kernel and initramfs files; they should be somewhere like:

/boot/vmlinuz-3.4.5-2.fc17.x86_64
/boot/initramfs-3.4.5-2.fc17.x86_64.img

Assuming they’re in /boot on /dev/sda6, this will translate to grub paths as something like:

(hd0,5)/boot/vmlinuz-3.4.5-2.fc17.x86_64
(hd0,5)/boot/initramfs-3.4.5-2.fc17.x86_64.img

Go ahead and write something like the following to /boot/efi/EFI/redhat/grub.conf:

splashimage=(hd0,5)/boot/grub/splash.xpm.gz
default=0
timeout=5
title Fedora (3.4.5-2.fc17.x86_64)
        root (hd0,5)
        kernel /boot/vmlinuz-3.4.5-2.fc17.x86_64 root=/dev/sda6 rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us SYSFONT=True rd.luks=0 ro LANG=en_US.UTF-8 rhgb quiet nointremap drm_kms_helper.poll=0 video=eDP-1:2880x1800@45e nomodeset
        initrd /boot/initramfs-3.4.5-2.fc17.x86_64.img

You will then need to symlink /etc/grub.conf to it, after which you can go ahead and reboot. You should be able to see an additional boot option now in the Alt boot menu. If not, you will need to boot into OS X and you will find a new volume labelled “untitled” most likely. You will need to bless the Linux bootloader in order to boot it, so as root in a terminal, run:

bless --folder /Volumes/untitled --file /Volumes/untitled/EFI/redhat/grub.efi

You should then be able to boot into Linux!