sourcediver.org

about software and freediving

Running Arch Linux ARM With Xen on Cubieboard 2

Cubieboard

I recently bought a Cubieboard 2 to do some benchmarking for my bachelor’s thesis. I wanted to benchmark the software ethernet switching (also known as bridging) capabilities of a mid-range ARM platform. Since Xen 4.4 has full support for the Allwinner A20/A30 processors, the cubieboard was an obvious choice. There are not many instructions out there how to install Xen and Linux on the Cubieboard 2. One of the best is Running Xen on the Cubieboard 2, where I borrowed a lot of instructions. But since I am a big fan of Arch Linux ARM, I am going to demonstrate here how to get Arch Linux running as your dom0 and domU.

Console output in this post

  • all commands without user@system means that you need to run them on your workstation/laptop
  • all commands with root@dom0 / root@alarm need to be executed on the dom0
  • all commands with root@domU need to be executed on the domU

IPs used in this post

  • dom0 has 192.168.178.70/24
  • domU has 192.168.178.71/24

Installing the arm toolchain

Arch Linux: Simply install this package from the AUR (e.g. using yaourt).

1
yaourt -S arm-linux-gnueabihf-gcc

For other distributions: you need the arm-linux-gnueabihf toolchain (hf = Hard Float).

Compiling uboot

Clone the u-boot-sunxi repository and switch to the sunxi-next branch as this has support for the ARMv7 virtualization extensions.

1
2
3
git clone git://github.com/jwrdegoede/u-boot-sunxi.git
cd u-boot-sunxi
git checkout -b sunxi-next origin/sunxi-next

Compile the image…

1
2
make CROSS_COMPILE=arm-linux-gnueabihf- Cubieboard2_config
make CROSS_COMPILE=arm-linux-gnueabihf- -j 9

Create a directory boot with this file (name it boot.cmd) in it.

We now need to compile the config.

1
2
cd boot
../tools/mkimage -C none -A arm -T script -d "boot.cmd" "boot.scr"

Compiling Linux

I tried the mainline kernel (with commit id e99cfa2d) but somehow the Kernel does not start. I will try to use the mainline Kernel from time to time but for now we have to use the sunxi port.

UPDATE: The mainline kernel does work now. Maybe I just had the wrong options enabled!

For linux-sunxi:

1
2
3
git clone git@github.com:linux-sunxi/linux-sunxi.git
cd linux-sunxi
git checkout sunxi-devel

Delete .config if it exists and download my .config

1
2
rm .config
wget https://gist.githubusercontent.com/mguentner/29d40fa3937d86a311a3/raw/142a6fd8b92da6684f33458503ad52e3056ecd95/.config

For mainline:

1
2
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux

Delete .config if it exists and download my .config

1
2
3
rm .config
wget https://gist.githubusercontent.com/mguentner/361deb378fc26045b094/raw/dba341396f8d0afe1b32f081e28a8b3b8a371821/.config
make ARCH=arm oldconfig

Start the make process

1
make ARCH=arm zImage dtbs -j 9

Compiling Xen

1:1 copy of http://openmirage.org/wiki/xen-on-cubieboard2#BuildingXen

Clone the Xen repository.

1
2
3
git clone git://xenbits.xen.org/xen.git
cd xen
git checkout stable-4.4

Optional: You can change debug ?= in Config.mk from n to y to enable debugging within the Xen console.

Start compiling the Xen binary.

1
make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabihf- CONFIG_EARLY_PRINTK=sun7i -j9

Partitioning

Partition your SD card (here /dev/sdh) with the following settings.

1
2
3
16 MB, FAT (c)
 4 GB, Linux (83)
 100%, Linux LVM (8e)

Example:

1
2
3
4
5
6
7
8
9
10
11
Disk /dev/sdh: 7.3 GiB, 7822376960 bytes, 15278080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7d72023a

Device         Boot     Start       End  Blocks  Id System
/dev/sdh1                2048     34815   16384   c W95 FAT32 (LBA)
/dev/sdh2               34816   8423423 4194304  83 Linux
/dev/sdh3             8423424  15278079 3427328  8e Linux LVM

Create the filesystems

1
2
mkfs.vfat /dev/sdh1
mkfs.ext4 /dev/sdh2

Flash and install the bootloader

Inside the u-boot-sunxi directory:

1
dd if=u-boot-sunxi-with-spl.bin of=/dev/sdh bs=1024 seek=8

Mount the first (FAT) partition and copy the necessary files.

1
2
3
4
5
6
7
8
9
sudo mount /dev/sdh1 /mnt/cubie-boot
cd u-boot-sunxi
cp boot/boot.scr /mnt/cubie-boot/
cd ../linux-sunxi
cp arch/arm/boot/zImage /mnt/cubie-boot/vmlinuz
cp arch/arm/boot/dts/sun7i-a20-cubieboard2.dtb /mnt/cubie-boot/
cd ../xen
cp xen/xen /mnt/cubie-boot/
sudo umount /mnt/cubie-boot

Install the Archlinux ARM dom0 Root FS

Download the rootfs.

1
wget http://archlinuxarm.org/os/ArchLinuxARM-sun7i-latest.tar.gz

Mount the second partition.

1
sudo mount /dev/sdh2 /mnt/cubie-root

Extract the image

1
sudo tar -xf ArchLinuxARM-sun7i-latest.tar.gz -C /mnt/cubie-root

Umount and sync

1
2
sudo umount /mnt/cubie-root
sync

Booting

We can now finally boot the Cubieboard 2. Remove the SD Card from your SD Reader and pop it into the Cubieboard 2.

Connect an UART-RS232 converter to the 4 pins next to the silicon.

Fire up your favorite terminal application, I use minicom, and set it to 115200 8N2.

You should now see this rewarding output.

Change the root password

1
2
3
4
5
[root@alarm ~]# passwd
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
[root@alarm ~]# 

Set the hostname of your dom0

1
[root@alarm ~]# echo dom0 > /etc/hostname

You might also append your hostname in /etc/hosts

[Optional] Remove unneeded packages

1
2
3
4
5
6
7
8
9
10
11
[root@dom0 ~]# pacman -R linux-firmware linux-sun7i
checking dependencies...

Packages (2): linux-firmware-20140603.a4f3bc0-1  linux-sun7i-3.4.90-4

Total Removed Size:   13.51 MiB

:: Do you want to remove these packages? [Y/n] y
(1/2) removing linux-sun7i       [############################################################################] 100%
(2/2) removing linux-firmware    [############################################################################] 100%
synchronizing filesystem...

Update your system

Do not select y when you are asked to flash a new version of uboot!

1
[root@dom0 ~]# pacman -Syu

Install openssh

1
[root@dom0 ~]# pacman -S openssh

Bridge Setup

Install bridge-utils

1
[root@dom0 ~]# pacman -S bridge-utils

Configure eth0 to be our uplink port

1
2
3
4
5
[root@dom0 ~]# cat /etc/netctl/eth0 
Description='Static ethernet connection for dom0'
Interface=eth0
Connection=ethernet
IP=no

Configure our bridge

1
2
3
4
5
6
7
8
9
10
[root@dom0 ~]# cat /etc/netctl/xenbridge 
Description="Xen bridge connection"
Interface=xenbr0
Connection=bridge
IP=static
Address=192.168.178.70/24
Gateway=192.168.178.1
DNS=(8.8.8.8)
BindsToInterface=('eth0')
ExecUpPost=('brctl addif xenbr0 eth0')

The last line is a hack since netctl 1.7 together with the custom kernel is somehow unable to add interfaces to the bridge. Bug Report

Enable and start both configs, eth0 and xenbridge:

1
2
3
4
[root@dom0 ~]# netctl enable eth0 
[root@dom0 ~]# netctl enable xenbridge
[root@dom0 ~]# netctl start eth0 
[root@dom0 ~]# netctl start xenbridge

Start the SSH Server

1
[root@dom0 ~]# systemctl start sshd

Connect to dom0 using ssh

1
ssh root@192.168.178.70

Install build tools

You can also cross-compile the tools on your development machine and skip the following steps! HowTo

1
[root@dom0 ~]# pacman -S base-devel python2 libaio yajl dtc

Select ‘all’ from the base-devel selection.

Install Xen Tools

Download this package and extract it

1
2
3
4
[root@dom0 ~]# wget http://www.sourcediver.org/downloads/xen-4.4.0-0.src.tar.gz
[root@dom0 ~]# tar xfz xen-4.4.0-0.src.tar.gz
[root@dom0 ~]# cd xen
[root@dom0 ~]# makepkg --asroot # I know...

Grab some lunch, work out or head over to reddit as this will take forever.

After an eternity, install the package

1
[root@dom0 ~]# pacman -U xen-4.4.0-0-armv7h.pkg.tar.xz

and start the Xen Services

1
2
3
4
[root@dom0 ~]# systemctl enable xenstored.service
[root@dom0 ~]# systemctl enable xenconsoled.service
[root@dom0 ~]# systemctl start xenstored.service
[root@dom0 ~]# systemctl start xenconsoled.service

You should now be able to list your domains (only dom0 at this point)

1
2
3
[root@dom0 ~]# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   256     2     r-----      17.8

Setup the LVM

1
2
3
4
[root@dom0 ~]# pvcreate /dev/mmcblk0p3 
  Physical volume "/dev/mmcblk0p3" successfully created
[root@dom0 ~]# vgcreate VolGroup00 /dev/mmcblk0p3
  Volume group "VolGroup00" successfully created

Setup the first DomU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@dom0 ~]# lvcreate -L 2G VolGroup00 --name archarm-guest-1
  Logical volume "archarm-guest-1" created
[root@dom0 ~]# mkfs.ext4 /dev/VolGroup00/archarm-guest-1
mke2fs 1.42.10 (18-May-2014)
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 6a01126f-48d8-4058-89d4-ddf499d3e223
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 
[root@dom0 ~]# mount /dev/VolGroup00/archarm-guest-1 /mnt/arch-guest/

Install pacstrap

1
[root@dom0 ~]# pacman -S arch-install-scripts

And install the domU root FS.

1
[root@dom0 ~]# pacstrap -i -c -d /mnt/arch-guest/ base systemd

Create a new directory for the custom kernel (the Arch Linux ARM kernel does not have the Xen extensions).

1
[root@dom0 ~]# mkdir /kernels

Now use scp to copy the kernel to that dir.

1
2
cd linux-sunxi
scp arch/arm/boot/zImage root@192.168.178.70:/kernels/.

Create /etc/xen/arch_domu_1.cfg with this content.

Edit /mnt/arch-guest/etc/fstab

1
2
3
4
5
# 
# /etc/fstab: static file system information
#
# <file system> <dir>   <type>  <options>       <dump>  <pass>
/dev/xvda       / ext4   rw,relatime,data=ordered       0 1

Finally, set the hostname.

1
[root@dom0 ~]# echo domU > /mnt/arch-guest/etc/hostname

and umount the partition

1
[root@dom0 ~]# umount /mnt/arch-guest/

You can now start your guest/domU by running

1
[root@dom0 ~]#  xl create /etc/xen/arch_domu_1.cfg -c         

Once you see the login prompt, login with root (no password needed).

You can now configure this guest. A good starting point is to bring up the network.

First create /etc/netctl/eth0

1
2
3
4
5
6
7
Description='static ethernet connection for the first domU'
Interface=eth0
Connection=ethernet
IP=static
Address=('192.168.178.71/24')
Gateway='192.168.178.1'
DNS=('192.168.178.1')

and start/enable it.

1
2
[root@domU ~]# netctl enable eth0
[root@domU ~]# netctl start eth0

You should now be able to ping the domU from your workstation.

1
2
3
4
ping 192.168.178.71
PING 192.168.178.71 (192.168.178.71) 56(84) bytes of data.
64 bytes from 192.168.178.71: icmp_seq=1 ttl=64 time=0.675 ms
64 bytes from 192.168.178.71: icmp_seq=2 ttl=64 time=0.714 ms

You can continue by installing ssh etc.

Profit!

Have fun with your embedded virtualized Arch Linux cluster!

Comments