With some effort and frustration, I got VirtualBox running on four of our CentOS 6.3 client machines, jay, finch, lark, and robin. It can now run headless VMs that (re)start when the host (re)boots, and can even run 64-bit guests, such as the 2012 MIT/LL Capture the Flag VM. Networking is a work-in-progress; presently the guests use NAT with selected ports forwarded. Here’s how to do it.
But, first, why VirtualBox? It’s free (as in beer, and speech, except for a proprietary extension pack that provides a few extra features). And it it’s been actively developed for many years and seems pretty solid, if not quite as polished as VMWare. It’s cross-platform, running pretty-much identically on Mac, Windows, and Linux machines. And it’s backed by a giant company, Oracle, which isn’t going away anytime soon, even if it’s focused on other things and is evil.
VMWare would have been nice; it’s the king of the hill and is very mature, polished, and enterprise-y. The MIT/LL CTF VMs are native VMWare and probably easiest to run on that platform. But it costs a ton, so it’s a non-starter, unless you’re willing to use a 1-month trial, or score an activation code from the Grace Hopper conference.
There’s a free VMWare player that might work, but it doesn’t run on Macs, and lacks useful features such as (I think) snapshots and virtual network tweaking.
I’d really like to use KVM, the main Red Hat virtualization technology, to learn it. I was fiddling with it on Tempest last year and it seemed pretty nice. But, it’s Linux only, and so useless on our home machines, and I don’t have the time or patience to deal with 2 platforms at once.
So, VirtualBox it is.
BIOS Tweaks
First, reboot and enter the setup screen. Get there by hitting Enter repeatedly until the machine beeps, then hit F1.
Power –> After Power Loss:
Change from “Last State” to “Power On”. You’d think this would be unnecessary, but if power comes on, then goes off immediately, then comes back up, “Last State” doesn’t work.
Update: Maybe not. This was true for the previous micro-focus machines. At least I inferred it after most of them stayed off after a Science Center power outage last spring. But wren has “Last State” set, and I just unplugged/plugged it in a bunch of times, and it came back on every time.
Advanced –> CPU Setup:
–> Enable Intel(R) Virtualization Technology
–> Enable VT-d (this may be unnecessary)
–> Leave TxT disabled (explanation)
That will allow you to run 64-bit guests even under the 32-bit host OS. These steps required being at the machine; everything else can be done remotely.
Software Installation
Get the VBox repo and make sure everything’s up to date (note that the wget line is wrapped, but it will copy-paste just fine):
mount puma:/share /share cd /etc/yum.repos.d/ wget http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo yum update
Observe whether the kernel was upgraded at the last step, and reboot if so. If not sure, compare the outputs of ‘rpm -qa kernel |sort |tail -n 1’ and ‘uname -r’.
Oops, all packages are updated periodically, including VBox, if it’s not running. Empirically, that’s bad: /etc/vbox/vbox.cfg will be removed for no good reason, and the VMs won’t run unless the new extension pack is installed, which is a PITA. So on each machine, change the ‘enabled=1’ line in /etc/yum.repos.d/virtualbox.repo to ‘enabled=1’. And then if you want to update VBox later, you can do it manually, with ‘yum –enablerepo=virtualbox update’.
We need EPEL for dkms (Dynamic Kernel Module Support), which enables kernel device drivers, such as those for VBox, to be automatically rebuilt when a new kernel is installed:
rpm -Uvh http://epel.mirror.freedomvoice.com/6/i386/epel-release-6-7.noarch.rpm
Now install a bunch of necessary packages (all but dkms are probably already installed), then VBox:
yum install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms yum install VirtualBox-4.2
Specify who can use VBox:
usermod -aG vboxusers <user1> usermod -aG vboxusers <user2> . .
Now you can stop being root, and can start the graphical VBox Manager from the Gnome panel if you’re local, or if you’ve ssh -Y’d in, with:
VirtualBox &
VBox Manager settings
You should make a couple tweaks in the VBox Manager prefs/settings. First, the default host key for Linux is the right control key. You need this to escape from the VM to the host environment in certain situations, such as when guest additions are not installed on the client. There’s no right-ctrl on my Mac, so I make it F3, which is more OS-neutral.
Second, we need to install the proprietary Extension Pack to get VRDP support (discussed below) as well as some hardware-y stuff like USB support that we probably don’t need. This is problematic, either from the GUI prefs, or from the command line:
[cs342@finch InstallingVBoxOnClients] VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack 0%... Progress state: NS_ERROR_FAILURE VBoxManage: error: Failed to install "/home/cs342/InstallingVBoxOnClients/Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack" VBoxManage: error: The installer failed with exit code 127: Error executing command as another user: No authentication agent was found. VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ExtPackManager, interface IExtPackManager VBoxManage: error: Context: "int handleExtPack(HandlerArg*)" at line 1112 of file VBoxManageMisc.cpp
Is this because of the GID conflict between the local group vboxusers and the LDAP group cs342? No idea. But we can do it as root, provided we move the file off the root-squashed NFS-mounted filesystem:
[cs342@finch InstallingVBoxOnClients] cp Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack /tmp [cs342@finch InstallingVBoxOnClients] su Password: [root@finch InstallingVBoxOnClients] VBoxManage extpack install /tmp/Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Successfully installed "Oracle VM VirtualBox Extension Pack". [root@finch InstallingVBoxOnClients]
Now we need to do the following:
- Install a VM and set up SSH if necessary (internally) and port forwarding (externally).
- Arrange for external access to the console in case SSH isn’t working.
- Arrange for the VM to start on host boot.
I won’t discuss setting up a VM (but see the section “Putting VM files on clients” below). On finch, I’m going to install a copy of the CTF 2012 VM. SSH is already enabled; if not, ‘sudo apt-get install openssh-server’ is all you need for Ubuntu. In the VBox Manager, I’m forwarding port 2222 to 22 in Settings > Network > Advanced > Port forwarding. And now you can SSH in like so:
ssh -Y -p 2222 <username>@finch
Note that the host port needs to be >1024. I believe that the guest can only listen to ports < 1024 if it’s run as root, which sounds like a bad idea. We need to use a different port for SSH anyway because we want to be able to SSH into the host and guest separately. But if we run a web server in the guest, we’ll need to either change the port it’s listening on or forward, say, 8080 to 80.
VRDP Access (or lack thereof)
VBox allows you to run VMs “headless”, but access a virtual console via the Microsoft Remote Desktop Protocol when necessary. This requires opening up a port, default=3389. Security for this under VBox is a mess. I’ll record the details below for posterity, but I think that, for us, it should just be disabled. When the machine is running in its normal headless mode you’ll need to ssh in. If ssh is hosed or not set up yet, ssh into the host, open the VBox Manager with ‘VirtualBox &’, and enable VRDP with null authentication via Settings > Display > Remote Display > Enable Server (this can be done with the machine running). Then access the machine with RDP (more below). When done, disable the RDP server and close the VBox Manager; the VM will continue to run.
To connect to a VM on finch, say, just enter ‘rdesktop finch’ at a machine with rdesktop installed (currently jay, finch, and robin). If there are multiple VMs running the port may be greater than 3389; see the machine’s settings. Other RDP clients might work, e.g. MS’s Remote Desktop Connection for Mac, or there’s obviously a native Windows client.
An alternative to this is to shut down the and re-start the VM non-headless through the VBox Manager, bringing up the console. This requires an X11 connection. I’ve found that doing this through X11 on a Mac can cause the keyboard to be remapped unusably. This only happens sometimes, not always, and I don’t know why. Interestingly, I had very similar problems accessing KVM running on tempest through my Mac last year.
Here are some notes on VRDP authentication. As mentioned above, this is broken, and it’s probably best to use null authentication and disable it most of the time. There are four authentication modes:
- Null. Works fine, but insecure.
- Guest. Uses authentication on the guest. Experimental, not supported, and in fact it doesn’t work for us.
- External. Uses authentication on the host. Causes segfaults.
- External with simple authentication. Sort-of-works, see below.
External/simple uses authentication provided by VBox. You supply VBox with a hashed password to insert into a machine’s XML config file, <Machine Name>.vbox. Then you supply the password at login. Problem is that, for us anyway, entering the password interactively doesn’t work and you need to type:
rdesktop -u <user> -p <password> HOST
and then the password’s in your bash history file. Also, less importantly, MS’s Remote Desktop Connection for Mac doesn’t work for this.
Anyway, here’s how to set it up if you want to. I’m following Sec. 7.1.5, “RDP authentication” of the VBox manual:
[cs342@jay Minimal-Ubuntu-Server] VBoxManage setproperty vrdeauthlibrary "VBoxAuthSimple" [cs342@jay Minimal-Ubuntu-Server] VBoxManage modifyvm <Machine-name> --vrdeauthtype external [cs342@jay Minimal-Ubuntu-Server] VBoxManage internalcommands passwordhash "<password>" Password hash: 447a9e95f60aa7c586558c6192283c22786981632b035e3978c78b143d9396c3 [cs342@jay Minimal-Ubuntu-Server] VBoxManage setextradata Minimal-Ubuntu-Server "VBoxAuthSimple/users/<User>" 447a9e95f60aa7c586558c6192283c22786981632b035e3978c78b143d9396c3
VM Auto-start
To make a VM auto-start on the host’s reboot requires that we create a service and configure it appropriately. Fortunately, this capability was introduced in VBox 4.2, just released a few weeks ago. (An alternate solution from 2009 might have worked, with some customization:
http://www.kernelhardware.org/virtualbox-auto-start-vm-centos-fedora-redhat/ )
Refs:
- The latest VBox manual: Sec. 9.24.1.: “Linux: starting the autostart service via init”, and sec. 8.2, with some commands for setting machines to auto-start/stop.
- A recent blog post with more detail.
I’ll follow the blog post, mainly.
As root, create /etc/default/virtualbox, containing:
# virtualbox defaults file VBOXAUTOSTART_DB=/etc/vbox VBOXAUTOSTART_CONFIG=/etc/vbox/vbox.cfg
And create /etc/vbox/vbox.cfg containing:
# Default policy is to deny starting a VM, the other option is "allow".
default_policy=deny
# Create an entry for each user allowed to run autostart
cs342={allow=true}
fturbak={allow=true}
mdawson={allow=true}
(The examples in the manual and blog post has more line feeds, but produce errors for me. A CRLF translation issue?)
Permissions/ownership:
chown root:vboxusers /etc/default/virtualbox /etc/vbox /etc/vbox/vbox.cfg chmod 1775 /etc/vbox chmod 660 /etc/default/virtualbox /etc/vbox/vbox.cfg
It may be that /etc/default/virtualbox and /etc/vbox/vbox.cfg can have group root, not vboxusers, I’m not sure.
Now become cs342 and do the following. For modifyvm, the VM needs to
be off.
VBoxManage setproperty autostartdbpath /etc/vbox VBoxManage modifyvm <Machine Name> --autostart-enabled on VBoxManage modifyvm <Machine Name> --autostop-type [disabled|savestate|poweroff|acpishutdown]
(I’m going with savestate, but maybe acpishutdown would be safer)
Problems:
- ‘service vboxautostart-service start’ indicates that it’s starting VMs, but doesn’t.
- ‘service vboxautostart-service stop’ doesn’t stop running VMs
- The ‘savestate’ option doesn’t work; apparently the plug is pulled on the VM, since it will indicate zero uptime when restarted. Is some sort of delay required? Or is this connected to the fact that ‘service vboxautostart-service stop’ doesn’t work?
However, the autostart-enabled machines do start up on host system boot (albeit without saved state), which is the main thing we want.
Wait, hold on, that’s true on jay, finch, and lark, but not robin. ‘VBoxManage modifyvm CTF-2012-robin –autostart-enabled on’ doesn’t create the 1-byte ‘cs342.start’ and ‘cs342.stop’ files in /etc/vbox that it does on the other machines. Don’t know why, despite a fruitless comparison of configurations, permissions and ownership, etc. on finch vs. robin. The steps I took setting up the CTF-2012-robin VM were a little different than for the others, but I don’t know what could have created this difference. Anyway, manually creating these two files with the right contents (a single ‘1’) and permownership did the trick.
To shutdown a running VM properly, log in to it and execute ‘shutdown -P now’. Use -P, not -h, since -h causes the VM to get stuck in the GRUB boot screen on restart. As far as I can tell, you can’t effect a controlled shutdown externally; the best you can do is
VBoxManage controlvm <vm> savestate
To start a VM that you’ve shut down without rebooting, you should use VBoxHeadless. Otherwise your VM will die when your session closes:
VBoxHeadless --startvm <vm> --vrde off &
Moving Configuration and VM files to clients
By default, the VM files live in ~/VirtualBox\ VMs. That’s on the NFS server, so access is slower, and VMs are simultaneously accessible, and even simultaneously runnable, which is bad! So let’s put the VM files on the clients. Note that the VMs will no longer be automatically backed up, which is a problem I should think about.
Also, a user’s VBox config and log directory is by default ~/.Virtualbox/. This means that all hosts running VBox under the same user will share configuration, including the list of available VMs; again, bad. So lets move that too. This can be done with symlinks to the local file system or by changing the environment variable VBOX_USER_HOME; since VBox starts running in various ways and at various times, including soon after boot, and I’m not an expert on environment variables, I’ll use symlinks.
So I’ll create /var/VBox-local/<account-name>/ for each account, containing subdirectories ‘VMs’ and ‘Config’, and symlink these from ~acount-name/VirtualBox\ VMs and ~account-name/.VirtualBox, respectively. As root:
mkdir /var/VBox-local chgrp vboxusers /var/VBox-local/ chmod 770 /var/VBox-local/ mkdir /var/VBox-local/cs342 chown cs342 /var/VBox-local/cs342/ chmod 770 /var/VBox-local/cs342/
Now backup ~/.Virtualbox by mv-ing it to ~/.Virtualbox-backup, and likewise for ~/VirtualBox\ VMs. The second backup takes up lots of space, so get rid of it once everything’s running properly. Create the symlinks mentioned above, which you only have to do once. Then on each machine that will be running VBox:
- cp -a ~/.Virtualbox /var/VBox-local/cs342/Config
- cp -a ~/VirtualBox\ VMs /var/VBox-local/cs342/VirtualBox\ VMs
- Remove VMs you don’t want on this machine
- Remove the associated lines from the <MachineRegistry> and any other sections of …Config/VirtualBox.xml
Through the VBox Manager, each machine’s settings should be tweaked like so:
- Choose a new random Mac address if another copy of the same VM exists on another host and there’s any chance they’ll be on the same network.
- Adjust port forwarding if necessary
- Turn off the Remote Display server
- Under Storage, select the controller for the VMs virtual disk, and click ‘Use Host I/O Cache’. Otherwise, you’ll get a scary warning about a linux kernel bug possibly corrupting the virtual disk when it’s mounted on an ext4 partition.
External Port Forwarding
We need to be able to ssh independently to both a VM and its host, so I’ve been forwarding external port 2222 to the VM’s port 22. Why not 222? Because non-root processes can’t serve ports below 1024 on Linux. Likewise, I’ve been forwarding port 8080 to 80 for HTTP, even though the hosts aren’t running web servers. Now to connect from outside the host, we need to open host ports. I’ll open 2222, and forward 80 to 8080 so it will look from the outside like we’re using the standard HTTP port. This is pretty basic firewall stuff so we can just use system-config-firewall. Use “Other Ports”, “User Defined” to open a port, and “Port Forwarding”, “Local Forwarding” to forward a port.
If there’s a possibility of running multiple VMs simultaneously on a host, you’ll need to be careful about that, e.g. use ports 2223, 2224, etc. for SSH, and likewise for HTTP.
The CTF VM uses WordPress, which has been giving me huge headaches with port forwarding. I was able to get it working with an open high port, so I may do it that way rather than forwarding to port 80.
Setting up a Virtual Network Environment on One Host (added 11/25/2012)
I’m helping a student with a botnet project, which will require a little network of VMs. We want the network to have these properties:
- VMs should be able to freely communicate with each other, without our having to set up lots of complicated ad hoc port forwarding rules.
- VMs should be able to connect to the internet for software downloads, etc.
- Host should be able to connect to VMs via SSH at least. Connection from global internet to host isn’t necessary.
In sensible virtualization environments like VMWare and (I think) KVM, all 3 can be satisfied with NAT, which behaves like a home router setup. But VMWare costs money, and KVM requires a 64-bit host OS; it won’t work on our 32-bit CentOS machines. Maybe we should have installed 64-bit; the hardware would accept it. But it’s too late now.
VBox’s NAT setup won’t work because the guests are isolated from one another, unlike VMWare and KVM. Don’t know why, but that’s the way it is.
So we’ll use VBox’s host-only network. That satisfies requirements 1 and 3. For #2, we’ll modify the host’s firewall. (I’ve also seen suggestions that one use both NAT and host-only on each guest. That sounds complicated. Maybe it’s not–I didn’t try it–but I’d prefer that the student’s VM network settings be as simple as possible.)
I’m going to set everything up on one host, finch. I hope the resources will be sufficient. Memory-wise, finch has 4 GB, so 6 VMs using 1/2 GB each will leave 1 GB for the host, which should be enough. I don’t think the VMs will need lots of simultaneous CPUs; they’d better not, since there are only 2 of them. There shouldn’t be a lot of network traffic. Hard drive space is a concern, there’s only 30 GB available of a 50GB hard drive (small nowadays). The student could save space by using linked clones. And if things get tight, I could free up 7 GB by deleting a couple VMs.
If we needed to split things up on multiple hosts it would be more complicated–maybe we’d use openvpn?
Anyway, first, I shut down the running VM, and prevented it from auto-starting with:
VBoxManage modifyvm <machine> --autostart-enabled off
Then, on finch, I added the user to the vboxusers group and set up her local VBox config and VM directories:
# Allow to use VBox [root@finch ~] usermod -aG vboxusers <user> # Set up directories and fix perms/ownership [root@finch ~] cd /var/VBox-local/ [root@finch VBox-local] mkdir -p -m770 <user>/VMs [root@finch VBox-local] mkdir -m700 <user>/Config [root@finch VBox-local] chown -R <user> <user> [root@finch VBox-local] chgrp vboxusers <user>/* # Make symlinks. Need to be <user> since root on # public clients not trusted [root@finch <user>] su - <user> [<user>@finch ~] ln -s /var/VBox-local/<user>/VMs/ VirtualBox\ VMs [<user>@finch ~] ln -s /var/VBox-local/<user>/Config/ .VirtualBox
Then I logged in to finch as this user, started up VBox with ‘Virtualbox &’, and added the default host-only network vboxnet0 with its default settings.
At this point, VMs on the host-only network should be able to communicate with one another, and the host should be able to talk to them. But we want the VMs to be able to connect to the wider internet. So we add iptables rules to finch’s firewall.
I learned a lot about iptables at this point. My main references were:
- The man page for iptables
- The book Linux Firewalls
- The Packet Filtering HOWTO
But, I ended up basically copying the rules for KVM’s virtual bridge and IP range (virbr0, 192.168.122.0/24) over for VBox (vboxnet0, 192.168.56.0/24). To add the new rules, I used system-config-firewall, which I’d already used to open/forward some ports for the CTF practice VMs. In the “Custom Rules” section, I specified files containing the new rules, one each for the forward, nat, and mangle tables. The file contents are copied below for reference.
Probably some of these rules are unnecessary or redundant. But they seem to work, and finding a minimal set isn’t worth my time right now.
Now we just give the student some advice about working in this new environment:
- Use static IPs in 192.168.56.(2-99).
- Make sure the MAC addresses are different. When you change MACs on Ubuntu machines, there’s a network config problem; it’s fixed by deleting /etc/udev/rules.d/70-persistent-net.rules and rebooting.
- Look into VBoxManage for scripting.
- Run headlessly with VBoxHeadless to save resources and prevent the VMs from dying when the connection drops (make sure you can ssh in first).
And hopefully everything will Just Work. We’ll see…
Custom Firewall Rules:
[root@finch MikeVBox-relatedIPtablesFiddling] pwd /root/MikeVBox-relatedIPtablesFiddling [root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-filter-rules # VBox iptables rules for the filter table # Loaded with system-config-firewall's custom rules # # Cribbed from the firewall rules created by libvirtd # (obtained by 'iptables-save|grep virbr0 > virbr0-rules', # and see http://libvirt.org/firewall.html ) # # Accept DNS and DHCP requests from guests (not sure I need this) -A INPUT -i vboxnet0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i vboxnet0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i vboxnet0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i vboxnet0 -p tcp -m tcp --dport 67 -j ACCEPT # # Allow incoming connections iff related to existing outbound connections -A FORWARD -d 192.168.56.0/24 -o vboxnet0 -m state --state RELATED,ESTABLISHED -j ACCEPT # # ... and allow outgoing connections -A FORWARD -s 192.168.56.0/24 -i vboxnet0 -j ACCEPT ## # Allow guests to communicate among themselves -A FORWARD -i vboxnet0 -o vboxnet0 -j ACCEPT # # Nuke everything else to/from vboxnet0 -A FORWARD -o vboxnet0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i vboxnet0 -j REJECT --reject-with icmp-port-unreachable [root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-nat-rules # VBox iptables rules for the nat table # Loaded with system-config-firewall's custom rules # # Cribbed from the firewall rules created by libvirtd # (manually copied and altered from the nat section of iptables-save) # -A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -j MASQUERADE [root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-mangle-rules # VBox iptables rules for the mangle table # Loaded with system-config-firewall's custom rules # # This has something to do with DHCP replies. Not sure I need it. # Wish I understood it better. -A POSTROUTING -o vboxnet0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill [root@finch MikeVBox-relatedIPtablesFiddling]
2 Responses to Installing VirtualBox on our Linux Clients