An Introduction to Xen, Presentation to Linux Users Victoria, July Meeting. 2007 First Slide: Xen running NetBSD and three Linux distributions simultaneously The cup half full... The cup half empty... Twice as much cup for what one is using leads to; ================================================= Slide 1: Efficient Allocation of Scarce Resources ================================================= Server Consolidation. Move multiple servers onto a single physical host with performance and fault isolation provided at the virtual machine boundaries. Hardware Independence. Allow legacy applications and operating systems to exploit new hardware. Multiple OS Configurations. Run multiple operating systems simultaneously, for development or testing purposes. Kernel Development. Test and debug kernel modifications in a sand-boxed virtual machine -- no need for a separate test machine. Cluster Computing. Management at VM granularity provides more flexibility than separately managing each physical host, but better control and isolation than single-system image solutions, particularly by using live migration for load balancing. Hardware support for custom OSes. Allow development of new OSes while benefiting from the wide-ranging hardware support of existing OSes such as Linux. ====================================== Slide 2: Virtualisation, Emulation etc ====================================== In computing, virtualization is a broad term that refers to the abstraction of computer resources. The term is an old one: Virtualisation The original sense of the term virtualisation, dating from the 1960s, is in the creation of a virtual machine using a combination of hardware and software. The term virtual machine apparently dates from the IBM M44/44X experimental paging system and in the System/360 Model 67. With the ability to run operating systems on other operating systems, the term hypervisor resulted in the 1970s. The VMM ran directly on the underlying hardware, permitting multiple virtual machines (VMs). P-Code P-code (or pseudo-code) machine. P-code is a machine language that is executed in a virtual machine rather than in actual hardware. P-code was made famous in the early 1970s by the University of California, San Diego (UCSD) Pascal system, which compiled Pascal programs into P-code, and then executed them on a P-code virtual machine. This allowed P-code programs to be highly portable and run anywhere a P-code virtual machine was available; Modula-2 DEC-8 running on an Amiga (1990). Java Virtual Machine (JVM) The Java language followed the P-code model for its virtual machine. This permitted the wide distribution of Java programs over countless architectures by simply porting the JVM. Virtualisation and games Multiple-Arcade Machine Emulator (MAME). MAME is a full machine emulator of many arcade games of the past. In addition to virtualizing the processors used in those games, the entire machine is virtualized, including sound and graphics hardware and controls. Types of virtualisation Native virtualization and full virtualization The virtual machine simulates enough hardware to allow an unmodified "guest" OS (one designed for the same CPU) to be run in isolation. Hardware enabled virtulalization The virtual machine has its own hardware and allows a guest OS to be run in isolation. In many instances the virtual machine runs an operating system different than that of the host computer. Partial virtualization (and including "address space virtualization") The virtual machine simulates multiple instances of much (but not all) of an underlying hardware environment, particularly address spaces. Such an environment supports resource sharing and process isolation, but does not allow separate "guest" operating system instances. Paravirtualization The virtual machine does not necessarily simulate hardware, but instead (or in addition) offers a special API that can only be used by modifying the "guest" OS. e.g., Xen Operating system-level virtualization Virtualises a physical server at the operating system level, enabling multiple isolated and secure virtualized servers to run on a single physical server. The "guest" OS environments share the same OS as the host system – i.e. the same OS kernel is used to implement the "guest" environments. e.g., OpenVZ and FreeBSD Jails. Application Virtualization Runs a desktop or server application locally, using local resources, within an appropriate virtual machine. e.g., Sun's Java Virtual Machine. An attempt at a definition: "Software that pretends to be hardware" is emulation, "Software that encapsulates and grants access to physical devices" is virtualisation. ====================== Slide 3: Xen's Friends ====================== Linux-related virtualization projects Project Type License Bochs Emulation LGPL QEMU Emulation LGPL/GPL VMware Full virtualization Proprietary z/VM Full virtualization Proprietary Xen Paravirtualization GPL UML Paravirtualization GPL Linux-VServer Operating system-level virtualization GPL OpenVZ Operating system-level virtualization GPL Bochs is an x86 computer simulator that is portable and runs on a variety of platforms, including x86, PowerPC, Alpha, SPARC, and MIPS. What makes Bochs interesting is that it doesn't just simulate the processor but the entire computer, including the peripherals, such as the keyboard, mouse, video graphics hardware, network interface card (NIC) devices, and so on. Bochs can be configured as an older Intel 386, or successor processors such as the 486, Pentium, Pentium Pro, or a 64-bit variant. QEMU (emulation) QEMU supports two modes of operation. The first is the Full System Emulation mode which emulates a full personal computer (PC) system with processor and peripherals. The second mode is User Mode Emulation. In this mode (Linux only) a binary for a different architecture can be launched. VMware (full virtualization) VMware is a commercial solution for full virtualization. A hypervisor sits between the guest operating systems and the bare hardware as an abstraction layer. This abstraction layer allows any operating system to run on the hardware without knowledge of any other guest operating system. The entire virtualized environment is kept as a file, meaning that a full system (including guest operating system, VM, and virtual hardware) can be easily and quickly migrated to a new host for load balancing. z/VM (full virtualization) IBM's operating system hypervisor for the System The z/VM can also emulate a guest local area network (LAN) virtually for those guest operating systems that want to communicate with each other. This is emulated entirely in the hypervisor, making it highly secure. Xen (paravirtualization) Xen is a free open source solution for operating system-level paravirtualization from XenSource. Recall that in paravirtualization the hypervisor and the operating system collaborate on the virtualization, requiring operating system changes but resulting in near native performance. As Xen requires collaboration (modifications to the guest operating system), only those operating systems that are patched can be virtualized over Xen. From the perspective of Linux, which is itself open source, this is a reasonable compromise because the result is better performance than full virtualization. But from the perspective of wide support (such as supporting other non-open source operating systems), it's a clear disadvantage. It is possible to run Windows as a guest on Xen, but only on systems running the Intel Vanderpool or AMD Pacifica. Other operating systems that support Xen include Minix, Plan 9, NetBSD, FreeBSD, and OpenSolaris. User-mode Linux (paravirtualization) User-mode Linux (UML) allows a Linux operating system to run other Linux operating systems; each guest Linux operating system exists within a process of the host Linux operating system. Linux-VServer (operating system-level virtualization) Linux-VServer provides operating system-level virtualization. Linux-VServer virtualizes the Linux kernel so that multiple user-space environments, otherwise known as Virtual Private Servers (VPS) The Linux-VServer is supported by both the 2.4 and 2.6 Linux kernels and operates on a number of platforms, including x86, x86-64, SPARC, MIPS, ARM and PowerPC. OpenVZ (operating system-level virtualization) OpenVZ is another operating system-level virtualization solution, like Linux-VServer, but it has some interesting differences. OpenVZ is a virtualization-aware (modified) kernel that supports isolated user-spaces, VPS, with a set of user-tools for management. For example, you can easily create a new VPS from the command line: ======================== Slide 5: Xen's Structure ======================== Xen was first described in a paper presented at SOSP in 20031.1, and the first public release (1.0) was made that October. Since then, Xen has significantly matured and is now used in production scenarios on many sites. Features: * Virtual machines with performance close to native hardware. * Live migration of running virtual machines between physical hosts. * Up to 32 virtual CPUs per guest virtual machine, with VCPU hotplug. * x86/32, x86/32 with PAE, and x86/64 platform support. * Intel Virtualization Technology (VT-x) for unmodified guest operating systems (including Microsoft Windows). * Excellent hardware support (supports almost all Linux device drivers). Xen runs on the x86 architecture, requiring a P6 or newer processor (e.g. Pentium Pro, Celeron, Pentium II, Pentium III, Pentium IV, Xeon, AMD Athlon, AMD Duron). Multiprocessor machines are supported, and there is support for HyperThreading (SMT). In addition, ports to IA64 and Power architectures are in progress. The default 32-bit Xen supports up to 4GB of memory. However Xen 3.0 adds support for Intel's Physical Addressing Extensions (PAE), which enable x86/32 machines to address up to 64 GB of physical memory. Xen 3.0 also supports x86/64 platforms such as Intel EM64T and AMD Opteron which can currently address up to 1TB of physical memory. Xen is an open-source para-virtualizing virtual machine monitor (VMM), or "hypervisor", for the x86 processor architecture. Xen can securely execute multiple virtual machines on a single physical system with close-to-native performance. A Xen system has multiple layers, the lowest and most privileged of which is Xen itself. Xen may host multiple guest operating systems, each of which is executed within a secure virtual machine. In Xen terminology, a domain. Domains are scheduled by Xen to make effective use of the available physical CPUs. Each guest OS manages its own applications. This management includes the responsibility of scheduling each application within the time allotted to the VM by Xen. The first domain, domain 0, is created automatically when the system boots and has special management privileges. Domain 0 builds other domains and manages their virtual devices. It also performs administrative tasks such as suspending, resuming and migrating other virtual machines. Within domain 0, a process called xend runs to manage the system. Xend is responsible for managing virtual machines and providing access to their consoles. Commands are issued to xend over an HTTP interface, via a command-line tool. ===================== Slide 6: Installation ===================== Start with a minimal Linux install. e.g., a disk partition of; /boot (100 MB) /swap (2 * RAM) / (1.5 GB) plus: LV (everything else, 'xenimages') You will need iproute, python, python-twisted, bridge-utils. Extracting and instal Xen binary tar -zxf xen-3.0.2-install-x86_32.tgz; cd xen-3.0.2-2-install; ./install.sh; Or; Pre-built RPMs are available for download from the XenSource downloads page: http://www.xensource.com/downloads/ Once you've downloaded the RPMs, you typically install them via the RPM commands: # rpm -iv rpmname Users of the XenLinux 2.6 kernel should disable Thread Local Storage (TLS) (e.g. by doing a mv /lib/tls /lib/tls.disabled) before attempting to boot a XenLinux kernel2.4. You can always reenable TLS by restoring the directory to its original location (i.e. mv /lib/tls.disabled /lib/tls). The reason for this is that the current TLS implementation uses segmentation in a way that is not permissible under Xen. If TLS is not disabled, an emulation mode is used within Xen which reduces performance substantially. An entry should be added to grub.conf (often found under /boot/ or /boot/grub/) to allow Xen / XenLinux to boot. This file is sometimes called menu.lst, depending on your distribution. The entry should look something like the following: title Xen 3.0 / XenLinux 2.6 kernel /boot/xen-3.0.gz dom0_mem=262144 module /boot/vmlinuz-2.6-xen0 root=/dev/sda4 ro console=tty0 Check this! grub often breaks Xen after an update. ================================ Slide 7: Create Virtual Machines ================================ Before you can start an additional domain, you must create a configuration file. We provide two example files which you can use as a starting point: * /etc/xen/xmexample1 is a simple template configuration file for describing a single VM. * /etc/xen/xmexample2 file is a template description that is intended to be reused for multiple virtual machines. Setting the value of the vmid variable on the xm command line fills in parts of this template. An example configuration file. kernel = "/boot/vmlinuz-2.6.16-xen" ramdisk = "/boot/initrd-2.6.16-xen.img" memory = 700 vcpus = "2" name = "virtualmachine01" #vif = ['bridge=xenbr0'] vif = ['mac=22:11:33:21:30:34,bridge=xenbr0'] disk = ['phy:xenimages/virtualmachine01-root,hda1,w','phy:xenimages/virtualmachine01-swap,hda2,w'] ip = "10.0.1.101" netmask = "255.255.255.0" gateway = "10.0.1.1" hostname = "virtualmachine01" root = "/dev/hda1 ro" extra = "4" If you do not set this variable, xend will automatically generate a random MAC address from the range 00:16:3E:xx:xx:xx, assigned by IEEE to XenSource as an OUI (organizationally unique identifier). XenSource Inc. gives permission for anyone to use addresses randomly allocated from this range for use by their Xen domains. # Initial setup of Xen image environment # Assumes a volume group called "xenimages" already exists # Create the logical volume to store the guest image lvcreate -L1000 -tguest xenimages mkfs.ext3 /dev/xenimages/guest mkdir /xenimages/mnt mount -o loop /home/ivt/guest.img /xenimages/mnt mkdir /xenimages/guest mount /dev/xenimages/guest /xenimages/guest Assuming you've created a configuration file myvmconf based around /etc/xen/xmexample2, to start a domain with virtual machine ID 1 you can type: # xm create -c myvmconf vmid=1 The -c switch causes xm to turn into the domain's console after creation. The vmid=1 sets the vmid variable used in the myvmconf file. It is possible to have certain domains start automatically at boot time and to have dom0 wait for all running domains to shutdown before it shuts down the system. To specify a domain is to start at boot-time, place its configuration file (or a link to it) under /etc/xen/auto/. =========================== Slide 9: Xend and XM Tools =========================== The Xen Daemon (Xend) performs system management functions related to virtual machines. It forms a central point of control for a machine and can be controlled using an HTTP-based protocol. Xend must be running in order to start and manage virtual machines. Xend must be run as root because it needs access to privileged system management functions. A small set of commands may be issued on the xend command line: # xend start start xend, if not already running # xend stop stop xend if already running # xend restart restart xend if running, otherwise start it # xend status indicates xend status by its return code Usage: xm [args] Control, list, and manipulate Xen guest instances xm common subcommands: console Attach to domain DomId's console. create [-c] [Name=Value].. Create a domain based on Config File destroy Terminate a domain immediately help Display this message list [--long] [DomId, ...] List information about domains mem-set Adjust the current memory usage for a domain migrate Migrate a domain to another machine pause Pause execution of a domain reboot [-w][-a] Reboot a domain restore Create a domain from a saved state file save Save domain state (and config) to file shutdown [-w][-a][-R|-H] Shutdown a domain top Monitor system and domains in real-time unpause Unpause a paused domain vcpu-set Set the number of VCPUs for a domain can be substituted for in xm subcommands. Some examples; ivt@xen105:~$ sudo xm reboot web155 One useful command is # xm list which lists all domains running in rows of the following format: name domid memory vcpus state cputime Xen configuration files contain the following standard variables. Unless otherwise stated, configuration items should be enclosed in quotes; see the configuration scripts in /etc/xen/ for concrete examples. To create a domain, first mount the image... ivt@xen1:/dev$ sudo -s root@xen1:/dev# mount /dev/xenimages/thiess-57-root /xenimages/thiess-57.d/mnt then create it.. =============== Slide 10: Koans =============== Xen will typically run on laptops, but there's currently no support for APM (Advanced Power Management) or ACPI (Advanced Configuration and Power Interface), hence you'll experience reduced battery life and no suspend/resume. We hope to add ACPI support in the future, exploiting Linux's existing support. Xen currently runs only on the x86 architecture, requiring a "P6" or newer processor (that's any Intel or AMD x86 CPU purchased in the last five years). Multiprocessor machines are supported, including basic support for Hyper-Threading (SMT). Support for x86/64 is available since Xen 3.0. In addition a port to the IA64 architecture is approaching completion. We hope to add other architectures such as PPC and ARM in due course. You must disable hyperthreading in you BIOS else your Xen Kernel will reboot indefinitively With an IntelVT enabled system ensure that the IntelVT capabilities are enabled via the BIOS and power-cycle the system. If you do not, on this system, even though the BIOS says VT is enabled Xen will not recognize it as such (that cost me a couple of days =O). ============================================== Slide 11: Acknowledgements and Further Reading ============================================== Special thanks to Jonathon Oxer of Internet Vision Technologies for his support and assitance. Xen Wiki Source: http://wiki.xensource.com/ Xen Users Mailing List: Subscriptions; http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users Archives; http://lists.xensource.com/archives/html/xen-users/ .- ( \ .-. .-. | (-. \/.- ) _..---"""""-. / '. ` '--' `\__.: /dd / / /--' '-.| _.; \ ( -' `--`-.)\ |-..____.-;-. > / / / `--' .' / jgs .'.'_/ `--' `` ``