Harvesting
From Vd
Contents |
Dione
USIT increases its effort to offer cost efficient IT solutions to the university academia by offering harvested CPU cycles to user communities requiring computational calculations to produce scientific results. The CPU power will be gathered from UiO campus computers by use of sand box technology to guarantee a non-intrusive use of the campus computers. This project is CPU Harvesting.
The harvesting approach, code named Dione, include:
Dione - a virtual TITAN compute cluster
The Dione project is built on standard "open" technology. It is basically a ROCKS cluster with virtual compute nodes. The virtual nodes is implemented using VMware as virtual machine (VM) layer on both Windows and Linux (machine) and in each virtual machine a ROCKS based Linux OS (node) is the framework for executables. NFSv4 is used as network file system between Dione file servers and virtual nodes. As IP-addresses of a virtual compute node (node) may change during job execution, both the file system and the queuing system will need to be able to adapt to changes in node names and IP-addresses.
- In this text we use the following notation
- Machine: The physical machine
- VM: The Virtual Machine / VMware
- Node: The operating system instance running on top of the Virtual Machine / VMware
Dione Servers
Dione is composed of the following:
- ROCKS Server
-
- SGE queueing system
- Ganglia - monitoring system
- ROCKS OS-images
- File Server
-
- Serves home directories
- Stores Virtual Machine (VM) snap shots
- Uses NFSv4 software
- Login Server
-
- Interactive login and job submission
- Checkpoint Server
-
- Stores Virtual Machine (VM) snap shots
- Uses NFSv4 software
Network
- Dione nodes
- The number of Dione nodes is adjusted by the need for compute resources. A default number of idle and ready nodes could be between 10-100 nodes, depending on typical usage pattern and the size of the pool of available machines. The life of a Dione node is typically:
- Client startup
- A free IP-address is requested from the IP-pool
- The IP, name etc is inserted into ROCKS/queuing system
- WMware image files are prepared and copied to the idle machine
- VMware is started on a idle machine and the Linux OS inside the VM takes on the host name of the readied IP
- The newly created node receives a job
- Snapshot
- Based on statistics/best guess Dione may order a snapshot of a node
- VMware on the node is suspended and copied to the Dione checkpoint server, VMware is then resumed
- The VM might be killed if the machine goes into a active state
- The VM-snapshot will be matched against other idle machine resources, and if a match is found:
- The VM-snapshot image will be copied to the idle machine
- The ROCKS database and queuing system is updated
- The VM is resumed on the new machine
- Job completion
- When a job has completed, results are copied to the user home directory
- The VM is suspended or is kept running depending on the job queue situation
- A ROCKS server traditionally has one leg on an open network and one on a private network. In Dione, nodes may exist on a number of networks
File system
- The Dione distributed file system for home directories is based on NFSv4. This is due to the fact that it is a NFSv4 strong candidate to become a widely used network file system in the not so distant future.
- For VM-image staging, we use NFSv4 (Windows NFS4 client).
Dione Compute nodes
Dione compute nodes reside inside a sand box. The sand box (VM) runs on a physical Windows or Linux machine. At this stage, the sand box will be a VMware virtual machine (VM), more specific Dione will use VMware Player to host the ROCKS/CentOS based nodes in the cluster. The nodes have an environment and a queuing system client suitable to run compute intensive applications requiring 1-2GB memory and 1 GB scratch disk space.
The larger the job and works space, the more strain Dione will put on network and machine resources.
Activity schedule
There will be several PILOTs:
PILOT 1 - ROCS in a box
Set up a test ROCKS cluster "in a box": Set up a ROCKS server and 5 ROCKS compute nodes as virtual machines sharing a virtual network inside a SMP-machine (a 16 core SUN X4600 with 32GB memory was used).
- Tests
- Duplication of VM images with generated MAC addresses
- Shrinking of VM images
- Running of test-jobs
PILOT 2 - Dione @ VD
Set up a Dione cluster with compute nodes in virtual machine running on USIT HPC group work stations and on HPC servers in the USIT machine room. A dedicated server is set up on the 129.240.15 net.
- Tests
- Set up NFS4 server
- Test jobs (performance)
- Start and stop VMs done
- Start and stop nodes
- Suspend VM, checkpoint (copy) VM, move VM to another host, resume VM without changing IP
- Suspend VM, checkpoint (copy) VM, move VM to another host, resume VM with changed IP
- Test that NFS-handles survive suspend-resume of VMs
- Test suspend-migrate-resume on Windows to Windows, UNIX to UNIX, Windows to UNIX and UNIX to Windows
- Test different schemes to suspend and copy images to a central/distributed repository
- Test install ATLAS at the file server, enabling running on ATLAS-jobs on the virtual nodes (under the motto: 'if it can run ATLAS-jobs it can run anything')
PILOT 3 - Dione @ USIT
Set up a Dione server in a suitable spot in the university network. Expand PILOT 2. It is time to include machines from a larger part of USIT: Some months ago, there was bought around 25 desktop computers with 2 cores (CPUs) and 4GB of memory, most these nodes will be ideal as compute nodes in the Dione cluster. The 4GB machines will have to be identified and the users notified before PILOT 3 is started.
- Tests
- Several scaling tests:
- Test network/fileserver impact when suspending and copying all images to central repository
- Test network/fileserver impact when all nodes start a large job at the same time (ATLAS jobs typically do this from time to time)
- Etc
- "Owner feedback" activity is required to gen an idea about how the equipment owner react to the external activity
- Connect the Dione SGE queuing system together with the TITAN SGE queuing system, enabling certain one-CPU jobs to float from TITAN into the Dione space
- Continue to optimize for the running of ATLAS-jobs. ATLAS-jobs allready run quite well uner Condor, but are quite demanding one-CPU jobs. The running of ATLAS-production jobs is something we are committed to be able to do for many years, and it would be good if the jobs could be run on non-dedicated resources, freeing up compute nodes to parallel tasks.
Status
PILOT 1 - ROCKS in a box
- Status
- finished
- Results
-
- Easy to set up
- Size counts: ROCKS images are too large and has to be shrinked before they can be moved around in the network
- The VMware images tended to suspend for shroter or longer time intervals, annoying, but unknown if this is a VMware problem or an issue cased by a combination of much memory and many cores - should be further investigated
- Dione status is available on the Web
- Physical machines with Xen enabled Linux OS may crash (at least the SUN X4600 did), we should probably avoid machines like this
PILOT 2 - Dione @ VD
- Status
- planned/started
- Results
PILOT 3 - Dione @ USIT
- Status
- planned
- Results