Titan

From Vd

TITAN is the computing facilities at UiO hosted by USIT by the VD - Research Computing Services group.

TITAN is a powerful computing cluster composed of several hundred computers (number streadily increasing) and several thousand cores (CPUs). TITAN compute nodes typically has 16GB or more memory and 400-900 GB scratch disk space. Hundreds of dual core and quad core nodes combined with super fast Infini Band interconnects and the close to 1 PB [Xanadu] storage capacity capacity, combine to an awesome number cruncher.

1 Hardware
2 Interconnects
3 Storage
4 Queuing system
- 4.1 Checkpoint(/wrap)/migrate/resume
5 Software
6 FAQ
7 Pictures

Hardware

TITAN Hardware consists of nodes and servers.

Illustration of TITAN computing nodes (February 2008)

Interconnects

All TITAN nodes are connected by 1Gbit Ethernet
Around 10 file servers have 10Gbit Ethernet to the core TITAN Ethernet switch
128 SUN X2200s are connected by a 128 port Infiniband switch
All SUN X2200 nodes are being connected to the new SUN Magnum Infiniband switch (2008Q2)
The Xanadu storage facility will become Infiniband connected when the Magnum switch arrives

Storage

See the Xanadu documentation

Queuing system

TITAN has a unique queuing system.

There are a number of TITAN projects, these own from a few to hundreds of nodes (and may own from a few to hundreds of TB of storage in Xanadu).

The TITAN queuing system is such that when a project submit a job, the job will be started - as long as the project has not used all its CPUs. All non-utilized CPUs go into a pool of publically available CPUs (the libero queue) that all participants can use according to a fair share scheme. This way, all projects are guaranteed access to as many nodes as they have purchased and in addition they will have access to more nodes that they actually bought as there will alway be projects that do not utilize their share of TITAN 100%.

Checkpoint(/wrap)/migrate/resume

When a project is running a job in the publically available libero queue, the job may have to be stopped due to fact that a job witch has a right to run on the CPU needs to start. All single CPU jobs run in the libero queue may be check-pointed by BLCR (-ckpt blcr), moved to a free CPU and then resumed.

We strongly recommend users that want to run long jobs (jobs running for a month or more) in the publickly available libero queue to use the upcoming job-wrap utility. The wrap utility is basically a general VMware sand box that ensures that if your job is stopped, it will resume on another node. The perfomance hit using the wrap utility is 10% or more, it is limited to 2GB memory and 1GB scratch.

We are experimenting with moving wrapped jobs out of the TITAN cluster to finish on harvested resources, this is done to expand the size of TITAN further than what is physically possible to fit into our existing macine room.