kali's webpage at http://www.math.umbc.edu/kali.
kaliand software installed on
kali, in addition to the information on
kali's main webpage. Moreover, this document explains how to compile code for parallel computing using MPI on
kali. Additionally, it explains how to get set up to use MPI and collects some additional information that might be useful upon your first login. While you generally need to read this information only once, you may want to come back to it if you see unexplained performance of
kalithat this information might explain.
If you find mistakes on this page or have suggestions, please contact me.
kali.cl.math.umbc.eduor for short
kali) (a combined management and user node) and 32 computational node; the latter are subdivided into 1 storage node (
storage1), and 31 compute nodes (
node31). All machines have two Intel Xeon 2.0 GHz CPUs with 512 kB cache. The compute nodes have 1 GB of memory, while the head and storage nodes have 4 GB each. The 32 computional nodes are connected via a high-performance Myrinet interconnect, thus making 64 processors available for computations.
Only the head node can be accessed from the outside of the machine; it communicates to all other nodes via the cluster network comprising a 100 Mbps ethernet network. The main file storage is on a RAID array on the storage node, which is NFS-mounted to the head node and all other nodes via the cluster network. This RAID array consists of 8 SCSI harddrives with 1 hot swap and striped check-sums as backup; this setup allows for the failure of up to 2 disks without having to retrieve data from backup tapes.
This network schematic illustrates visually the network connectivities of the Myrinet and ethernet networks, the connection to the outside world, and the placment of the RAID array; notice how the connection speeds are marked, pointing out that the ethernet switch is a special one with two faster 1 Gbps ports to prevent the two most-often used connections, to the management and storage nodes, from becoming bottlenecks. A third network, the management network, is not shown.
Finally, after the abstract information given so far, the rack schematic shows how the components discussed are arranged in the two racks. The labels H5 and H6 denote the placement of the racks on the floor of the operations room of UMBC's Office of Information Technology, where the machine is housed. This schematic might be useful to know, when looking at the "Physical View" in Ganglia. (Notice that the nodes have been renamed from the form node0XY with three numerical digits to the form nodeXY with two numerical digits during the operating system upgrade in March 2005.)
Some photographs of kali:
storage1constitutes the main user disk space on
kali. It is nominally a 0.5 TB array, but due to redundancies for data safety and a spool partition for the tape backup system, about 367 GB are the usable space.
The RAID array is divided into two partitions,
one for users' home directories (
/home, 100 GB),
which is backed up onto tape,
and a scratch partition (
/scratch, 267 GB) that is never backed up.
Users should do their calculations in the
and copy relevant results to the
home directory (or other computers than
kali) for safe-keeping.
There is a quota of 1 GB that applies to your part of the
/scratch partition is accessed by changing
is your username on
For your convenience, a
symbolic link called
scratch in your home directory
links to this directory directly;
it appears and is used just as if it were a real directory, i.e.,
you can say cd scratch in your home directory
to get to
kali.cl.math.umbc.edu, but from the outside, it appears as
kali.math.umbc.edu. To log in to the system, you must use a secure shell like
ssh. Therefore, you can use ssh kali from inside the department (with identical username) or more generally ssh -l username kali.math.umbc.edu or equivalently
ssh firstname.lastname@example.org anywhere on the internet, where username denotes your username on
kali. Your username and password will be the same as on other campus systems such as the
glsystem. See man ssh for more general information.
When you connect to a machine for the first time with
your own machine should warn you about this fact and ask for
confirmation that you want to connect.
If you use
ssh routinely, you will be used to this.
I wanted to point out that, additionally,
kali will perform some setup
that allows you to connect to the nodes of
without typing in passwords. You will see lines such as
generating ssh file /home/gobbert/.ssh/id_dsa ... Generating public/private dsa key pair.in this process.
Let me explain the use of scp by the following example:
username has a file
math627/hw1 from his home directory
kali. To copy the file to the current
directory on another Unix/Linux system with scp, use
scp email@example.com:math627/hw1/hello.c .Notice carefully the period "." at the end of the above sample command; it signifies that you want the file copied to your current directory (without changing the name of the file).
As with ssh, you can leave out the username@,
if your username agrees on both machines. If issuing the command
from within the Mathematics domain at UMBC, you can also abbreviate the
machine name to
See man scp for more information.
To specify the programming environment completely, you still
have to select which compiler to use; you must then use the
MPI installation compiled against the selected compiler.
Two suites of compilers are available on
one from Intel and one from the GNU.
Specifically, the Intel compilers are
icc for C/C++ and
ifort for Fortran 90/95.
And the GNU compilers are
gcc for C/C++ and
g77 for Fortran 77.
The corresponding MPI implementations are
/usr/opt/mpich-126.96.36.199-ch_gm-icc directory tree
for the Intel compilers and
for the GNU compilers.
Notice that you should always compile your MPI code using
the MPI scripts for your language such as
mpicc for C;
see below for more information.
The names of the serial compilers above is only provided
as background information.
As a matter of background, I mention that for successful
compiling, linking, and running, several things have to come
together correctly: Your path has to include the
location of the correct compile script, the MPI header files
have to be found, and the correct MPI libraries have to be found.
This is all conveniently set up correctly for you by the
switcher utility. See its man page for
more information (
man switcher), but the most
needed commands are these:
switcher mpi --listThis should list
mpich-ch_gm-gcc-188.8.131.52. The names involving
gccindicate the compilation against the Intel and GNU compilers, respectively. Additionally, you notice that we have now the LAM implementation of MPI available as well, but I have not tested it myself; if someone does test it, please let me know!
switcher mpi --showThe GNU compiler suite is the default, so upon your first login, this should show
mpich-ch_gm-gcc-184.108.40.206. If you are satisfied with this default choice, you do not need to do anything else.
switcher mpi = mpich-ch_gm-icc-220.127.116.11Notice furthermore that this selection remains in effect across sessions, so you will only need to do this once during your first login to
kali(unless you wish to change your selection later again). Notice carefully that this selection becomes "effective for future shells", as it correct states (no matter what the
--showoption might tell you); so, you either have to log out and log in again before attempting to compile and run MPI code, or you can run the script
switcher-reloaddesigned exactly for this purpose; see the man page for
switcherfor this information.
sample.cthat contains some MPI commands. The compilation and linking of your code should work just like on other Linux clusters using the mpich implementation of MPI. Hence, compile and link the code both in one step by
mpicc -o sample sample.cThe script mpicc works on the surface just like any regular compiler. For instance, the option -o sample chooses the name of the output file (here the executable file), and mpicc compiles and links in (seemingly) one step.
If your code includes mathematical functions (like
cos, etc.), you need to link to the mathematics library
libm.so. This is done, just like for serial compiling,
by adding -lm to the end of your combined compile and link command,
mpicc -o sample sample.c -lmIn a similar fashion, other libraries can be linked.
More formally, I could also separate the compile from the link step.
That is, the C file
sample.c is first compiled into object code
sample.o (default output name, if the option
is used), which then gets linked to the required libraries
to obtain the executable
The sequence of the two commands
mpicc -c sample.c mpicc -o sample sample.o -lmaccomplishes this, where the option
-cstands for "compile-only".
See the man page of mpicc for more information by saying
If you want to compile C++ or Fortran programs, the logic of the commands is the same as above. Simply replace mpicc by mpiCC, mpif77, or mpif90. See their man pages for additional information.
Notice that the MPI compile scripts mpicc, mpiCC, mpif77, and mpif90 have the same name for each of the MPI implementations available. Which one is used depends on which implementation you have selected with switcher. Notice also from their man pages that any option that is not recognized as a MPI option is passed through to the underlying compiler; so you might want to know the name of the underlying compiler to be able to pull up its man page. Caution: As you saw from the above list, GNU does not have a Fortran 90 compiler, so it is not clear what mpif90 might do when using the GNU MPI implementation; let me know if you find anything out about this and other issues, so I can post more details here.
Finally, to be doubly sure which compiler is accessed by your MPI compile script, you can use the -show option as in
mpicc -o sample sample.c -showThis option causes mpicc to show what compile commands it would issue without actually issuing them; then notice at the beginning of every line what compiler you see.