High Performance Computing

August 28, 2008

The University of Kentucky has replaced its HP Superdome and XC clusters with a cluster of IBM Intel Blade Centers and pSeries SMP nodes. In contrast to the former cluster of The new "supercomputer" consists of 340 nodes each containing 4 64-bit x86 processors, and 8 nodes each containing 16 64-bit pSeries processors. Transfer to the new system will entail certain changes that will affect all users of UK's supercomputing facility.

The purpose of this document is to assist current users in their transition from previous systems to UK's new IBM BladeCenter/pSeries complex.

• 03/30/07 : Per previous announcements, the SDX/XC systems are no longer available. Old SDX accounts and files have been migrated to the new IBM system ( BCX ).
  
  
• The IBM cluster login nodes can be accessed at: bcx.uky.edu
• In order to authenticate (login) to the new IBM Intel commodity cluster (bcx.uky.edu) you must use your U-Connect (aka e-Dir) password. Note: your HPC and U-Connect account name must match ! If they don't, contact the Help address below.
  Once you login the first time, you may set up an ssh public key to authenticate from then on if you desire (or continue to use your U-Connect passwd). Please change your default password with passwd.
• The BCX (commodity cluster) nodes have 8GB of memory (each); ie 2GB per core. If your memory requirements for non-distributed code exceeds that, you will have to port your code to the P-series (PowerPC) SMP machines.
• To see important system related notices, use the sysstatus command when on the system.
• General job examples, see: /share/cluster/examples .
BCX - Intel commodity cluster
• To compile Fortran (F90), use ifort (Intel Fortran).
• To compile for OpenMPI use mpif90,mpicc
• BLAS and LAPACK functions are available via the Intel MKL libraries.
• To link with Intel MKL, use -L$(INTEL_MKL_LIBS) in your makefile along with: -lmkl
• To link LAPACK, link MKL above along with: -lmkl_lapack
• Intel MKL docs here: /share/cluster/SLES9/x86_64/apps/intel/mkl_docs.d
P-series - large SMP
• ssh from BCX via plogin
no password required.
• To link LAPACK, see: hpcadmin:/share/cluster/examples/p-series/lapack



• Note: LSF is no longer the batch software: replaced by Moab+SLURM ; see the examples above.
• bjobs (LSF command) is not currently available; use showq or squeue to check jobs.
• To kill a job:
1. bkill job_id_from_showq
or
2. squeue (to get job_id)
3. scancel job_id
• The HP VED editor is no longer available; the closest replacement we have at this time is NEdit.
• To currently access the Gambit grid generator, use "mygambit".