How to compile, build, and run jobs
After a code is written, it has to be compiled to machine language,
linked with libraries, and executed as a command. This section
provides some tips on compiling, building executables with the
loader, and running executables. This section then introduces
basic examples that cover the five principal operating modes for
running jobs on lightning.
For best results, use the 64-bit PGI compilers under
/opt/pgi/linux86-64/5.2/bin/
This should already be specified in your $PATH as part of your
account creation, but you should verify this before you proceed.
These compilers generate 64-bit code by default and use the
appropriate lib64 default pathnames /lib64,
/usr/lib64, etc. If you want to generate 32-bit code,
you can do so by using the compiler flag -tp=k8-32.
We recommend using and building 64-bit memory APIs because there
are definite runtime benefits for doing so. Running in 64-bit mode,
the Opteron has twice as many General Purpose Registers available (16
vs 8) and twice as many 128-bit XMM registers (16 vs 8) available.
This enables the compiler to generate significantly faster code in
64-bit mode than 32-bit mode. Typical performance gains of 8-9% and,
in certain cases, gains of up to 52% result from using 64-bit mode.
An additional benefit is that 64-bit executables get a larger default
stacksize and so avoid some segmentation violation errors caused by
the small default stacksize in Linux.
At this time, CISL provides full support for some (but not all)
compilers on lightning, and of these, only the 64-bit versions.
For the PGI compiler, the most common options are -O,
-fast, -Mfree, and -mp. For byte-swapping
binary data, please use option -byteswapio. Obviously,
for sophisticated applications, you will need to consider many
other options. Please see this web page for PGI compiler
documentation:
http://www.pgroup.com/resources/docs.htm
Nonsupported compilers are provided in the
/contrib directory. CISL Consulting
Services offers only limited support of these other compilers.
The "modules" utility is available for modifying your environment to
find alternate compilers on lightning, such as PathScale.
To get set up to use modules, you will need to add the following lines
to your .cshrc file after all other path-setting commands:
#module setup
if( ! $?SHELL ) then
setenv MODULE_SHELL $SHELL:t
else
setenv MODULE_SHELL tcsh
endif
setenv MODULESROOT /contrib/Modules
setenv MODULESVERSION 3.1.6
setenv MODULESHOME ${MODULESROOT}/${MODULESVERSION}
setenv MODULESPATH ${MODULESROOT}/modulefiles
setenv DEFAULT_MODULES "modules lsf pgi ImageMagick"
set minit = ${MODULESHOME}/init/${MODULE_SHELL}
source $minit
module purge
module load ${DEFAULT_MODULES}
After this setup is executed upon login, you can use the module
command. To see which modules are available, type:
module avail
To see which modules are in force, type
module list
To load a new module (PathScale example), type:
module purge
module load pathScale.2.4-64
For help, type:
module help
This policy describes CISL's level of service commitment for
compilers on Linux clusters.
The compiler path naming convention specifies compiler type,
vendor, version number, and bittedness in the name. Primary compiler suite support
The primary compiler suite on lightning is the 64-bit Portland
Group compiler suite (PGI). CISL's Consulting Services Group and
Supercomputing Systems Group will give top priority to problems
with this compiler suite, and will build and support appropriate
MPICH and NetCDF libraries for this compiler suite. The default compiler version will be the latest stable
64-bit version available, and it will be linked to a path with
a name conforming to the nomenclature in point 1. The previous version of the compiler will be retained for
regression testing purposes in an explicitly labeled path conforming
to nomenclature in point 1. The current and previous version of the compiler will be fully
supported by CISL. Versions older than the previous one will not be
supported, but they will be placed in the /contrib directory.
Full library support
The same support policies and nomenclatures described in
point 2 for compilers apply to the related MPICH MPI, netCDF,
and other similar libraries.
Other compiler suites
All other compiler suites (e.g. Pathscale and Lahey)
and related libraries desired by users will be placed in a
user-accessible and user-maintained area called /contrib. CISL will initially install the unsupported compiler and
libraries that have already been built on lightning in /contrib
using the naming convention in point 1. CISL Consulting Services Group and Supercomputing Systems Group
will make a best effort to investigate and forward problem reports to
compiler vendors when that activity does not interfere with point 2a.
Support for compilers other than that designated as the production
compiler in 2a will be conducted on a time-permitting basis. CSG urges
users to review information available in /usr/local/examples for usage
requirements with special note of library and manpage paths that may
be required to augment the default user environment as well as special
compiler options that might be necessary. Further MPI, netCDF, etc. library builds in support of
Lahey, Pathscale, or other compilers are the responsibility of the
developer building and installing in /contrib. Developers using /contrib are expected to follow the
path nomenclature described in point 1. Periodic upgrades of Lahey and Pathscale compilers in
/contrib will be the responsibility of CISL. /contrib will be a file system mounted across all
Opteron-based Linux cluster nodes.
The compiler can also be used to link object modules into an
executable file. This is especially useful for inter-language program
development. Assume, for instance that source code written in Fortran
90, C, and C++ needs to be linked together. This can be done by
following these steps:
pgf90 -c a.F90
pgcc -c b.c
pgCC -c c.cc
pgf90 -o exe a.o b.o c.o
Caution: The small memory model of the linux86-64 environment
limits the combined area for a user's object or executable to 1 GB,
with the Linux kernel managing usage of the second 1 GB of address for
system routines, shared libraries, stacks, etc. Programs are started
at a fixed address, and the program can use a single instruction to
make most memory references.
The medium memory model:
pgf90 -mcmodel=medium myprog.f
allows data areas larger than 2 GB, or .bss sections. Program
code for the medium memory model must be compiled .fpic, or
position-independent, and will require additional instructions to
reference memory. The effect on performance is a function of the
way the data is allocated in memory by the compiler.
You can run executables interactively at the command line by
invoking the appropriate command. These jobs should not exceed 30
minutes of wallclock time. Longer jobs must be run in batch mode
using LSF as described in
Batch job submittal
with LSF.
The following examples cover the five principal operating modes for
running jobs on lightning. All of the examples use different methods
to calculate results for this equation:
Each example below has a script that contains the code,
compiles the code, configures the runtime environment, and runs
the code. Each example is written in three languages: Fortran,
C, and C++. Important: By studying all of these
examples, you will develop a fundamental understanding of how
to use each programming method on lightning to get results on
your jobs.
Note: These examples show the invocations for each of the
five principal operating modes, and they are not intended
to demonstrate techniques for building a parallel progamming
application. (If you need to understand the implementation differences
between SPMD and MPMD, see the classic hardcopy text Practical
Parallel Programming by Barr E. Bauer, ISBN 0-12-082810-3.)
The example codes are implemented in five different ways as:
- Single process using one CPU and a single thread (serial
job)
- Single process using multiple CPUs on a single node and
multiple threads (threaded job using OpenMP)
- Single process using message passing (MPI) and a single
thread; one code acting on multiple data structures (SPMD)
- Multiple processes using message passing (MPI) and a single
thread; multiple codes acting on multiple data structures (MPMD)
- Single or multiple processes using message passing (MPI) and
multiple threads (OpenMP): Hybrid
This example runs a single process on one CPU to sum a set of
1,000,000 exponentials. This
serial job example
is implemented in three programming languages: Fortran, C, and C++.
This example runs a single process that spawns a thread on
both CPUs on a single node. This is called an OpenMP threaded job
because it uses threads that comply with the OpenMP standard.
OpenMP is a portable, scalable Application Program Interface
(API) that gives shared-memory parallel programmers a simple and
flexible interface for developing parallel applications for all
architectures.
This
threaded job example
uses threads to sum a set of 1,000,000 exponentials.
This example runs multiple processes using message passing that
conforms to the MPI standard, and each process uses a single thread.
This is one code acting on multiple data structures (SPMD).
This
SPMD parallel job example
uses MPI SPMD (Message Passing Interface-standard Single Program Multiple
Data structures) to sum a set of 1,000,000 exponentials. The Message
Passing Interface is a standardized programming paradigm for parallel
computers designed to allow separate processes to efficiently
communicate with each other via messages.
This example runs multiple processes using message passing that
conforms to the MPI standard, and each process uses a single thread.
This is multiple codes acting on multiple data structures (MPMD).
This
MPMD parallel job example
uses MPI MPMD (Message Passing Interface-standard Multiple Programs
Multiple Data structures) to sum a set of 1,000,000 exponentials. The
Message Passing Interface is a standardized programming paradigm for
parallel computers designed to allow separate processes to efficiently
communicate with each other via messages.
This example runs multiple processes using message passing that
conforms to the MPI standard, and since one or more processes use
more than one thread, it is called a hybrid code. The threads in
the example conform to the OpenMP standard.
This
hybrid parallel job
example uses both SPMD parallel processes and threads to sum
a set of 1,000,000 exponentials. A "hybrid" code combines two types
of parallel computing tasks: SPMD (Single Program Multiple Data
structures) or MPMD (Multiple Programs Multiple Data structures)
parallel processes where one or more processes use threads.
Next page |
Table of contents - Lightning user
guide
If you have questions about this document, please contact
CISL Customer Support.
You can also reach us by telephone 24 hours a day, seven days a week at
303-497-1278.
Additional contact methods:
consult1@ucar.edu
and during
business hours
in NCAR Mesa Lab Suite 39.
© Copyright 2004-2006. University Corporation for Atmospheric
Research (UCAR). All Rights Reserved.
Address of this page:
http://www.cisl.ucar.edu/docs/lightning/cbr.jsp
|
