System information overview
This document provides the basic information new users need
before they can start using this computer. All users should read
all sections in this document.
If you are new to computing at NCAR, you can greatly increase
your ability to work productively here by reading
Overview of computing at NCAR.
The lightning system is an IBM e1350 Linux cluster. It has two
interactive logon nodes and 128 batch computing nodes.
Each e325 node has:
- Two 2.2 GHz AMD Opteron processors
- Two-way set-associative, 64-byte lines, 64 KB of data and 64 KB
of instruction L1 cache in each processor
- 16-way set associative, 64-byte lines, 1 MB of L2 cache in each
processor
- 4 GB of shared ECC RAM memory accessible through dedicated memory
controller on each processor delivering a peak bandwidth of 5.3 GB/s
per processor
Interactive nodes have 8 GB of memory.
Each Opteron chip provides 64-bit integer registers and 64-bit
virtual addressing, which allows 64-bit and 32-bit applications
to run at full hardware speed.
A 128-port Myrinet switch interconnects all the batch nodes through
a single-port Myrinet PCI adapter for message passing. The switch has
~6.3 microseconds of latency and 248 MB/s each way point-to-point
asymptotic bandwidth with full duplex link. Ideally it may reach close
to theoretical peak of 496 MBps for point-to-point both-way traffic
e.g. MPI_Sendrecv function between two tasks in two distinct nodes.
A Cisco Gigabit Ethernet network hosts the TCP/IP traffic including
GPFS, NFS, MSS, etc.
More information about lightning's system architecture appears in
the seminar materials for
Lightning, the IBM e1350 Linux
Cluster (PowerPoint slides).
- Operating System: SUSE Linux Enterprise Server 8
- Compilers: PGI Server suite: pgf90, pgf77, pgcc, pgCC, pghpf
- Profiler: pgprof
- Debugger: pgdbg
- Utilities: gmake, gcc, contact the
CISL consultants for more
information
- Software libraries:
- • ACML
- • netcdf-3.5.1
- • NCAR Graphics
- • LAPACK
- • SPHEREPACK
- • HDF
- Batch subsystem:
LSF
- System information commands: bqueues -l for queue
information; bjobs -u all to check the status of all
users' jobs on lightning; lsfq to see a summary of all
running jobs; /usr/lpp/mmfs/bin/mmlsquota to see your
GPFS disk quota
More information about lightning's software environment appears
in the seminar materials for
lightning compilers
(PowerPoint slides).
Lightning is inside the UCAR supercomputer security perimeter, so
access from outside must be via the gatekeeper system roy.ucar.edu
using a CRYPTOCard. If you do not have an CISL-issued CRYPTOCard,
you can request one by submitting a
Extraview work request or calling 303-497-1200. Instructions
for navigating the UCAR security perimeter are provided in the document "Overview of
computing at NCAR" and the relevant section is
Accessing computing services.
You must use ssh, scp, and sftp on lightning; there are no rsh,
rcp, and ftp commands.
You will need Secure SHell (SSH) on the workstation you use to
access IBM SP-cluster systems. Information about using SSH for
computing at NCAR is described in the document "Overview of
computing at NCAR" and the relevant section is
SSH - Secure SHell.
Once you have authenticated your access at roy.ucar.edu using your
Common Authentication System (CAS) password or your public key and
one-time password from your CRYPTOCard, then you may access lightning
from your proxy prompt by keying in your CAS password (or public key
if you have installed key-based authentication at roy).
The General Parallel File System (GPFS) is installed on
lightning. The user home ($HOME) as well as scratch
(/ptmp) are both GPFS.
The large amounts of data generated by executables should be
written to temporary disk in individual /ptmp directories
(/ptmp/$LOGNAME). Users may generate subdirectories on /ptmp and
expect that /ptmp data will exist for the duration of the user
job or interactive login. Valuable data on /ptmp should be moved
to the MSS or other storage before a batch job or user login
session terminates.
The scratch file system /ptmp is not backed up, and files are subject
to deletion as soon as the batch job completes or the user login
terminates.
Please do not use /tmp in your work; this disk space is quite limited
and reserved for the OS.
Divisional file systems: CISL provides file space for users
that can be used to supplement user home directories and /ptmp space.
Space on these divisional file systems is provided on the basis of
requests made to designated divisional representatives who implement
their own policies regarding space and quota allocations. These
divisional file systems are not backed up by CISL nor are they
scrubbed.
To acquire space on the file system provided for your division,
contact the proper
divisional repsresentative
listed under the password-protected link in this sentence.
CSL users may receive a lightning login as part of their CSL
allocation. See the
CSL announcement of opportunity
for details on submitting proposals.
Community Computing users who have General Accounting Unit (GAU)
allocations are eligible to apply for an account on lightning.
Community users may request a lightning login by contacting
CISL Customer
Support. Please include the following information with your
login request:
- Your login name
- Your project number
- A short description of how you intend to use lightning
New users are placed in the bash shell by default. The csh (tcsh),
ksh, and few other less common shells (see /etc/shells) are also
available.
Basic shell startup files are provided for all users at the time
of account creation. To see the default environment for your login,
type:
printenv
when you are logged in.
If you have questions about the locations of libraries or other
software, contact CISL Consulting Services at
consult1@ucar.edu or
303-497-1249.
If you want to change your shell:
- Log on to lightning.
- Connect to storm using ssh storm.
- Your display shows an interactive menu saying:
30141: Warning: Permanently added 'storm' (RSA) to the list of known hosts.
Directory: /home/lightning/user
Directory: /home/lightning/user
Mon Jan 31 13:34:44 MST 2005
You are only permitted to change your default shell you
use as an interface to the operating system. To change your
PASSWORD; please use the procedure for changing CAS password.
Do you want to change your default shell
Please enter Yes or No.
- After typing "Yes," select the pathname of your desired shell.
Normally, you would specify /bin/bsh (Bourne shell), /bin/csh (C shell),
/bin/ksh (Korn shell), or /bin/tsh (trusted shell). This will
become your default shell on the Linux system.
- The menu terminates your storm session automatically.
- Log off lightning and wait 45 minutes before logging back in.
To change your password:
Your password will be in sync with the UCAR Central Authentication
Services (UCAS) password. Instructions for changing your UCAS password
are provided at
Password guidelines and instructions for
changing passwords on NCAR computers.
Details about job queues and their uses on lightning, and
details about charges for running jobs on lightning are provided in
Queues and charging
for resource usage on lightning.
If you experience an unexplained program failure, you can
seek help from CISL Consulting Services. First, prepare a brief
description of the problem and collect the following information:
- Job number
- Nodes on which the job was running, if known
- Time the problem occurred
- Location of the executable, run script, and error output
Then contact CISL
Consulting Services by email at
consult1@ucar.edu
or by phone at 303-497-1278.
Next page |
Table of contents - Lightning user
guide
If you have questions about this document, please contact
CISL Customer Support.
You can also reach us by telephone 24 hours a day, seven days a week at
303-497-1278.
Additional contact methods:
consult1@ucar.edu
and during
business hours
in NCAR Mesa Lab Suite 39.
© Copyright 2004-2006. University Corporation for Atmospheric
Research (UCAR). All Rights Reserved.
Address of this page:
http://www.cisl.ucar.edu/docs/lightning/overview.jsp
|
