bluevista links
Bluevista welcome for bluesky users
Change your bluevista password
SCD links
Visualization and Enabling Technologies
bluevista main page
|
|
Overview
Bluevista is an IBM clustered Symmetric MultiProcessing (SMP) system based on the POWER5 processor. Users familiar with the POWER4 system bluesky will find that running jobs on bluevista is very similar to running on bluesky. (See comparison below.) There are several POWER5 performance improvements over POWER4, and the High-Performance Switch is also improved in several ways.
The purpose of bluevista is to provide a high-performance, scalable, parallel, production platform for NCAR Community Computing and Climate Simulation Lab (CSL) programmers and scientists to run their numerically intensive jobs.
Hardware
- Processors: 624 POWER5 processors with a 1.9-GHz clock cycle; each can perform four floating-point operations per cycle. Peak of 4.74 TFLOPs. Bluevista utilizes the single-core POWER5 chip.
- The processors are grouped eight per node, referred to as an 8-way node. There are 78 8-way nodes dedicated as follows:
- 72 nodes reserved for batch workload
- 2 nodes reserved for user login and interactive workload
- 4 nodes reserved for I/O and MSS connectivity
- Memory: 2 GB memory per processor; 16 GB shared memory on a node; 1,248 GB memory total.
- Memory caches:
- Level 1 cache: 32 KB data; 64 KB instructions; two-way associative
- Level 2 cache: 15.2 MB L2 cache per node
- Level 3 cache: 288 MB L3 cache per node
- RAID disk storage capacity: 55.0 TBytes. By default, each bluevista user will have 3 GB of space in their home directory and 240 GB of space in /ptmp. Scrubbing of old data in /ptmp begins when /ptmp usage reaches 85% and continues until usage drops to 70%. This /ptmp scrubbing policy is identical to that of bluesky.
- High Performance Switch (also known as "interconnect fabric"): The IBM pSeries High Performance Switch (HPS), previously known as the Federation switch, provides a single-link, unidirectional, point-to-point communication bandwidth of 1700 MB per second and latency of 5 microseconds. Each node of the system has a bidirectional, 2-link interface to the HPS.
- Connectivity to the Mass Storage System: There is Gigabit Ethernet connectivity to the MSS Storage Manager. The MSS Storage Manager writes files to the MSS disk cache and to MSS cartridges over fiberchannel and fiberchannel tape drives, respectively.
- Login connectivity: A Gigabit Ethernet network provides login connectivity. Two of the 8-way nodes are reserved for interactive usage.
- Security considerations: bluevista resides within the SCD security perimeter and can only be accessed via a CRYPTOCard.
Software
Note: Except for the batch system, bluevista software is identical with bluesky software, although versions may differ and there may be some difference in product names due to the difference in switches.
- Operating System: AIX (IBM-proprietary UNIX)
- Batch system: Load Sharing Facility (LSF)
- Compilers: Fortran (95/90/77), C, C++
(Note: The compilers will produce 64-bit APIs. To produce 32-bit APIs, set environment variable OBJECT_MODE to 32.) - Utilities: These include pmrinfo, spinfo, batchview, and mssview. Please refer to /bin and /usr/local/bin on bluevista for a more complete list of user utilities.
- Software libraries: These include IBM's parallel libraries for OpenMP and MPI usage. Users may also request single-threaded libraries maintained at NCAR, including Spherepack and Mudpack. SCD prefers that users download the source code for these libraries and install them for their own use.
- Debugger: TotalView.
- File System: General Parallel File System (GPFS), a UNIX-style file system that allows applications on multiple nodes to share file data. GPFS supports very large file systems and stripes data across multiple disks for higher performance.
- System information commands: spinfo for general information; lslpp for information about libraries; batchview for batch jobs; bjall for more detailed information on batch jobs.
Who can use this system / Job scheduling
Scheduling of batch jobs will be via the Load Sharing Facility (LSF) batch system rather than the LoadLeveler system that runs on bluesky. Please see the documentation section below for pointers to LSF documentation.
If you are a present user of CISL/SCD supercomputer resources and you do not have a bluesky account but want a bluevista account, please request this via the web form at SCD Customer Support.
How to use this system
Parallel programming on bluevista is done with OpenMP, MPI, and a mixture of both (hybrid).
- To use more than one processor on a node, use OpenMP threading directives on the node, or use MPI processes on the node, or use a mixture of both.
- To pass information between nodes, you must use MPI.
- To take full advantage of parallelism: use OpenMP threads, MPI, or a mixture of both on the node plus use MPI between nodes.
How to get an account
All users will receive a bluevista login if they have a bluesky login and have logged in to bluesky in the six months preceding December 31, 2005. This applies to CSL and Community Computing users.
Community Computing users who have General Accounting Unit (GAU) allocations are eligible to apply for an account on bluevista. Community users may request a bluevista login by contacting SCD Customer Support at https://remedy.ucar.edu/troublet/index.html. Please include the following information with your login request:
- Your login name
- Your project number
Queues and charging
The class (queue) structure for bluevista is described in the document Queues and charging for resource usage on bluevista.
Examples
Please see the directory /usr/local/examples on bluevista for examples of commonly used batch and interactive jobs.
Documentation
LSF for Bluevista Users, an introductory PowerPoint presentation for new users.
Platform Computing provides High Performance Computing (HPC) documentation for their Load Sharing Facility (LSF) batch job subsystem. To access this LSF HPC documentation, you need access instructions which are inside the UCAR security perimeter.
The following pdf file offers a presentation about Simultaneous Multi-Threading: AIX 5.3 and XL Fortran 10.1 upgrade on bluevista.
POWER5 Processor and System Evolution by Charles Grassl, IBM.
Comparison of bluevista and bluesky
| Resource | Bluevista, POWER5 | Bluesky, POWER4 |
|---|---|---|
| Clock cycle | 1.9 GHz | 1.3 GHz |
| Memory/processor | 2 GB | 2 GB |
| L1 cache | 32 KB data, 64 KB instructions; two-way associative | 32 KB data, 64 KB instructions; two-way associative |
| L2 cache | 1.92 MB per processor pair | 1.44 MB per processor pair |
| L3 cache | 36 MB per processor pair, shared by processor pair | 32 MB per processor pair, shared by all processors |
| Switch latency | 5 microseconds (HPS) | 15 microseconds (Colony) |
| Switch bandwidth | 1700 MB/sec | 350 MB/sec |
| Multiple functional units | 2 floating point units; 3 fixed point units; 2 load/store units | 2 floating point units; 3 fixed point units; 2 load/store units |
| Memory bandwidth | Typical: 4 GB/sec small page, 8 GB/sec large page. See Grassl, 5/2005 for qualifications to this statement. | Typical: 2 GB/sec small page, 4 GB/sec large page. See Grassl, 5/2005 for qualifications to this statement. |
| Simultaneous Multi-Threading (SMT) | Yes. SMT appears to the OS as multiple CPUs. Threaded applications may take advantage of this. | No. |
| Batch job subsystem | Load Sharing Facility (LSF) | LoadLeveler |
| XL Fortran options | Bluevista and bluesky compiler options are the same when the compiler versions match. Produces 64-bit APIs by default. | Bluevista and bluesky compiler options are the same when the compiler versions match. Produces 32-bit APIs by default. |
