Unified Queuing System

The primary purpose of the Unified Queuing System project is to establish a consistent batch environment for all systems serving as computational engines within the NCAR environment. In addition, the Unified Queuing System project intends to establish a flexible, priority-based job queuing and scheduling structure that can be easily adapted to the evolving needs of the NCAR user communities.

In recent years, NQS has become a standard product supplied by high-end computer vendors for managing batch jobs. Because of this, SCD believes that as new computational systems are installed and old systems are decommissioned, NQS will provide a common subsystem for maintaining some form of continuity in the user environment.

In addition, SCD has developed and implemented a layer of software named BPS (Batch Priority Scheduler) during FY96-97 to provide enhanced job scheduling capabilities that have not been available in vendor-released NQS systems. BPS provides a number of features that include priority-based pre-emptive job load management, time-based job queue scheduling, and the ability to schedule jobs in a round-robin fashion within the priority-ordered queues. Round-robin scheduling uses NCAR-defined projects, or proposals, to fairly distribute limited computational resources. BPS also gives SCD the ability to respond to special scheduling needs; for example, accommodating jobs with unusually large resource requirements that might not be able to run effectively in a normal job mix.

The Unified Queuing System project is an effort to unify the batch processing user interface across all of the computational platforms at NCAR with a common set of queues and job submission rules for all users. This project uses the vendor-supplied NQS system, front-ended by the SCD-developed BPS system, to meet the unique, demanding, and evolving computational requirements of the NCAR user community.

Standardized queues

A common set of queues has been established to which users will submit jobs, regardless of the machine on which the job is to be run. This set of queues is priority based; the BPS and NQS subsystems will schedule system resources based on queue priority. There will be four basic queues: prem ("premium"), reg ("regular"), econ ("economy"), and sb ("standby"). The regular queue is meant to be the primary target of most work, and should provide timely job turnaround. The other queues are then available for users to selectively discriminate the priority, or urgency, of their work.

In addition to the four basic queues, two special queues ("spec" and "nd") have been established: "spec" is for special projects and "nd" is a near-dedicated queue used in a similar fashion to the "nd" queue on the old Y-MP/8 (shavano). The ability of any user to submit work to the "spec" and/or "nd" queues is enabled and disabled based on policy decisions and special, refereed requirements. Limits for these special queues will be relatively unrestrictive, thus allowing a single application access to the full resources of a given system.

Premium queue jobs receive top-priority scheduling and pre-empt lower-priority work, if necessary, so that sufficient system resources are made available. Economy queue jobs are allowed to run when regular and higher-priority work does not require pre-emptive access to system resources. Standby queue jobs receive system resources only when little or no other higher-priority work exists.

The Supercomputing Systems Group plans to implement the unified queue structure on the SGI Power Challenge XL (winterpark) and the HP SPP-2000 (sioux) during FY98.

| Next page | Top of this section | Table of contents |