![]() |
|||||||||||
|
|
|||||||||||
![]() |
Visualization and enabling technologiesThe overall thrust of the Visualization and Enabling Technologies Section (VETS) is to provide the software, facilities, and human expertise to support a broad range of analysis and visualization capabilities while extending best practices, extending domain knowledge through basic research, and using results effectively for outreach. VETS made a number of major advances during FY2001. We completed the creation of a next-generation Visualization Lab that included a sophisticated new physical space, integrated visual supercomputing, and collaboration technology in the form of an Access Grid node. NCL capabilities were dramatically expanded, new experimental software work was undertaken, and the Community Data Portal was populated successfully with a number of pilot projects. We succeeded in securing a leadership role in the DOE-sponsored Earth System Grid project, moved our NSF-funded VGEE project toward completion, and played a major role in the development of the Knowledge Environment for the Geosciences (KEG) concept and proposal to NSF's ITR program. We contributed to quite a number of other proposals as well. We also began work on NCAR Strategic Initiatives in the area of web-based data provision and next-generation web environments. VETS staff engaged in a broad portfolio of collaborative interactions with people and projects at NCAR and other organizations. We also continued our tradition of maintaining a high level of visibility at conferences and in the public press and media. FY2001 will serve as a foundation for an exciting FY2002 program that will focus on advanced collaboration environments, sophisticated data and media portals, terascale analysis and visualization tools, and a new generation of web infrastructure and applications. This document briefly summarizes our progress for the year. Facilities for analysis, visualization, and collaborationSCD has operated its highly successful Visualization Lab for several years now, and it has proven to be a valuable tool for enabling new understanding of large-scale scientific data. It has proven equally useful as an important and popular window into our research activities at NCAR and UCAR. During FY2000, SCD developed plans for a next-generation facility aimed at collaborative group visualization and analysis with major enhancements in capability, capacity, and accommodation of larger groups. A new physical space was identified, pre-visualized, and designed. In FY2001 we completed the design specifications and proceeded to begin building the new facility. It was a large, complex project that took most of the year and had a number of phases and components. These included the remodeling of the physical space, moving and integration of primary computational systems, building the networking infrastructure (wired and wireless), managing a contract for control and room integration by an outside contractor, integrating a new generation of high-resolution 3D projection systems, and last but not least, developing an entirely new suite of software to drive the room's capabilities. We also upgraded the lab's storage array to a full RAID configuration and added some high-performance storage dedicated for our frequent demonstrations. The new room sets a mark for scientific workspaces. It is a pleasant space, provisioned with comfortable seating, powered and networked modular furniture supplemented with wireless networking. We can comfortably seat about 35 people for a boardroom-style meeting and a few more for a theater-style presentation.
The facility is a completely "fly-by-wire" system: the computers, lights, and audio systems are all operated by a dedicated touchpad display or a web interface via a browser. Based on our early experiences, we have begun the process of making some incremental improvements to the new lab. In the upcoming year we'll be adding additional audio support for the AccessGrid, a new release of the electronic control system, and an expanded video-switching matrix that will allow more systems and better sharing of resources with offices. Community software for data analysis and visualizationAs part of SCD's FY2000 effort to develop roadmaps for our future, Open Source software was identified as a strategic element of the overall plan. At this writing, VETS manages approximately 1.5 million lines of analysis and visualization application code and conducts portability testing for most of the major Unix variants including Linux with Windows and Mac OSX on the horizon. Virtually all of the software has either been moved under a GPL source code license or a freely available binary license with the intent of having all applications available under a GPL license in FY2003. Open Source is an important step for our research community because they have specialized software that is well adapted to our domain and that may be shared and extended for new purposes. It's an important step for us because we can extend our application software offerings with developments from the growing mass of other developers who have also adopted an Open Source path. Our software development and sharing efforts are detailed below. NCAR GraphicsUsed by thousands of researchers, NCAR Graphics is a mainstay in the geosciences community. The package provides software libraries that support a wide variety of graphing and visualization functions with extensive and robust mapping capabilities. NCAR Graphics delivers superb, publication-quality visuals that are as good or better than anything available from any toolset, including commercial tools. In FY2000, SCD released NCAR Graphics as an Open Source distribution under the GPL license. During FY2001 we saw nearly 9,000 downloads of the software, a remarkably large number for a "legacy" software package. It still appears to be popular and useful in the community. While we no longer actively develop or promote NCAR Graphics, we continue to supply a quality distribution, provide bug fixes, and add libraries as needed to support new functionality in NCL. The NCAR Command Language (NCL)The NCAR Command Language (NCL) is a scripting language for computation and visualization. While designed as a general-purpose language, SCD has worked extensively with CGD and others to equip and position NCL as the analysis environment of choice for the Community Climate System Model (CCSM). NCL handles netCDF, HDF, ASCII, binary, and GRiB files (including ECMWF coefficients) and can natively access CCM "history tapes." Layered interfaces tuned for climate analysis coupled with nearly 500 additional mathematical and graphical functions make NCL a formidable tool for the researcher. SCD supplies NCL to the broad science and educational community and provides ongoing support and development for this 1.5-million-line software package. In FY2001, NCL was released for free to the public as a binary distribution along with new documentation, an FAQ, and new web-based support and distribution functions. We saw roughly 1,500 downloads of NCL over the web. Over 60 new data-processing functions were added to NCL, including ones that do the following:
Several plotting functions and resources were updated to add more options for customizing graphics, including the generation of curly vectors, having more control over paneling multiple plots, and creating specialized XY plots. New functions were created for generating streamline and vector plots in pressure/height formats or over various map projections, and for creating histograms. Some utility functions were created for manipulating color maps and annotating plots. Work began on integrating the RANGS/GSHHS database, a map database of very-high-resolution coastlines, and a new release including this important new functionality is planned for FY2002. Work also began on porting NCL to other environments, including Windows and Mac OSX. Our FY2002 plans call for the provision of comprehensive support for all components of CCSM-2, much better capabilities for weather models (e.g. WRF, MM5), and a modest expansion of our staff resources for day-to-day porting, testing, and distribution. From a technical standpoint, this translates to completing interfaces for high-resolution map databases (particularly useful for oceanography), adding support for generalized 2D coordinate systems (e.g. curvilinear), and re-engineering higher-level software to expose all of this to users. Several examples of NCL visualizations are shown here: NCAR version of Vis5DVis5D is a mainstay visualization tool in the scientific community and has proven to be a versatile application for the mesoscale community as well as for climate, turbulence, and ecosystems research. VETS has created significant extensions to the package, including provisions for use in a virtual environment, adaptations for visualizing very large datasets, a particle system feature, and interfaces for creating VRML (Virtual Reality Modeling Language) output. The enhanced application has been shared with the broad community and is used at universities, NOAA, NASA, other NSF centers, and the Naval Research Lab. Like the parent version of Vis5D, this software is extended and shared under the Gnu Public License (GPL). During FY2001, we integrated another software toolkit into Vis5D called "qslim," a very good surface decimation package. This allows the geometric complexity of objects to be optimized such that they're a tenth the size of the original, and correspondingly faster to render. This is one useful approach to dealing with the increase in dataset resolution in an era when high-end hardware advances have stagnated. During FY2001, the Vis5D project became an Open Source community effort called Vis5D+, and we happily joined in, offering up our version of the software. We began the process of merging our technology with the public version, and we intend to continue and advance this process in FY2002. GvolshGvolsh is a direct volume-rendering tool based on the shear-warp factorization approach that can effectively utilize multiple processors and deal with very large regularly gridded datasets. We continued to use gvolsh as an environment for experimenting with new rendering approaches, and we also developed a set of utilities that permit tightly coupled interaction between IDL and gvolsh. These utilities enable a user to seamlessly move back and forth between the IDL and gvolsh environments. A researcher may use gvolsh's powerful visual exploration capabilities to identify a feature of interest. Once a region of interest is identified, the new utilities enable a researcher to instantly apply IDL's data manipulation and analysis capabilities to the desired focus area. xmovieOur new Visualization Lab required new software with which to drive our tiled 3D display system. DOE has been developing a new animation tool called xmovie that is targeted at precisely this application. VETS collaborated with Lawrence Livermore National Lab (DOE) to add stereo capability and other scaling and placement functions to the tiled-display movie playback software, integrated it into our own production environment, and fed the improvements back into the core DOE distribution of the package. Enterprise web servicesSince 1993, SCD managed and maintained many individual websites for most of the divisions and programs at NCAR and UCAR using surplus equipment and an ad hoc arrangement for site administration. As the growth and use of the World Wide Web matured, access to data and information via web browsers increased dramatically. Web-based technologies began to form the basis for a large fraction of the interactive access to data and information throughout UCAR. Working with members of the UCAR Information Technology Committee's (ITC) Web Advisory Group, SCD identified the major hardware and site-authoring tools required to meet all divisional and UCAR web services for the next three to five years. With funds from the ITC, a new system was purchased in late 1999 and was placed into service early the next year. Since that time, SCD has concentrated on providing enterprise-wide web services to the organization. Toward this end, SCD installed and is now supporting a UCAR-wide search engine: the excellent Inktomi engine. Additionally, SCD is actively researching web technologies that can be used to support the many scientific efforts taking place at NCAR and UCAR. For example, SCD has researched the use of SourceForge for supporting distributed software development groups. SCD has also been actively developing ways of supporting centralized user authentication methods that will be necessary for web-based applications. Over the past year, the usage of the NCAR/UCAR websites has accelerated at an incredible pace. The web servers currently sustain a load of almost 20 million hits per month, with mid-day peaks of over 30,000 hits per hour. To sustain this high load level with high reliability, the web architecture was completely redesigned during the FY2001. This was a major upgrade, and it comprised most of the work the Web Engineering Group accomplished over the last half of the reporting period. The new architecture is cluster-based. It is centered on a centralized high-availability file server. Then there are two tiers of web server system employed to deliver the content to the user. The first tier is comprised of small, cheap systems. The intent is to be able to incrementally grow the capacity of the web cluster as needed for the non-dynamic content that is generated from the websites. This can easily be done with small, cheap systems. The second tier systems, currently under development, are intended to be the systems where more CPU-intensive and/or data-intensive web applications are run. The first-tier systems will be configured to pass requests for these more intensive applications on to the second-tier systems. This architecture allows the web servers to be incrementally "right-sized" for the types of loads that take place on the system. The following diagram illustrates the new architecture.
We have also prepared a new program plan for this effort that includes additional software engineering support for the effort in FY2002. Plans call for much-improved access to web logs and statistics, web gateways to services such as mail, staff/user directory information services, security, streaming media, and search engine agents. Research and special projects in data and visualizationThe Earth System Grid I and IIDuring FY2001 we continued our efforts on the Earth System Grid project with our DOE collaborators. Mid-year we developed a Phase II proposal for the effort and were awarded The Earth System Grid II (ESG, http://www.earthsystemgrid.org/), a three-year DOE-sponsored collaboratory research project involving NCAR and several DOE labs (ORNL, ANL, LBNL, LLNL) and NCAR. The project is focused on building a DataGrid for climate research that facilitates management and access to climate model data across high-performance broadband networks. It builds on one of the more exciting recent developments in computational science, Grid Technologies and the Data Grid. The Data Grid is a next-generation framework for distributed data access in the high-performance computing arena, and it addresses security, transport, cataloging, replica management, and access to secondary storage. We successfully demonstrated the results of this work at the SC2000 conference and established NCAR as a node on the Data Grid. We intend for this important effort to have significant linkage with DLESE, DODS, NSDL/THREDDS, the Community Data Portal, and emerging international efforts in the DataGrid area. Securing the next-generation project is one of our more important accomplishments this year, as it positions us well to develop NCAR's computational environment as an important element of the international Grid. The Visual Geophysical Exploration Environment (VGEE)It is widely acknowledged that there are myriad opportunities to incorporate new information technologies into educational programs.
This is a collaborative effort between SCD and UCAR's Digital Library for Earth System Education (DLESE), the University of Illinois, and the University of Georgia with SCD's role in the area of dataset preparation and visualization tool integration. During the year we made major advances in the development of the actual visualization tool, developed an innovative new "smart probe" capability, and crafted a new and forward-looking graphical user interface (that is applicable to other higher-end environments). We also established linkage with the Unidata-led THREDDS project, with the intent of integrating the VGEE tools with the metadata discovery environment that is the focus of the project. We are in the final phases of VGEE, and in the upcoming year we will be completing the development of the visualization tool, deploying it in one or two classroom settings, producing a final report for the overall project, and exploring options for new funding sources. The Community Data Portal (CDP)Responding to a growing demand for flexible, web-based access to scientific data -- both by research investigators as well as the community of data consumers -- SCD launched a new project in FY2000 called the Community Data Portal (CDP). There are some nice new community technologies that serve as a substrate for this activity: DODS (the Distributed Oceanographic Data System), LAS (PMEL's Live Access Server), Ferret (PMEL's analysis system), and our own NCL. Building on these technologies in FY2001, SCD cultivated a number of collaborative pilot projects including ACACIA (CGD), Reanalysis (SCD/DSS), VEMAP (CGD), and space weather simulation (HAO), and moved this effort from the concept stage into an active development and release phase. We also began the process of analyzing system requirements in the context of simulated user loads, and subsequently specified and moved into acquisition phase for a new system, to be deployed in FY2002. Relationships were established between the CDP and the Earth System Grid project, the THREDDS project, and it was identified as an NCAR Strategic Initiative. FY2002 plans call for a dramatic scaling of the effort, support for one or more workshops, integration of university data holdings, and experimental work with very large geophysical turbulence datasets. Chromium/WireGLDuring FY2001, we started a small pilot project aimed at learning about and evaluating the potential of cluster-based visualization. Collaborating with Stanford University and Lawrence Livermore National Labs, VETS staff contributed to the design of the Chromium distributed-visualization framework. This technology appears to be useful and promising, and we plan to move forward by building a visualization cluster and deploying it in our lab for evaluation in FY2002. A Knowledge environment for the geosciencesVETS worked with CSS in providing leadership to the development of an NSF ITR proposal for the Knowledge Environment for the Geosciences. The proposal was not funded but came close, and we continued to develop the idea during the year with the intention of moving it forward in FY2002. Next-generation analysis and visualization softwareOur current generation of analysis and visualization tools is approaching end-of-life relative to many of today's challenges and most of tomorrow's. Rather than continue to incrementally bandage existing software codes, a modern framework is required to facilitate the development of new applications that can cope with data volume (1-10 TB and more) and complexity, collaborative capabilities (e.g. AG), and efficient integration of the most promising new developments from computer science. During FY2001, we began the process of defining a new project that would address this important area. A proposal was prepared and submitted as an NCAR Strategic Initiative entitled "Frameworks and Applications for Terascale Data Analysis and Visualization," and VETS staff began meeting to discuss our future work in this area. This proposed new effort will build on our own long-term work in data analysis and visualization software (i.e. NCL, Vis5D, volume rendering, etc.), a variety of emerging community efforts by DOE, NSF, and university researchers, and our own collaborative research efforts. While the effort is not funded yet, we intend to pursue it aggressively in FY2002. In the meantime, we have been conducting exploratory research work in a variety of areas. A white paper was authored on developing multi-resolution capabilities for large-scale volume visualization with an emphasis on turbulence simulation data. In the software realm, VETS staff experimented with Python, an object-oriented scripting language that has excellent capabilities for integrating legacy and extant software. Python's popularity is growing rapidly in the scientific community, and it's a strong candidate as an integrating layer for next-generation capabilities. We also began to explore the functionality and performance offered by Java, Java3D, and VisAD in the context of the Visual Geophysical Exploration Environment (discussed in Research). Collaborative effortsVETS collaborates with many modeling efforts, research projects, and working groups in the area of advancing data management, analysis, and visualization capabilities. We also work with other organizations to jointly develop new technology, and our move to an Open Source model will accelerate such activities in the future. Staff members work directly with researchers to identify and deploy analysis and visualization solutions on a project-by-project basis, a process that not only advances the research programs but also provides valuable insights into current and future requirements. The list below summarizes collaborative efforts engaged in by one or more VETS staff members: Magneto-hydrodynamic simulations
NCL and CSMNCL, a general-purpose scripting language for geosciences data analysis and visualization, was described above. NCL continues to be a tight and productive collaboration among SCD and CGD staff. Paleoclimate models
Circulation of the lower stratosphereJohn Clyne collaborated with Bill Randel (ACD) on visualizing trajectory simulations of circulation in the lower stratosphere. Supercell storm model with electrification
Decaying stratified turbulenceA collaboration of John Clyne (SCD), Jack Herring (NCAR/MMM), and Yoshi Kimura (Japan) to study decaying stratified turbulence. This is a demanding problem that combines a combination of flow visualization with scalar representation for datasets that range from 256-cubed to 512-cubed. Jupiter jet simulationsJohn Clyne collaborated with Leo Rivier (NCAR and Princeton) to develop visualizations of a massively parallel model of jets in the Jovian atmosphere. Weather Research Forecast (WRF) modelVETS staff served on the WRF Data Management and Analysis working group, providing technical guidance as well as NCL-based software tools that provided new analysis capabilities. A major FY2001 emphasis will be developing 2D and 3D analysis and visualization tools for the WRF effort. The Community Data Portal (CDP)VETS staff are collaborating with a number of projects and efforts including ACACIA, DSS/Reanalysis, THREDDS, VEMAP, and HAO's TIMEGCM effort. The Visual Geophysical Exploration Environment (VGEE)VETS staff collaborate with UCAR/DLESE, the University of Illinois, and the University of Georgia, as described above. Chromium/WireGLCollaboration with Stanford and DOE, as described above. The Earth System Grid (ESG)Collaboration with DOE centers and USC, as described above. Common Model Infrastructure Working Group (CMIWG)
|
||||||||||
SCD FY2001 ASR table of contents