Research data goals for SCD's Data Support Section
- Prepare the world's surface and upper air
observations for 50 years (1948-1997)
- Analyze the world's atmosphere for 50 years at
6-hour time steps (determine temperature, winds,
radiation, etc. at 28 levels), and soil conditions
- Help many climate assessment research projects
(crops, rivers, forests, etc.)
- Establish an archive of data from three
mesoscale models over North America
- Help prepare the world's best dataset of world
surface ocean data from ships and buoys
- Help users obtain access to many datasets
The Data Support Section (DSS) maintains a large, organized archive of
computer-accessible research data that is made available to scientists
around the world. The archive represents an irreplaceable store of observed
data and analyses and is used for major national and international
atmospheric and oceanic research projects. The DSS group started working
in 1965 and has been working on large projects and building the data
archives ever since.
There are now about 500 distinct datasets in the archive, ranging in size
from less than 1 MB to over 1 TB. The total volume of data in the DSS
archive was 2.4 terabytes in August 1990 and 10 terabytes in September
1998. We have been adding a lot of reanalysis data and other analyses.
The change of data storage with time has been as follows:
| Data stored for Data Support, and total mass store
|
|
|
| Data Support Section
| Total mass store
|
|
| Date
| Bit Files
| Volume
(TB)
| Bit Files
| Volume
(TB)
| DSS %
of MSS
|
| 13 Aug 90
| 61,335
| 2.437
| --
| 14.430
| 16.9
|
| 4 Aug 91
| 65,518
| 2.689
| 715,000
| 19.400
| 13.9
|
| 3 Aug 92
| 80,538
| 3.085
| 1,060,000
| 27.270
| 11.3
|
| Aug 93
| 103,314
| 4.072
| 1,351,271
| 36.280
| 11.2
|
| 15 Sep 94
| 119,703
| 4.751
| 1,849,466
| 47.423
| 10.0
|
| 14 Feb 95
| 123,877
| 5.085
| 1,966,990
| 52.456
| 9.7
|
| 24 Jan 96
| 137,680
| 5.950
| 2,486,471
| 67.590
| 8.8
|
| 28 Aug 96
| 143,340
| 6.770
| 2,888,639
| 78.964
| 8.6
|
| 28 Feb 97
| 151,509
| 7.513
| 3,289,224
| 91.399
| 8.2
|
| 17 Oct 97
| 159,945
| 8.482
| 4,046,678
| 110.359
| 7.7
|
| 2 Sep 98
| 167,073
| 10.032
| 5,038,611
| 147.439
| 6.8
|
The DSS staff provides assistance and expertise in using the archive to
help researchers locate data appropriate to their needs. Users may obtain
copies of data by network access, on various tape media, or they may use
data directly from the NCAR MSS. DSS staff also assist scientists by
providing data access programs (to read and unpack data), other software
for data manipulation, and dataset documentation.
This is a brief overview of DSS tasks: gather and prepare data,
provide user services, work on major projects, and track new
technology and prices.
Update the archives and add new datasets
Many datasets continue in time. We have about 500 datasets and need to
update a number of them. This becomes a big task. Also, some datasets
(such as world sea surface temperature analyses) are sometimes revised
by using better procedures.
- We have to do many dataset updates on monthly, yearly, or
three-year time scales.
- We keep obtaining new datasets as they emerge from the community,
and we develop datasets.
- We keep developing a base of information about the data we have,
and some summary information about data in the community.
Provide user services
- Many people read data directly from the NCAR MSS into programs
that run on NCAR computers. All the DSS archives are available from
the MSS. We provide access software and file location descriptions
so that data processing is fast and relatively easy.
- We send data and access software to many places in the U.S. and
the world, often on tape. We have shipped over 2500 CD-ROMs.
- We answer user questions. Keeping the web site up to date is a
significant task.
Work on several large projects
The Data Support Section has always had some large projects that we work
on, often in cooperation with other groups in the U.S. and the world.
These projects usually support a certain research activity, and they
usually have tasks that develop datasets, improve the archives, and add
new data products. Some of the main projects in FY1998 are as follows:
- NCEP/NCAR reanalysis of the global atmosphere for 1948-1998
- Make global analyses at 28 vertical levels, every 6 hours, for 50 years.
- We have gathered and prepared many datasets of observations. This has
been a huge project for us.
- A great deal of effort was required for data preparation in 1991-1995.
The workload from this project increased in 1996 - March 1998 because data
preparation was more difficult and large volumes of output data products
were received at and distributed from NCAR.
- This project completed 50 years on 23 July 1998.
- We compiled a list of tasks to do before the next reanalysis.
- More information is provided at the DSS website
NCEP/NCAR Global
Reanalysis Project.
- Work with ECMWF (send world observations for a 40-year reanalysis)
with world coverage for 50 years.
- ECMWF plans to start a reanalysis production run in May 1999. In
preparation, we have been providing data since May 1998.
- Obtain climate model data for Assessment studies (a new set of projects).
- Obtain data for three mesoscale models for GCIP (data for 1995-on).
- Model output began arriving more rapidly in April 1998.
- MSS archives have been created and documented for user access.
- More information is provided at the DSS website
NCAR's GCIP Model Output
Archive.
- COADS (world surface ocean observations, 1854-1997).
- This project with other labs started in 1981.
- It expands and updates the world's best surface ocean dataset.
- It is used for reanalysis. It has temperature, pressure, wind,
clouds, etc. It provides much of what the world knows about ocean
surface temperature changes from 1860 to the 1990s.
- More information is provided at the NOAA website
Comprehensive Ocean-Atmosphere
Data Set (COADS).
- Maintain a data bank for the observations and models of the high
stratosphere (70-1000 Km). This work started in 1984.
- Kept some data exchanges going with other countries.
Track new technology and prices
We send much data to poor people, as well as to rich people. The
technology we use should depend on capability, reliability, ease-of-use,
and cost. We track the capability and cost of tape drives, disk media
(CD-ROM), hard disks, and computing power. Some documents are available
about the technology and the history for 1965-1998. One person (Jenne)
spends about 3% of his time on this.
The NCEP/NCAR reanalysis project
This huge project has completed 50 years of global analyses with 6-hour
time steps. The production of analyses started in June 1994. Twenty-two
years were finished in September 1996. Forty years were finished in
October 1997, and 50 years were completed in July 1998. A long paper
was in the Bulletin of the AMS in March 1996. A CD-ROM was included; the
Bulletin is mailed to about 13,000 people (worldwide).
- The project completed production of 40 years of analyses for 1957-97 on
October 14, 1997. And 50 years (for 1948-1997) were done on July 23, 1998.
- Send observations to NCEP for reanalysis. NCAR sent more years
of data to NCEP -- all the world's land and ocean surface, upper
air, and satellite observations. NCEP had data needed for 1968-95
by September 1996. We sent them observations for 1957-67 by May 23, 1997.
Data for 1948-1957 was ready
by March 1998. It is a lot of work to prepare the data (using about 60%
of our time). Many diagnostic checks are needed to detect and understand
problems. Some days have been like "Black Sunday."
- October 1997: the first international conference on reanalysis was
held. About 200 people were there. A 487-page document with the papers
is available. Many reports on the NCEP/NCAR 50-year reanalysis (1948-97),
the ECMWF 15 years (1979-93), and the Goddard one (about 13-15 years).
- Handle the output from reanalysis (54 GB per year of analyses).
These data arrived at NCAR at the rate of about 100 GB of data per
month during 1996-1997.
Data from three mesoscale models for North America
We have been involved with data plans and preparation for the GCIP
project since 1993. Some data have been flowing for two years, but the
fast data flows started about April 1998. Also, much more information
about the models is now available, and the pace of model improvements
has been good.
A new archive of climate model data for assessment studies
From July 1997 to March 1998 we worked to establish data flows from three
to five of the world's climate models. During March - September 1998, the
data are flowing. It is working! But there is a lot of work left to do.
Ocean data
To support reanalysis and general research, the Comprehensive Ocean
Atmosphere Data Set (COADS) has been updated. The 1946-1979 and 1990-1995
periods were reprocessed and supplemented with additional data. Surface
winds estimated from the NSCAT and ERS2 satellite sensors have also been
made available, as have near-real-time and long-term SST analyses.
Preparations and tests for the upcoming QSCAT satellite data have been
completed.
Maintain the archives and help users
We now have about 500 datasets. We updated many datasets during the year,
and added some new ones. Some data flows are large. This is a large task.
Global analysis and observations from operational centers
The normal and advanced analysis archives from NCEP operations were
routinely updated. The advanced archives started in 1990. Forecasts are
included. In addition, we archive all of the global observations from
NCEP. Analyses also come from ECMWF. Most of these sets are now current
through July 1998. In addition, we obtain the reanalysis output data
from both NCEP and ECMWF.
Data exchange
Under the U.S.-Russian bilateral agreement, the data exchange has been
active. Jenne and two other U.S. PIs visited Russia in August 1997 to
draw up plans for another year of work. Two Russian (and four U.S.) PIs
visited NCAR for a week in September 1998.
A good data exchange has started with the Chinese Academy (IAP Institute,
Beijing). Jenne visited Beijing in September 1996. A considerable amount
of data had been exchanged by December 1996. Documents are available.
Two Chinese visitors spent two weeks at NCAR in September 1998.
NCAR and NOAA also reached an exchange agreement with State Oceanic
Administration of the Chinese National Oceanographic Data Center in
Tianjin, China (in April 1998). We furnished a land surface station
archive (in December 1997). Beginning in 1999, 1.8 million ship
observations from U.S. logbooks, which were discovered in the U.S.
National Archives, will be digitized by China as part of the exchange.
The NCEP/NCAR project has completed 50 years of global reanalyses, with
output each six hours. The analyses are done at a resolution of T62
(208 Km) and 28 levels in the vertical. The project started in 1991,
based on many earlier years of data gathering, model development, and
related experience. The task of NCAR (DSS section) is to provide the
observations and handle the output. There are millions of surface and
upper air observations (from balloons, aircraft, and satellite temperature
and cloud wind data) that are being used. The operational production of
analyses started in June 1994, and it was finished in July 1998. The last
10 years of data output were sent to NCAR during September and
October 1998.
Help for several major global reanalysis programs
Our main work started early in 1991 to prepare observations for the
NCEP/NCAR 50-year reanalysis project. This work and its extensions
is also helping other reanalysis projects around the world.
- We prepared many old datasets and provided NCEP with world
observations needed to make new analyses for 1948-1998 (50 years).
- During 1997-1998: Give ECMWF (in England) global observations so that
they can make a new reanalysis for at least 1957-1998 (over 40 years).
Production will start in May 1999.
- About 1993-1997: Help the NASA Goddard efforts to do a 13-15-year
reanalysis. NCEP has passed most of the data from NCAR on to Goddard.
- An interim new reanalysis by NCEP. Production of 1979-1995 started
in April 1998, using a Cray J90 computer at DOE, Livermore. On May 7,
it was restarted with a model having improved radiation parameterizations.
The model has been updated to take care of known problems. The resolution
is still T62 (208 Km) with 28 levels.
- Plans for 1998-2000: help a new reanalysis by NCEP to do 1948-2000
(about 54 years), that will start about year 2001. NCAR will find some of
the missing observations and carry out 20 steps to enhance the archives.
What main observations are used for reanalysis?
The analysis methods use assimilation methods and forecast models to
obtain the best analyses. We prepared global collections of data for
the following main types, as much as was available:
- Rawinsondes: Upper air data from balloon ascents (temperature,
moisture, pressure, winds)
- Pibals: Winds at different height levels in the atmosphere.
- Aircraft reports: Wind and temperature from commercial aircraft and
reconnaissance flights.
- Surface land synoptic data (temperature, pressure, etc. from 7500 stations)
- COADS: Ocean surface data from ships and buoys (pressure,
temperatures, winds, etc.)
- Cloud drift winds (cloud motions from satellite photos): Some
start in 1967; most start 1974 to 1978.
- Satellite sounder data: Temperature at different levels in the
atmosphere. The first data start in April 1969.
What we are doing for mesoscale research
We have archived mesoscale model data from NCEP since 1971 (for North
America). We are now the model data center for GCIP (this work started
about 1993-1994).
- The DSS archive of mesoscale analysis and forecast grids (from LFM and
NGM models started in 1971.
- Archive of hourly North American GTS observations started 1976 (1000
stations).
- Archive model data for the North American GCIP experiment. Grids
each three hours from three models: Eta (NCEP), Maps (FSL-Boulder),
and MRF (Canada). The data start in 1995. The data have analyses
(each three hours) and there are forecasts.
- Help start a 20-year Mesoscale Reanalysis Project (the planning was
from July 1997-April 1998). NCEP will use the Eta model for operations.
- Our global archives of observations are essential for mesoscale work.
Our work with global climate assessment studies
The new round of studies in 1998-1999 is going to tax the resources of the
Data Support Section. A short history of our involvement with studies of how
crops, rivers, forests, etc. could respond to climate changes is as follows:
1987: Assessment studies of USA start
Congress wanted studies of effects of climate changes for USA
DSS worked with 40 PI groups in USA
We were very busy September 1987 - April 1988
1989-90: EPA sponsors international studies
Crops and climate (25 countries)
Forests (10 countries)
10 rivers in the world, 10 countries
Note: DSS provides climate model data
1992-1995: The U.S. Country Studies program (with 56 countries)
June 1992 (RIO): The U.S. promised to help nations make studies
DSS very busy Oct 1993 - Feb 1994 and Dec 1994 - Apr 1995
Big work-study conferences held January 1994 and February 1995
1997-98: The U.S. Government is starting new studies of USA
NIGEC studies: Build up plans, June 1997 - May 1998
U.S. agencies: Plans Sep 1997 - Sep 1998
DSS: Work to line up climate model data: Sep 1997 - Sep 1998
The main archive for reanalysis output data is at NCAR, but parts of these
data are also at three other archives (Goddard, NOAA/ERL, and NOAA/NCDC).
NCAR delivers the data mainly in three ways (shown below): read it
directly from the NCAR MSS into programs run on NCAR computers; send it
on tapes or CD-ROMs; deliver smaller amounts by Internet.
| Method
| Amount of use
|
| On computers at NCAR
| 7049 GB (Jan 95 - Dec 97)
|
| Sent on tapes
| 3840 GB by Sep 3, 1998
|
| Sent on CD-ROMs
| 2504 CD; with 1653 GB, by 27 Aug 1998
|
- Reanalysis data first became available on the NCAR MSS in late 1994.
The annual summaries for 1995, 1996, and 1997 are shown for the NCAR and
University user communities. The number of unique users, numbers of MSS
files, and GB accessed on a read transfer are shown. The NCEP/NCAR
Reanalysis data have been popular.
|
| NCAR
| University
| Total
|
| Year
| Users
| Files
| Gbytes
| Users
| Files
| Gbytes
| Users
| Files
| Gbytes
|
| 1995
| 4
| 1417
| 292
| 9
| 569
| 123
| 13
| 1986
| 416
|
| 1996
| 14
| 3753
| 811
| 45
| 8516
| 1869
| 59
| 12269
| 2679
|
| 1997
| 30
| 4223
| 893
| 66
| 13017
| 2907
| 101
| 17896
| 3954
|
- Sending reanalysis data by tape.
The first order was in December 1994. The cumulative orders are given
in this table:
NCEP/NCAR reanalysis data sent by tape
| Date
| Number
of Orders
| Cumulative Data
Volume
(GB)
|
| 1 Jan 1996
| 13
| 85
|
| 1 Jan 1997
| 68
| 1187
|
| 13 Oct 1997
| 117
| 1931
|
| 7 Apr 1998
| 166
| 2800
|
| 3 Sep 1998
| 197
| 3840
|
- All CD-ROM sales by February 1998:
One CD-ROM is prepared with a subset of data for each year of reanalysis.
There is a short text about the CD-ROMs. Sales of the NCEP/NCAR reanalysis
CD-ROMs (one per year) are given below. The sales price is about $10 per
CD-ROM:
Reanalysis CD-ROM sales by given date
(Cumulative orders and CDs sold)
| Date
| Unique
CD
| Orders
| CD-ROMs
sold
| Volume on
CDs (GB)
|
| Apr 21, 1997
| 8
| 14
| 81
| --
|
| May 27, 1997
| 10
| 31
| 203
| 134
|
| Jul 23, 1997
| 10
| 58
| 387
| --
|
| Aug 12, 1997
| 10
| 72
| 502
| 331
|
| Oct 13, 1997
| 12
| 106
| 755
| --
|
| Feb 28, 1998
| 18 (79-96)
| 185
| 1563
| 1032
|
| May 11, 1998
| 21
| 219
| 1955
| --
|
| Aug 27, 1998
| 25 (73-97)
| 269
| 2504
| 1653
|
Special reanalysis CD-ROMs
The Bulletin of the AMS (Mar 1996) had a long paper about the NCEP/NCAR
reanalysis. It also had a CD-ROM with 13 years of monthly reanalysis
data (1882-94) and selected daily data for 1993.
It was mailed to 12,500 people (all who receive the Bulletin of the AMS).