Databases Galore

Ancient Data

Transparency is becoming a buzz word these days.

People want to take a look at the wizard behind the bells, whistles, smoke, mirrors, and the big curtain.

When scientific and political matters are being analyzed this can lead to databases of information.

We have looked at Pentagon 1033 Program databases (Will The Military Become The Police? - 10), NOAA data concerning tornadoes (On The Origin of Tornadoes - 3), and now I am looking at large volumes of NOAA, NASA, and other sources for climate databases.

One database I have processed via software I am developing has about 12,348,532 individual fields of data concerning temperatures around the globe from circa 1878 to quite recently.

That data is a digest, in the sense of it being average temperatures for each month in each year.

It would be about 30 times larger if it was daily temperatures instead of monthly average temperatures.

Since I am looking at trends, the monthly averages are fine since they span close to a hundred and fifty years (a less detailed database goes back another hundred years to the late 1700s).

The NASA and NOAA folks are quite transparent, even to the point of making the data available for downloading.

The software parses the data to unify and clarify it, then it is stored in a MYSQL database where SQL queries can then be made on tables of that data.

By "unify and clarify" I mean changing the names of the columns to make them easier for me to read.

For example, the data that comes from old COBOL or BASIC styled methods of data storage have a certain way about them.

This is how the data came down:

ID,YEAR,ELEMENT,VALUE1,DMFLAG1,QCFLAG1,DSFLAG1,
. . .
. . .
. . .
VALUE12,DMFLAG12,QCFLAG12,DSFLAG12

I renamed them to:

country, station, year, jan, feb, mar, apr, may, jun, jul, aug, sep, oct, nov, dec

The database builders have been collecting data for a long, long time, and the technicians working with it understand the archaic names.

Since I just laid eyes on the data the renaming helps me work with the data more easily.

The "station" value links to another database that has very useful information about the weather station where the monthly temperatures were recorded:

Variables:

ID, LATITUDE, LONGITUDE, STNELEV, NAME, GRELEV, POPCLS,
POPSIZ, TOPO, STVEG, STLOC, OCNDIS, AIRSTN, TOWNDIS, GRVEG, POPCSS

Variable Definitions:

ID: 11 digit identifier, digits 1-3=Country Code, digits 4-8 represent
the WMO id if the station is a WMO station. It is a WMO station if
digits 9-11="000".

LATITUDE: latitude of station in decimal degrees

LONGITUDE: longitude of station in decimal degrees

STELEV: is the station elevation in meters. -999.0 = missing.

NAME: station name

GRELEV: station elevation in meters estimated from gridded digital
terrain data

POPCLS: population class (U=Urban, S=Suburban, R=Rural)

POPSIZ: the population of the city or town the station is location in
(expressed in thousands of persons).

TOPO: type of topography in the environment surrounding the station,
(Flat-FL, Hilly-HI, Mountain Top-MT, Mountainous Valley-MV).

STVEG: type of vegetation in environment of station if station is Rural
and when it is indicated on the Operational Navigation Chart
(Desert-DE, Forested-FO, Ice-IC, Marsh-MA).

STLOC: indicates whether station is near lake or ocean.

OCNDIS: distance to nearest ocean/lake from station (km).

AIRSTN: airport station indicator (A=station at an airport).

TOWNDIS: distance from airport to center of associated city or town (km).

GRVEG: vegetation type at nearest 0.5 deg x 0.5 deg gridded data point of vegetation dataset (44 total classifications).

BOGS, BOG WOODS, COASTAL EDGES, COLD IRRIGATED, COOL CONIFER, COOL CROPS, COOL DESERT, COOL FIELD/WOODS, COOL FOR./FIELD, COOL GRASS/SHRUB, COOL IRRIGATED, COOL MIXED, EQ. EVERGREEN E. SOUTH TAIGA HEATHS, MOORS, HIGHLAND SHRUB, HOT DESERT, ICE, LOW SCRUB, MAIN TAIGA MARSH, SWAMP, MED. GRAZING NORTH. TAIGA, PADDYLANDS, POLAR DESERT, SAND DESERT, SEMIARID WOODS, SIBERIAN PARKS, SOUTH. TAIGA, SUCCULENT THORNS, TROPICAL DRY FOR, TROP. MONTANE, TROP. SAVANNA TROP. SEASONAL, TUNDRA, WARM CONIFER, WARM CROPS, WARM DECIDUOUS, WARM FIELD WOODS, WARM FOR./FIELD, WARM GRASS/SHRUB, WARM IRRIGATED, WARM MIXED, WATER, WOODED TUNDRA

POPCSS: population class as determined by Satellite night lights (U=Urban (greater than 50,000 persons);  (S=Suburban (greater than or equal to 10,000 and less than or equal to 50,000 persons);
(R=Rural (less than 10,000 persons) City and town boundaries are determined from location of station on Operational Navigation Charts with a scale of 1 to 1,000,000. For cities greater than 100,000 persons, population data were provided by the United Nations Demographic Yearbook. For smaller cities and towns several atlases were uses to determine population.

The linkage of the monthly temperatures database to the station database allows one to look at temperature changes then ponder the affects to the local flora, fauna, and community around the weather station.

Anyway, this week I will be generating reports from the data recorded at interesting locations around the globe (here are some interesting weather station names in the country of Burkina Faso: Dori, Ouahigouya, Ouagadougou, Fada N'Gourma, Bobo-Dioulass, Boromo, Gaoua).

I hope to share some of it, including links to the sources of the data.

Why do the deniers hide everything and shroud themselves in darkness?

The climate scientists are very open with their data.

Have a good Monday.

The next post in this series is here.