WDC for Geomagnetism, Kyoto — HAPI Server: Output Formats

HAPI Server › Output Formats · Geomagnetic Data Explorer

The data endpoint returns the same values in up to four formats, chosen with &format=. csv and json are the HAPI formats and are available for every dataset. iaga2002 and wdc are the native exchange formats the data have traditionally been distributed in; they are available for the datasets listed below. This page describes what each format looks like as written by this server, and where it differs from the formal specification.

Which formats each dataset supports

The version-pinned datasets (_final, _definitive, …) support the same formats as their Best Available counterpart. Requesting an unsupported format returns status 1409. The formats a dataset supports are also listed in catalog as x_nativeFormats (in addition to csv and json).

Common rules

• Same values. All four formats are written from the same stored values; only the layout, precision and missing-value codes differ.
• Columns. &parameters= selects columns in csv and json only. iaga2002 and wdc always contain every element of the dataset, as the original distribution files did.
• Time. All times are UT. CSV and JSON stamp the centre of each averaging interval (timeStampLocation in info), e.g. HH:29:30 for hourly values; IAGA-2002 and WDC label each value by the start of its interval (HH:00), as the original files do.
• Range. Records whose time stamp lies in start ≤ t < stop are returned. In WDC formats a record holds a whole hour or day, so the first and last records may be partly filled with missing-value codes.
• Observatory components. Observatory data are served as reported, with no coordinate conversion. CSV/JSON always have the seven columns D, H, X, Y, Z, F, I (unreported ones are fill); IAGA-2002 and WDC contain only the reported components — DHZF or XYZF.
• Line endings. CSV lines end with CRLF (RFC 4180); IAGA-2002 and WDC lines end with LF.

CSV (format=csv, the default)

One record per line, comma-separated, first column the ISO 8601 time, last column versionCode. Column order and units are those of the parameters array in info. With &include=header the response starts with one line holding the info response as JSON, prefixed with #.

2015-03-17T00:00:00Z,99999.0,99999.0,29742.30,-3828.50,35710.80,46631.70,99999.0,30
2015-03-17T00:01:00Z,99999.0,99999.0,29742.30,-3827.80,35710.30,46631.40,99999.0,30

min_kak, columns Time, D, H, X, Y, Z, F, I, versionCode. In this period Kakioka reported X, Y, Z, F, so D, H and I are fill.

• Fill values are given per parameter as fill in info: 99999 for the Dst/AE/ASY/SYM indices, 99999.0 for observatory data, 99.9 for Kp, 999 for ap/Ap, 0 for the quiet/disturbed day lists.
• Precision. Real numbers have 2 decimals, or the number given as x_digits in info (3 for D and I in 1-second data, for example). D and I are in minutes of arc, field intensities in nT.
• Kp is a decimal: the thirds map to .3 and .7 (3− = 2.7, 3o = 3.0, 3+ = 3.3).
• versionCode is a two-digit integer: the tens digit is the processing level (for indices 0 real-time, 1 provisional, 2 final; for observatories 0 variation, 1 provisional, 2 quasi-definitive, 3 definitive), the units digit a revision number. The versionCode description in each info lists the codes that apply.

JSON (format=json)

The info response for the dataset, with "format": "json" and a data array added. Each record is an array in the same column order as CSV; numbers are JSON numbers and fill values appear as numbers too (not null).

{"HAPI": "3.3.1", "status": {"code": 1200, "message": "OK"}, ...,
 "parameters": [...], "format": "json",
 "data": [["2015-03-17T00:00:00Z", 50, -19, 31, 6, 10],
          ["2015-03-17T00:01:00Z", 43, -16, 27, 6, 10]]}

min_ae: Time, AE, AL, AU, AO, versionCode.

IAGA-2002 (format=iaga2002)

The IAGA-2002 data exchange format: fixed-width text of 70 characters per line, 12 header lines, optional comment lines, one data header line, then one line per sample.

 Format                 IAGA-2002                                    |
 Source of Data         INTERMAGNET IRDS                             |
 Station Name           Kakioka                                      |
 IAGA CODE              KAK                                          |
 Geodetic Latitude      36.232                                       |
 Geodetic Longitude     140.186                                      |
 Elevation              36                                           |
 Reported               XYZF                                         |
 Sensor Orientation     HDZF                                         |
 Digital Sampling       1.000                                        |
 Data Interval Type     1-minute                                     |
 Data Type              Definitive                                   |
DATE       TIME         DOY     KAKX      KAKY      KAKZ      KAKF   |
2015-03-17 00:00:00.000 076     29742.30  -3828.50  35710.80  46631.70
2015-03-17 00:01:00.000 076     29742.30  -3827.80  35710.30  46631.40

Header

Comment lines start with #. Comments present in the source file are passed through unchanged; this server adds its own after them, for example the caution and approximate baselines of the monitor stations, or the source metadata of each period when it changes within the request.

Data records

DATE TIME DOY (YYYY-MM-DD HH:MM:SS.sss DDD) followed by four 10-character value columns (F9.2 preceded by a blank). For observatories the column titles are the IAGA code plus the element letter (KAKX); for indices they are the index names. Units: nT, and minutes of arc for D and I. Hourly values are labelled HH:00:00; 7.5-second values carry .500 where applicable. sec_kyo_<stn>_press has a single column P in hPa with 3 decimals.

Missing values

• Missing (observed but no value): 99999.99 for indices. For observatories the code follows the precision of the column, as in WDC Kyoto’s other IAGA-2002 files: 99999.00, 99999.90 or 99999.99 when the column’s valid values have 0, 1 or 2 significant decimals. A column with no valid value at all therefore shows 99999.00.
• Not reported (the element was not observed at all): the code the source used, 88888.00 or 88888.80, is kept. In CSV and JSON both cases are fill; info lists the not-reported periods as x_unreportedElements.
• Pressure (sec_kyo_<stn>_press): 99999.000.

Multiple blocks in one response

If the reported components change within the requested period (Kakioka, for example, changed from DHZF to XYZF in January 2005), the response contains one complete IAGA-2002 block (header, data header, data) per period, one after another. Read the header of each block. Header values are those of the last month of the block; where a source value such as Sensor Orientation changes within a block, the comment lines list the value for each period.

WDC (format=wdc)

The fixed-width WDC exchange formats. There are three record layouts plus the plain-text AE 2.5-minute file; which one a dataset uses is shown in the table above. All numbers are integers, right-aligned. Minute records are grouped by element over the whole request: for AE, all AU records come first, then AL, AE and AO, as if the separate original files had been concatenated. Hourly records are grouped month by month, and by element within each month. Columns in the tables are counted from 1. Where this output departs from the original format descriptions, the notes at the end of this section say so.

Index hourly — 120 bytes, one record per index per day (hour_dst, hour_ae)

DST1503*17  X220   0   4   3   3   2  13  45  25 -18 -54 -83 -75 -54 -55 -75 -93-118-143-162-154-177-182-198-234-225 -84
DST2607*01PPX120   0   6  11  -4   9   4  -2  -1   5  10  21  20  17  14  13  16  18   9  10  13  17  22  13  14  11  11
DST2610*01RRX020   0   7   4   5   7   9  11  12  13  10   7   7   9  10   8   6   6   4   1   4   4   0  -3   0   1   6
AU 6603*01    19 000 012 011 014 021 029 022 038 029 027 028 030 033 012 008 014 014 019 020 017 012 015 015 013 017 020
AL 6603*01    19 000-009-005-002-011-040-009-022-031-015-016-042-099-022-015-006-008-043-043-122-072-013-006-010-013-028
AE 6603*01    19 000 022 015 016 032 069 032 060 059 042 044 072 132 034 024 020 022 062 063 140 084 028 022 023 029 048
AO 6603*01    19 000 001 003 006 005-006 006 008-001 006 006-006-033-005-003 004 003-012-012-052-030 001 004 001 002-004

Value in nT = base × 100 + field. The base is 0 unless a value would not fit in four characters. Missing hours are 9999, and the daily mean is the mean of the hours present (9999 if none). AE values are written with at least three digits (022, -009) as in the original files. Each day carries the version of that day, so a request spanning final and provisional months shows both.

Minute — 400 bytes, one record per element per hour (min_ae, min_asysym, min_<obs>)

AEALAOAU    150317U00AU               31    27    24    27 ...    21    23    22    23    26
AEALAOAU    150317L00AL              -19   -16    -8    -7 ...   -13   -13   -12   -10   -11
ASYSYM N6E01150317D00ASYWDCC2KYOTO    19    18    16    16 ...
053768140186150317X00KAKFMIAGA2002 29742 29742 29742 29742 ... 29732 29732 29732 29738
053768140186900317D00KAK           -4061 -4061 -4060 -4060 ...

Units are nT, and 0.1 minute of arc for observatory D and I (-4061 = −406.1′ = −6°46.1′). Values are rounded to integers (half away from zero). Missing values are 99999; not-reported elements keep the source’s code (88888/88889).

Observatory hourly — 120 bytes, one record per element per day (hour_<obs>)

KAK9003D17        -8 745 742 717 691 676 677 687 700 706 705 705 705 705 707 709 713 715 717 719 719 718 720 736 753 712
KAK9003H17       297 466 467 461 460 462 467 469 466 460 461 460 460 460 460 461 462 463 465 467 470 474 480 478 474 466
KAK1503X17     0 292 538 527 517 513 528 567 556 518 463 424 442 461 455 433 411 401 393 373 416 404 432 425 365 372 456

Value = base × 600 + field in 0.1 minute of arc for D and I (-8, 745 → −4800 + 745 = −4055 = −405.5′), and base × 100 + field in nT for the others (297, 466 → 30166 nT). One base is used per element per month. Missing values are 9999; the daily mean is given only when all 24 hours are present, otherwise 9999. Records are ordered month by month, and within a month element by element. Only days within the requested period are written.

AE 2.5-minute — plain text (min2p5_ae)

1970 03 01 000000    433   -344   -127     89
1970 03 01 000230    392   -315   -119     77

One line per 2.5-minute sample: YYYY MM DD HHMMSS (start of the interval) followed by AE, AL, AO, AU in nT, each in 7 characters. Missing values are 99999.

Differences from the original format descriptions

• AE minute, columns 25–34. Written from the version of each hour, using the values of the original description (KYOTO/E02 final, PRVAE/E02 provisional) and QUICKLK for real-time values, which the description does not cover. Some older original files carry other, undocumented notes here; they are not reproduced.
• AE and ASY/SYM minute, hourly mean (columns 395–400). Computed here from the 60 minute values. The value in the original files may differ by 1 nT.
• AE hourly, base value (columns 17–20). Chosen here as described above, so it can differ from the original files; base × 100 + field is the same.
• Dst hourly, columns 11–12. PP marks provisional values, as in the provisional Dst files, in addition to RR for real-time values.
• Observatory minute, column 25. The original description reserves it for a data-origin code; this output does not carry one, and columns 25–34 hold FMIAGA2002 or blanks as described above.
• Observatory hourly, columns 15–16. Column 15 (international quiet or disturbed day) is blank, and column 16 is 0 for 2000 onwards. The base is fixed per element per month and set low enough that no hourly value is below −999, so it is never adjusted for a single day.

Format references

• IAGA2002 Data Exchange Format
• Dst index: record format (120 bytes)
• AE index: 1-minute record format (400 bytes)
• ASY/SYM indices: 1-minute record format (400 bytes)
• WDC exchange format for observatory 1-minute values (400 bytes)
• WDC exchange format for observatory hourly means (120 bytes)
• HAPI data specification (CSV and JSON)