.TH MGD77MANAGE l "15 May 2006" "GMT4.1.2" "GMT4.1.2"
.SH NAME
mgd77manage \- Manage extra columns in MGD77+ files
.SH SYNOPSIS
\fBmgd77manage\fP \fINGDC-ids\fP [ \fB\-A\fP[\fB+\fP]\fBa\fP|\fBc\fP|\fBd\fP|\fBD\fP|\fBe\fP|\fBg\fP|\fBi\fP|\fBn\fP|\fBt\fP|\fBT\fP\fIfileinfo\fP ] 
[ \fB\-C\fP\fBf\fP|\fBg\fP|\fBe\fP ] [ \fB\-D\fP\fIabbrev1,abbrev2,...\fP ] [ \fB\-E\fP\fIempty\fP ] 
[ \fB\-I\fP\fIabbrev\fP/\fIname\fP/\fIunit\fP/\fBt\fP/\fIscale\fP/\fIoffset\fP/\fIcomment\fP ] [ \fB\-Ne\fP|\fBk\fP|\fBm\fP|\fBn\fP ] 
[ \fB\-Q\fP[\fIvalue\fP] ] [ \fB\-V\fP ] [ \fB\-bi\fP[\fBs\fP|\fBS\fP|\fBd\fP|\fBD\fP][\fIncol\fP] ]
.SH DESCRIPTION
\fBmgd77manage\fP deals with maintaining extra custom columns in MGD77+ netCDF files.
You can either delete one or more columns, add a new column, update an existing
column with new data, or supply error correction information (*.e77 files).  New data may come
from a table (ASCII unless \fB\-b\fP is used), be based on existing columns and certain
theoretical expressions, or they may be obtained by sampling a grid
(choose between GMT grid or a Sandwell/Smith Mercator *.img grid) along track.  The new data
will be appended to the MGD77+ file in the form of an extra data column of specified type.
The data file will be modified; no new file will be created.  For the big issues, see the DISCUSSION
section below.
.TP
\fINGDC-ids\fP
Can be one or more of five kinds of specifiers:
.br
1) 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
.br
2) 2-character <agency> codes which will return all cruises from each agency.
.br
3) 4-character <agency><vessel> codes, which will return all cruises from those vessels.
.br
4) =<list>, where <list> is a table with NGDC IDs, one per line.
.br
5) If nothing is specified we return all cruises in the data base.
.br
(See mgd77info \fB\-F\fP for agency and vessel codes).
The ".mgd77" or ".nc" extensions will automatically be appended, if needed (use \fB\-I\fP
to ignore certain file types).  Cruise files will be looked for first in the current directory
and second in all directories listed in \fB$MGD77_HOME\fP/mgd77_paths.txt [If \fB$MGD77_HOME\fP
is not set it will default to \fB$GMTHOME\fP/share/mgd77].
.br
NOTE: Under Windows, only (1) and (4) are currently supported.
.SH OPTIONS
No space between the option flag and the associated arguments
.TP
.B \-A
Add a new data column.  If an existing column with the same abbreviation already exists in the file
we will cowardly refuse to update the file. Specifying \fB\-A+\fP overcomes this reluctance (However,
sometimes an existing column cannot be upgraded without first deleting it; if so you will be warned).
Select a column source code among \fBa\fP, \fBc\fP, \fBd\fP, \fBD\fP, \fBe\fP, \fBg\fP, \fBi\fP,
\fBn\fP, \fBt\fP, or \fBT\fP; detailed descriptions for each choice follow:
.br
.sp
\fBa\fP Append filename of a single column table to add.  File must have the same number of rows as
the MGD77+ file. If no file is given we read from stdin instead.
.br
.sp
\fBc\fP Create a new column that derives from existing data or formulas for corrections and reference
fields.  Append \fBc\fP for the Carter corrections subtracted from uncorrected depths, \fBg\fP for the
IGF gravity reference field (a.k.a "normal gravity"), and \fBm\fP for the IGRF total field magnetic
reference field.  For gravity we choose the reference field based on the parameter Gravity Theoretical
Formula Code in the cruise's MGD77 header.  If this is not set or is invalid we default to the IGF 1980.\"'
You can override this behaviour by appending the desired code: 1 = Heiskanen 1924, 2 = International
1930, 3 = IGF1967, or 4 = IGF1980.
.br
.sp
\fBd\fP  Append filename of a two-column table with the first column holding distances along track
and the second column holding data values.  If no file is given we read from stdin instead.  Records with matching
distances in the MGD77+ file will be assigned the new values; at other distances we set them to NaN.
Alternatively, give upper case \fBD\fP instead and we will interpolate the column at all record distances.
See \fB\-N\fP for chosing distance units and \fB\-C\fP for chosing how distances are calculated.
.br
.sp
\fBe\fP Expects to find an e77 error/correction log from \fBmgd77sniffer\fP with the name \fINGDC_ID\fP.e77 in
the current directory or in $MGD77_HOME/E77; this file will examined and used to make modifications to the
header values, specify a systematic correction for certain columns (such as scale and offset), and add or
update the special column \fBflag\fP which may hold bitflags (0 = GOOD, 1 = BAD) for each data field in the
standard MGD77 data set.  Any fixed correction terms found (such as needing to scale a field by 0.1 or 10 because the source
agency used incorrect units) will be written as attributes to the netCDF MGD77+ file and applied when the
data are read by \fBmgd77list\fP.  Emphemeral corrections such as those determined by crossover analysis are
not kept in the data files but reside in correction tables (see \fBmgd77list\fP for details).  By default, the
first character of each line in the e77 file (which is Y or N) will be consulted to see if the corresponding
adjustment should be applied.  You can override this behavior by appending one or more modifiers to the \fB\-Ae\fP command:
\fBH\fP will apply all header corrections regardless of Y/N, whereas \fBh\fP will ignore all header corrections,
\fBF\fP will apply all fixed systematic trend corrections regardless of Y/N, whereas \fBf\fP will ignore them,
\fBn\fP, \fBv\fP, \fBs\fP, and \fBg\fP will ignore bitflags pertaining to navigation, data values, data slopes,
and grid comparisons, respectively.  For the bitflags, \fB\-Ae\fP means we will replace any existing bitflags
in the file with the new values whereas \fB\-A+e\fP means we will merge new flags and existing flags.
.br
.sp
\fBg\fP Sample a GMT geographic (lon, lat) grid along the track given by the MGD77+ file using
bicubic interpolation (however, see \fB\-Q\fP).  Append name of a GMT grid file.
.br
.sp
\fBi\fP Sample a Sandwell/Smith Mercator *.img grid along the track given by the MGD77+ file using
bicubic interpolation (however, see \fB\-Q\fP).  Append the gridded data scale (typically 1 or 0.1),
the IMG file mode (0-3), and finally the img grid filename.  The modes stand for the following:
(0) Img files with no constraint code, returns data at all points, (1) Img file with constraints coded,
return data at all points, (2) Img file with constraints coded, return data only at constrained points
and NaN elsewhere, and (3) Img file with constraints coded, return 1 at constraints and 0 elsewhere.
.br
.sp
\fBn\fP Append filename of a two-column table with the first column holding the record number (0 to nrows - 1)
and the second column holding data values.  If no file is given we read from stdin instead.  Records with matching
record numbers in the MGD77+ file will be assigned the new values; at other records we set them to NaN.
.br
.sp
\fBt\fP Append filename of a two-column table with the first column holding absolute times along track
and the second column holding data values.  If no file is given we read from stdin instead.  Records with matching
times in the MGD77+ file will be assigned the new values; at other times we set them to NaN.
Alternatively, give upper case \fBT\fP instead and we will interpolate the column at all record times.
.TP
.B \-C
Append a one-letter code to select the procedure for along-track distance
calculation when using \fB\-Ad\fP|\fBD\fP (see \fB\-N\fP for selecting distance units):
.br
	\fBf\fP Flat Earth distances.
.br
	\fBg\fP Great circle distances [Default].
.br
	\fBe\fP Geodesic distances on current GMT ellipsoid.
.TP
.B \-D
Give a comma-separated list of column abbreviations that you want to delete from the MGD77+ files.
Do NOT use this option to remove columns that you are replacing with new data (use \fB\-A+\fP instead).
Because we cannot remove variables from netCDF files we must create a new file without the columns
to be deleted.  Once the file is successfully created we temporarily rename the old file, change
the new filename to the old filename, and finally remove the old, renamed file.
.TP
.B \-E
Give a single character that will be repeated to fill empty string values, e.g., '9' will yield
a string like "99999..." [9].
.TP
.B \-I
In addition to file information we must specify additional information about the extra column.
Specify a short (16 char or less, using letters, digits, or underscores only) abbreviation for the
selected data, its more descriptive name, the data unit, the data type 1-character code (\fBb\fPyte, \fBs\fPhort,
\fBf\fPloat, \fBi\fPnt, \fBd\fPouble, or \fBt\fPext) you want used for storage in the netCDF file,
any scale and offset we should apply to the data to make them fit inside the range implied by the chosen
storage type, and a general comment (< 128 characters) regarding what these data represent.  Note:
If text data type is selected then the terms "values" in the \fB\-A\fP discussion refer to your
text data.  Furthermore, the discussion on interpolation does not apply and the NaN value becomes
a "no string" value (see \fB\-E\fP for what this is).  Place quotes around terms with more than
one word (e.g., "Corrected Depth").
.TP
.B \-N
Specify the distance unit used when using \fB\-Ad\fP|\fBD\fP by appending \fBe\fP (meter), \fBk\fP (km),
\fBm\fP (miles), or \fBn\fP (nautical miles). [Default is \fB\-Nk\fP (km)].
.TP
.B \-Q
Quick mode, use bilinear rather than bicubic interpolation [Default].  Optionally, append \fIvalue\fP in the
0 <= \fIvalue\fP <= 1 range.  This parameter controls how close to nodes with NaN values the
interpolation will go.  E.g., a \fIvalue\fP of 0.5 will interpolate about 1/2-way from a non-NaN to a
NaN node, whereas 0.1 will go about 90% of the way, etc. [Default is 1, which means none of the four nearby
nodes may be NaN].  A \fIvalue\fP of 0 will just return the value of the nearest node instead of interpolating.
Only relevant when \fB\-Ag\fP|\fBi\fP is selected.  
.TP
.B \-V
Selects verbose mode, which will send progress reports to stderr [Default runs "silently"].
.TP
.B \-bi
Selects binary input.  Append \fBs\fP for single precision [Default is \fBd\fP (double)].
Uppercase \fBS\fP (or \fBD\fP) will force byte-swapping.
This applies to the input 1- or 2-column data files specified under some of the \fB\-A\fP options.
The binary input option is only available for numerical data columns.
.SH EXAMPLES
To append Geosat/ERS-1 gravity version 11.2 as an extra data column in the cruises
01010047.nc and 01010008.nc, storing the values as mGal*10 in a 2-byte short integer, try
.br
.sp
\fBmgd77manage\fP 01010047 01010008 \fB\-Ai\fP10/1/grav.11.2.img \fB\-I\fPsatgrav/"Geosat/ERS-1 gravity"/"mGal"/s/10/0/"Sandwell/Smith version 11.2" \fB\-V\fP
.br
.sp
To append a filtered version of magnetics as an extra data column of type float for the cruise 01010047.nc, and
interpolate the filtered data at the times given in the MGD77+ file, try
.br
.sp
\fBmgd77manage\fP 01010047 \fB\-AT\fPmymag.tm \fB\-I\fPfiltmag/"Intermediate-wavelength magnetic residuals"/"nTesla"/f/1/0/"Useful for looking for isochrons" \fB\-V\fP
.br
.sp
To delete the existing extra columns satfaa, coastdist, and satvgg from all MGD77+ files, try
.br
.sp
\fBmgd77manage\fP `cat allmgd77.lis` \fB\-D\fPsatfaa,coastdist,satvgg \fB\-V\fP
.br
.sp
To create a 4-byte float column with the correct IGRF reference field in all MGD77+ files, try
.br
.sp
\fBmgd77manage\fP `cat allmgd77.lis` \fB\-Acm \-I\fPigrf/"IGRF reference field"/"nTesla"/f/1/0/"IGRF version 10 for 1990-2010" \fB\-V\fP
.br
.sp
.SH DISCUSSION
.sp
\fB1. Preamble\fP
.br
The mgd77 supplement is an attempt to (1) improve on the limited functionality of the
existing mgg supplement, (2) incorporate some of the ideas from Scripps' gmt+ supplement\"'
by allowing extra data columns, and
(3) add new capabilities for managing marine geophysical trackline data stored in an
architecture-independent CF-1.0- and COARDS-compliant netCDF file format.  Here are some of the
underlying ideas and steps you need to take to maintain your files.
.br
.sp
\fB2. Introduction\fP
.br
Our starting point is the MGD77 ASCII data files distributed from NGDC on CD-ROMS, DVD-ROMS,
and via FTP.  Using Geodas to install the files locally we choose the "Carter corrected depth"
option which will fill in the depth column using the two-way traveltimes and the Carter tables
if twt is present.  This step yields ~5000 individual cruise files.  Place these in one or
more sub-directories of your choice, list these sub-directories (one per line) in the file
mgd77_paths.txt, and place that file in the directory pointed to by \fB$MGD77_HOME\fP; if not set
this variable defaults to \fB$GMTHOME\fP/share/mgd77.
.br
.sp
\fB3. Conversion\fP
.br
Convert the ASCII MGD77 files to the new netCDF MGD77+ format using \fBmgd77convert\fP.
Typically, you will make a list of all the cruises to be converted (with or without
extension), and you then run
.br
.sp
	\fBmgd77convert \-Fa \-Tc \-V \-Lwe+\fP `cat cruises.lis` > log.txt
.br
.sp
The verbose settings will ensure that all problems found during conversion will be reported.
The new *.nc files may also be placed in one or more separate sub-directories and these
should also be listed in the mgd77_paths.txt file.  We suggest you place the directories
with *.nc files ahead of the *.mgd77 directories.  When you later want to limit a search
to files of a certain extension you should use the \fB\-I\fP option.
.br
.sp
\fB4. Adding new columns\fP
.br
\fBmgd77manage\fP will allow you to add additional data columns to your *.nc files.  These
can be anything, including text strings, but most likely are numerical values sampled along
the track from a supplied grid or an existing column that have been filtered or manipulated
for a particular purpose.  The format supports up to 32 such extra columns.  See this man
page for how to add columns.  You may later decide to remove some of these columns or update
the data associated with a certain column.  Data extraction tools such as \fBmgd77list\fP
can be used to extract a mix of standard MGD77 columns (navigation, time, and the usual
geophysical observations) and your custom columns.
.br
.sp
\fB5. Error sources\fP
.br
Before we discuss how to correct errors we will first list the different classes of errors
associated with MGD77 data: (1) Header record errors occur when some of the information fields
in the header do not comply with the MGD77 specification or required information is missing.
\fBmgd77convert\fP will list these errors when the extended verbose setting is selected.
These errors typically do not affect the data and are instead errors in the \fImeta-data\fP
(2) Fixed systematic errors occur when a particular data column, despite the MGD77
specification, has been encoded incorrectly.  This usually means the data will be off by
a constant factor such as 10 or 0.1, or in some cases even 1.8288 which converts fathoms to
meters.  (3) Unknown systematic errors occur when the instrument that recorded the data or
the processing that followed introduced signals that appear to be systematic functions of
time along track, latitude, heading, or some other combination of terms that have a physical
or logical explanation.  These terms may sometimes be resolved by data analysis techniques
such as along-track and across-track investigations, and will result in correction terms
that when applied to the data will remove these unwanted signals in an optimal way.  Because
these correction terms may change when new data are considered in their determination, such
corrections are considered to be ephemeral. (4) Individual data points or sequences of data may
violate rules such as being outside of possible ranges or in other ways violate sanity.
Furthermore, sequences of points that may be within valid ranges may give rise to data
gradients that are unreasonable.  The status of every point can therefore be determined and
this gives rise to bitflags GOOD or BAD.  Our policy is that error sources 1, 2, and 4 will
be corrected by suppliying the information as meta-data in the relevant *.nc files, whereas
the corrections for error source 3 (because they will constantly be improved) will be
maintained in a separate list of corrections.
.br
.sp
\fB6. Finding errors\fP
.br
The \fBmgd77sniffer\fP is a tool that does a thorough along-track sanity check of the
original MGD77 ASCII files and produces a corresponding *.e77 error log.  All
problems found are encoded in the error log, and recommended fixed correction terms
are given, if needed.  An analyst may verify that the suggested corrections are indeed
valid (we only want to correct truly obvious unit errors), edit these error logs and
modify such correction terms and activate them by changing the relevant code key (see
\fBmgd77sniffer\fP for more details).
\fBmgd77manage\fP can ingest these error logs and (1) correct bad header records given
the suggestions in the log, (2) insert fixed correction terms to be used when reading
certain columns, and (3) insert the bit-flags found.  Rerun this step if you later find
other problems as all flags will be recreated.
.br
.sp
\fB7. Error corrections\fP
.br
The extraction program \fBmgd77list\fP allows for corrections to be applied on-the-fly
when data are requested.  First, data with BAD bitflags are suppressed.  Second, data with
fixed systematic correction terms are corrected accordingly.  Third, data with ephemeral
correction terms will have those corrections applied (if a correction table is supplied).
All of these steps require the presence of the relevant meta-data and all can be overruled
by the user. In addition, users may add their own bitflags as separate data columns and use
\fBmgd77list\fP's logical tests to further dictate which data are suppressed from output.\"'
.SH CREDITS
The IGRF calculations are based on a Fortran program written by Susan Macmillan,
British Geological Survey, translated to C via f2c by Joaquim Luis, and adapted to
GMT style by Paul Wessel.
.SH "SEE ALSO"
.IR mgd77convert (l),
.IR mgd77list (l),
.IR mgd77info (l),
.IR mgd77sniffer (l)
.IR mgd77track (l)
.IR x2sys_init (l)
.SH REFERENCES
Wessel, P., and W. H. F. Smith, 2005, The Generic Mapping Tools (GMT) version
4.1 Technical Reference & Cookbook, SOEST/NOAA.
.br
Wessel, P., and W. H. F. Smith, 1998, New, Improved Version of Generic Mapping
Tools Released, EOS Trans., AGU, 79 (47), p. 579.
.br
Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Mapping Tools
Released, EOS Trans., AGU, 76 (33), p. 329.
.br
Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Mapping Tools
Released, http://www.agu.org/eos_elec/95154e.html, Copyright 1995 by the
American Geophysical Union.
.br
Wessel, P., and W. H. F. Smith, 1991, Free Software Helps Map and Display Data,
EOS Trans., AGU, 72 (41), p. 441.
.br
The Marine Geophysical Data Exchange Format - "MGD77", see
\fIhttp://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt\fP
.br
IGRF, see \fIhttp://www.ngdc.noaa.gov/IAGA/vmod/igrf.html\fP
.br
