Data access of TLALOCNet GPS-Met data via Web Services API queries.
The TLALOCNet is a GPS-Met network in Mexico for the interrogation of the earthquake cycle, tectonic processes, atmospheric processes, land subsidence and space weather. All the data generated by this network is freely and openly available at the TLALOCNet data archive (http://tlalocnet.udg.mx). The archive has a simple, intuitive interfase to query for specific GPS and met data files. The archive holdings include GPS observational, navigational and meteorological data as Rinex formatted files (e.g. http://en.wikipedia.org/wiki/RINEX). This is fine for the casual data user. However, there may be situations when the GUI interfase is time consuming (too many clicks) or when we are seeking a fully automated routine operation.
This is a summary of a few command line search examples that can be used to query the TLALOCNet archive (which is running Datworks/GSAC, the engine developed by Unavco that runs behind the database for the archive), and which I am sure can be useful in your daily GPS-Met processing operations. While an FTP script for files with specific past dates would also work, this is a far more elegant approach while using web services from the data archive. This method also has the great advantage that fixes gaps in downloads due to periodic manual downloads and ingestion on the archive at unspecified intervals from stations that are downloaded manually
The TLALOCNet archive that runs Dataworks/GSAC can respond to queries in the form of command-line instructions under the 'curl' command that make easy detecting recently added/updated sites and its associated data files. These commands are most suitable for automation and processing newly added data files and simplify the administration and book keeping of files available at the archive and which have not been processed yet. Moreover they don't need to be run as a GUI.
The commands are run under the Unix/Linux 'curl' utility (they also run in your Mac under OSX; I suppose there may be an Win option, but I am not familiar with it) and the output response can be formatted in several ways, but perhaps the most useful would be the "file.wget" option of Datworks/GSAC (also available via the web page GUI) which outputs the wget command line+path to fetch each one of the available files on the archive. Simply redirecting to a text file would yield the foundations for an executable script to wget these files.
For example, to search all files from all stations in the archive between 2015-04-27 and 2015-04-28, run:
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=2015-04-27&file.datadate.to=2015-04-28&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget'
If you would like to search as above, but for only one station (e.g. TNAL), you would run this command:
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=2015-04-27&file.datadate.to=2015-04-28&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget&site.code=TNAL'
The following command can be used to search for sites TNAL and TNBA between 2015-04-27 and 2015-04-28:
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=2015-04-27&file.datadate.to=2015-04-28&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget&site.code=TNAL;TNBA'
In this case the command output is directed to your std output as:
wget ftp://tlalocnet.udg.mx/rinex/obs/2015/117/tnal1170.15d.Z
wget ftp://tlalocnet.udg.mx/rinex/obs/2015/117/tnba1170.15d.Z
wget ftp://tlalocnet.udg.mx/rinex/obs/2015/118/tnal1180.15d.Z
wget ftp://tlalocnet.udg.mx/rinex/obs/2015/118/tnba1180.15d.Z
the time arguments in the API queries also take "static" dates that allow for scripts that don't need to "update" their date search parameters. The date arguments can be used with the format yyyy-mm-dd, but they also take a generic one e.g., now, -1 week, +3 days, etc (dont forget to add a "+" on the spaces.... I know a bit it is a cumbersome, but see example below).
for example
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=-2+week&file.datadate.to=now&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget&site.code=TNAL;TNBA'
If we redirect the command output to a file instead of your screen, we can have the basics for creating a script for later use and automation. In this case we would have 2 options:
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=-2+week&file.datadate.to=now&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget&site.code=TNAL;TNBA' > TNAL-TNBA_files_from_2weeks_ago.txt
or
curl -o TNAL-TNBA_files_from_2weeks_ago.txt 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.datadate.from=-2+week&file.datadate.to=now&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget&site.code=TNAL;TNBA'
The first one uses a simple output redirection, while the second case uses curl's output option (-o filename)
The two above example queries look for Rinex observational files, for sites TNAL and TNBA, for the past 2 weeks and outputs the search results to a file that can be later run as as an FTP shell script input.
In the example below it is possible to search for all Rinex observational files for all sites in the archive, published (i.e ingested to the archive) between 2015-04-27 and 2015-04-28. This is very useful in cases where stations are not downloaded daily, but instead manually downloaded and ingested on the archive as sets of many daily files.
curl 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.publishdate.from=2015-04-27&file.publishdate.to=2015-04-28&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget'
For those running solutions everyday, the example below will be quite useful because it outputs the wget command line into a text file that contains all files added to the archive in the past day. This is very useful or those cases where sites with manual downloads get ingested in the archive at irregular intervals, and it may be harder otherwise to track those without apriory knowledge of the dates spanned for the files added.
curl -o myFTPscript.txt 'http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/search?file.publishdate.from=-1+day&file.publishdate.to=now&file.sortorder=ascending&limit=5000&file.type=RINEX+observation+file&site.interval=interval.normal&output=file.wget'
Note: Please bear in mind that the archive updates and mirrors its holdings starting ~11:30 pm (local) time and depending on the load and connection speed may carry over the next day. This is particularly important for those non-static queries that for example use "from=-1+day" because depending the time of the day that you run the above command, it may yield slightly different results.
Tip: How to double check your syntax for API new queries?
After a specific search is done via the "Search Files" webpage (http://tlalocnet.udg.mx/tlalocnetgsac/gsacapi/file/form), the result page will present you with a collapsible API search section. When you collapse it (click "+" box), the page will show the current search. This is a very useful way to learn how to build your API search syntax. The line shown on the API can be run with curl within the curl command (and using ' ') as in the examples above.
All the available Dataworks/GSAC different command structure and arguments are always summarized on the "Information" tab of the TLALOCNet archive webpage after you run a query using the web GUI. This tab gives you a command line structure that would yield the same results by using a curl command line.
Hope this is useful to guide for a better interaction with the TLALOCNet data archive.