This link saves the day again for python and NetCDF:
http://www-md.fsl.noaa.gov/eft/developer/netCDFPythonInterface.html
12/07 :
- IOOS MET teleconference to be set soon.
- Isite metadata server now up and operational, just need metadata to serve.
This represents a nice step forward to interoperability between metadata and
data.
- Created 3rd type of xml file for weather information to enable more
mouse over control.
12/05 :
- IOOS Archive-ET(AET)/Metadata-ET(MET): Teams need to pass on comments
and consult with each other. MET should be having a teleconference in the
next 14 days.
12/04 :
- Creating more NetCDF intermediate files which are available from
our OPeNDAP servers. This will feed the interactive and GoogleMap
subsystems as to cut down the demands on the relational database.
- A lot of email went out late Friday and Saturday. I expect to
be innundated with replies on Monday. Maybe setting the stage for
some solid standards or some initial testbeds.
- Working with XML, AJAX: http://www.webreference.com/programming/javascript/jf/column14/
http://www.webreference.com/programming/javascript/jf/column13/2.html
- Cool google tricks:
http://bb.domaindlx.com/geocodezip/prosptest_map3.asp
Links to add to documentation
- http://www.druid.net/pygresql/
- http://en.wikibooks.org/wiki/Programming:Python
- Standards: http://www.cgd.ucar.edu/pipermail/cf-metadata/2005/000549.html
12/03 :
- Changed map extents for our statewide mapscript page to epsg:102006
- Developed a utility to help compute
map extents. The
PHP and python script are available via our SVN code archive.
- Playing a bit with ka-Map for mapserver.
Great Lakes Commission is also playing with it.
- ka-map is now way behind GoogleMaps.
- Have to investigate more into:
12/02 :
- Status report to AOOS DMAC
- Light implementation of Google maps. It is a slick interface
but does not have the capabilities of Mapserver/Mapscript.
11/23 :
- Our database server is starting to get into trouble with the
number of simultanious connections. If a connection fails, it will
try again every 20 seconds, up to 2 minutes.
11/22 :
- DMAC presentation to Governance Board :
2005/11/21
- SHEF coded data is now available on the website.
- The database system is virtually maxed out now. Starting to see
problems with connecting to the database.
- Better coordination on scripting may help database connectivity.
Maybe finding ways to wrap the database service in a SOAP server which
then accesses the database via a local connection.
- Version of 8.1 Postgresql should have some performance improvements.
- New opendap servers and handlers have arrived. We have implemented
the server. We have implemented the netcdf, hdf4 and freeform servers.
We also were able to implement the relational database handler.
- We are very close to a solution for NASA/GSFC to open up the
OceanColor database via OpenDAP. The opendap group will be addressing
two concerns for the next version of the server and hdf4 handler.
- Started working with FGDC/Isite server software.
- Need to hook up with Jerome @ PaCOOS. He has information on
how to successfully spin up LAS and THREDDS services.
- We obtained a lot of information via email on how to hook up to
the openioos.org system... needs digestion
10/21 :
- Lots of items to report on. Been over a month since the last entry.
- AOOS servers have now been relocated twice.
- AOOS servers now have assigned IP space in the UAF DMZ server zone.
- Legacy names daved.ims.uaf.edu and penguin.sfos.uaf.edu do not function
at present. This will impact the SVN server services and lots of other services.
Data collection should not be impacted.
- New datasets: Codar, ice extent, snotel/AMBCS weather stations.
- New collections: NWS ANC/Cordova Boat and CTD/ADCP from NBDC.
- SHEF and CMAN information is also being archived at present. Once the
decoder/ingestion routine is built, all this information should be online.
- Slight progress on spinning up an OBIS node.
- Preparation on travel to back to back meetings.
09/09 :
- CMAN decoder built and put into beta testing. Code has been checked into
Subversion for download. Decoder is currently decoding information. Next
step is to build the python script, synccman, to pull data from the netCDF
file into the AOOS database system.
- Extended data capture range: 40N to 90N by +170E to -110W
09/07 :
- ADCP and ocean parameters at 2 meters (sal, temp, cond) are
now being pulled into the AOOS database for buoy 46060.
- Buoy data via the IDD are now automatically added to the
AOOS database when buoy2nc closes a NetCDF due to being
inactive for 5 minutes. That same action calls the syncbouy
python script to sync the netcdf file to the database.
The Unidata script buoy2nc was modified to allow more than
one observation per hour. Version 3.1.2 of the script currently
only allows one observation per hour to be stored. Patch was
forwarded to Unidata.
- We are currently writing a cman2nc script (child of buoy2nc)
that will automatically decode CMAN stations for AOOS
- SHEF coded messages are on the horizon adding real time access
to HADS and RAWS stations. We have reason to believe some AK SnoTEL
stations are included in this datafeed.
- A metadata solution has been formulated in the past 24 hours.
Now to find some programming time to implement FGDC basic templates
so they exist in a database and are easily accessible to Z39.50
national aggregators, can be added or updated by AOOS data providers,
and put to use in real time QA/QC control in OOS operations. Need to
look at the MetaDoor database template.
09/06 : Sample ADCP import into the database today. Will add salinity
oxygen, etc to the table shortly. Identified key ingrediants for
metadata. These will have to do until the ISO/FGDC standards merge and
roll out. Will work on getting the Isite Z39.50 server up and running
and begin rolling some metadata out to the general public. This is
good news for the OBIS folks waiting for metadata progress. How the
FGDC standards match up with the OBIS standards is yet to be understood.
09/04 : Hurricane Katrina took out buoy data for AOOS. We are
working to restore the feed via the Internet Data Distribution
system (IDD). The decoder script buoy2nc has been fixed up.
The synchronization script syncbuoy is stable. All that needs
to happen now is getting it automated. Next: SHEF decoder to
read SHEF, NWLON, CMAN, and other goodies from the IDD stream.
This should bring online a couple SNOtel stations automatically.
The remaining items will need to be screen scraped.
08/21 : Not much visible progress. A major infrastructure
component is being addressed; metadata. Top to bottom technology
has to be build to support metadata for the lifetime of datasets
at AOOS. This will have far reaching impacts on all parts of the
display and delivery system and guidance we give our data
providers. Plans this week include continuting metadata research
and adding PHP/Mapscript(Mapserver) to the remaining parts of
the AOOS website to complete the initial deployment for the
Statewide and PWS OOS pages.
08/08 : Beginning to conquer rasters. The Alaska Relief raster
finally works. There is definitely something wrong with
shdrlfi020l.tif (be warned).
08/05 : Streamlined some screen scraping routines for NWS METAR
and BUOY collections. Data delays are now about 2 - 3 hours now as
compared to 11 hours.
07/29 : AOOS on Mapscript arrives! Link This represents
a final milestone for the first iteration of GIS mapping for AOOS. There is enough functionality and flexiblity with
Mapscript (based on PHP) to allow customization of data visualization and data delivery and wrapping it into a MyOOS
type system. GoMMaP is an example of a professionally rendered
GIS system as developed by DM Solutions Group Inc.. The template that we
are going to use is called GMap75. It is a demo available from
this site.
07/21 : MapServer comes alive @ AOOS! PostGIS, shapefiles and CONTOURING of data! This is a MAJOR milestone for
data delivery technology for AOOS. DEMO is here.
07/11 : Updates to status page. Specific needs as addressed by the PWS meeting, generic punch list
was put together.
05/18 : Working with QuikScat level 3 data. Two passes per day of data.
QuikScat, Codar and sea level pressure fields from model output will be
the centerpiece of the OGC/MapServer display.
05/14 : AOOS Data Management officially begins. Next stage of
development to take place. June 1 deadline approaches for display of
PWS products.
05/13 :
- SFOS database moved to modern equipment. Not all facilties are operational. The main facility (table) that was restored is the IMS
media table. This contains all the tape, CD, DVD and other media locations within the department.
- Search for bathymetry came up with this URL, http://www.ngdc.noaa.gov/mgg/coastal/coastal.html. Will include
Alaska with 3-arc second resolution. Will contact David.Divins@noaa.gov to see if they have any sections ready, like PWS that we can get immediately.
- New AOOS office is nearly setup. New phone number: 907-474-7948
- Booked tickets, etc for PWS workshop.
05/12 : Otina Fox interviewed for AOOS web designer job.
04/26 : AOOS Data Management interviews completed. A tentative start
date of May 15. Waiting for Human Resources to post the web
visualization position.
04/20 :
- First set of retrospective data sent to SeaCoos. The time period July 27, 2004
to August 4, 2004. Display of CODAR movies will be possible once this data is installed.
- JPL QuikScat dataset sample ready for SeaCoos.
- Guts of initial workstations have arrived. Second batch of parts have arrived.
04/08 : Live Access Server returns to service at AOOS. As of today, it has
two basic datasets: the COADS climatology and Levitus climatology
04/07 : Officially applied as data manager for AOOS
04/05 :
- CDMP tape recovery project launch anticipated in the next couple of days upon approval of a task order.
- Job posting for AOOS System Manager is now posted.
- Completed first reading of Part I of the IOOS DMAC Plan. Current AOOS DMAC
technical implementation plans for data delivery are well within the IOOS DMAC recommendations.
- Will now have to reconsider the notes from the recent AOOS DMAC meeting and
a report that will was recently furnished by Molly. For additional guidance and
planning purposes. Are our current plans meeting everyones goals?
03/25 :
- Website for development of AOOS technical materials is now online. This
is NOT to replace general web content (policy, news, annoucements or other web)
content contained on http://www.aoos.org.
- Conversion of NWS surface, buoy and upper air data to NetCDF is now happening
on a regular basis. Needed by a few users that might be impacted by a NWS move
to close down a legacy data delivery service.
- Ready for regular delivery of data from AEFF (PWS and Cook Inlet model output).
- Allen Macklin sent email letting everyone know of updated
IOOS DMAC documents (March 2005).
This document will be examined and a response will be posted to the AOOS
website. Initial indications that our current strategy is 80% compliant.
02/24 - 03/19 : QARTOD II and vacation ate up most of this time
02/23 :
- Nearly completed the PWS HF Radar import to SEACOOS. Looks like we
need to define our demonstration period and get all data possible for
that period into SEACOOS.
- Lisa Baraff has been collecting Alaskan data sources.
- I'm proceeding with a basic GMT/data delivery system while I
find answers to LAS and MapServer. This is ready for a GUI to
set on front of the data collection scripts. Allow users to
select: date, time, parameters and get a visual and data.
- Vacation plans are back to the original dates.
- Setting up requests to get model data from Peter.
- Investigation of ADCP data from Alaska buoy stations.
02/18 :
- Fast track statewide/PWS display system will be done adhoc through
SeaCOOS folks
by populating data for the predefined region of Alaska. This will enable
a real live mock up of what we hope AOOS can bring to customers at the
statewide and local levels. This accomplishes a lot of tasks.
- A big thank you to Charlton and SeaCOOS!
- Eases up immediate delivery of a DM Solutions software package for AOOS -- allows for better planning
- A mock up plan for statewide and local deployment should happen on time.
- This will allow delvelopment of the AOOS display and delivery platforms.
- Data has been identified, I will be performing most of the work
getting it into netCDF for utilization by SeaCOOS. We will start with
a fixed 48 hour time period. This will enable us to complete a full
mock up for demonstration purposes and a target for development.
- Vacation plans are delayed by one week.
02/17 :
02/15 :
- Try and bring JPL QuikScat winds to the Alaska region.
- Develop task orders for the CDMP tape recovery grant to recover data from 2000+ tapes
stored at SFOS (due: end of Feburary 2005)
02/14 :
- Continuing work on pulling statewide datasets together. I have tapped the buoy
information in our database to produce a sea level pressure map. The
basic display tool is GMT and is the same tool Charlton uses to bring contours to the SEACOOS
MapServer. How that is done is still a large mystery. I will focus on data collection and
basic representation. I'm hoping our efforts with DM Soultions will make this connection.
- Molly forwarded a data source template to me. The only real time data source that I
can place in AOOS for now is the HF Radar project operating in PWS. I will get with Hank
to collect information for this form and return it to Molly.
02/10 : Spent a hour of time trying to flesh out a DMAC Manual outline of sorts. Not
sure of the full ramifications of this topic. Setup a LAS test server on a very slow
machine. Hope to get some operational machines in the near future.
02/09 : Voice conference with DM Solutions on a MapServer solution. I have recently
been added to the LAS development team. Effort to speed up development of the LAS system
for needs with OOS and specifically AOOS. Indications that LAS is moving towards including
OGC type communication.
02/02 : Found a technical meeting that I will need to attend.
MapServer User Meeting : June 16-18, 2005
(http://mapserver.gis.umn.edu/mum/mtg2005.html).
Completed reviewing tutorial on Chameleon
(http://chameleon.maptools.org/help/viewer.phtml).
Need to add some basic basemaps for Alaska and start display of data. This has to be done in
the next few weeks.
02/01 : Was given access to Gina's development machine to begin exploration of
MapServer for AOOS. Tom and Jason have been very accomodating. This skips an
installation step that will have to be revisited later.
01/31 : Tom Heinrichs made contact with DM Solutions Group to see about
an OOS software development kit (SDK) which would contain the majority of the
SEACOOS framework. Introduction to Jason Grimes, the primary developer of the
MapService for GINA.
01/27 - 01/28 : Get sped up on AOOS DMAC and PWS pilot project.
Various protocols and existing software was examined. Preference to keep to
open source side of things was expressed. Though quick development was cited
using ESRI's various ARC products. Specific action plan was formulated.
