Tech Staff Meeting Notes September 13, 2010

The top priority is always repair of down stations.
NM problem stations
ET problem stations

Duty Table


Priorities

  • Repair down stations and routine maintenance
  • broadband ARRA upgrades

  • News & Highlights

    NetOpsV will be focused on AQMS on November 3 and 4 in Menlo Park. It will likely be delayed until February, 2011.

    R&O Meeting this afternoon at 3 in the seminar room.

    See Healthcare email from Zettergren for important changes to insurance.

    Footware is required at CERI. In lab and field environments, footwear must include protection for the toes and top of the foot.


    Starting in October, we're going to have a show and tell each month. It will be nominaly scheduled for 1:30 the afternoon of the first monday of each month subject to change based on the needs of fieldwork. The first one will be a CMG5TD/MSS100 show an tell by Jim on October 11. The one for November will be getting data and viewing with pql and will be held on November 8 (Mitch is out of town the first week of November). Other potential topics include: Shaketable, discriminator alignment, CMG6TD, HowTo K2, Computer room power and generators, node resets dbb and other checks, Instrument response and metadata, ANSS performance goals, ANSS self assessment, GPS receivers, FDL tour.

    We're going to add the name of the person responible for each entry in the NMDown and ETDown lists. Note that everyone is still needed to keep things going but we need someone who is keeping track of each problem.

    David gave an update on what went on at GFM. The tree is on its last legs but is still pretty stable. If we choose GFM for an S-13/RT130 upgrade, and assuming we could get forest service permission and a decent place to land, we could probably get it all up there in two trips with a helicopter for a couple hundred bucks an hour.

    NMSZ

    Short Period & Broadbands

    ISIS routine maintenance & service 2010:
    14 of the 25 New Madrid node stations have been visited and 6 canisters were rebuilt and reinstalled.
    10 of the 24 Lenox stations have been visited and 6 canisters were rebuilt and reinstalled.
    4 of the 21 Marked Tree stations have been visited and 6 canisters were rebuilt and reinstalled.

    Dynamic NMDown list

    Down
    PARM/Jim
    PEBM/Jim (bad timing but status stream says its locked and good to 4us)
    NHIN/Dave (bad time in adsend, need new xlak)
    JHAR/Dave
    MPH.LL*/Jim

    Strongmo Down
    HDAR/Chris comm issues between mud island and the weigh station?
    RDST/Chris no contact with mss. power reset didn't fix it.
    CBHT/Chris mss is connected, no contact with cmg5
    MCAR/Chris down most of the time. Occasional brief periods of connectivity.
    The staff meeting notes show it needs a router and mss100 upgrade.

    Muted Amplitude
    BLAR.EHE/Dave
    BLAR.EHN/Dave
    LVAR.EHN/Dave
    STAM.EHE/Dave
    WADM.EHN/Dave
    BETM.EHN/Dave
    WALK.EHN/Dave
    DWDM.EHN/Dave
    NHIN.EHN/Dave

    Drifting
    HICK.HHE/Dave - see image
    PARM.HHN/Dave

    Gain Channel Problems
    CHRM/Dave
    MORT/Dave
    FLPT/Dave

    Isis Oxidation
    JHAR.EHE/Dave
    TMAR.EHE/Dave -see image

    Unknown problem
    CATM.EHE/Jim - see image
    TOPM/Jim -see image
    MLDM/Jim - see image
    HALT/Jim - see image
    HENM/Jim has a strange "vibration" on the vertical only (do these sensors somehow know they're headed for the graveyard).

    NODE Tasks

    Node reset instructions are at /gaia/smeagol/home/seisadm/ew/node.notes/phone.reset.proc
    Note that using the pc reset option with the GX620 system can cause it to hang. Only power cycles are recommended. The reset option has been disabled at most nodes.

    MKTA

    The phone line is recabled temporarily and the reset is reenabled.

    DSL is now installed but the networking is slightly different from lnxt. Steve is working out the problems making both the dsl and the dbb simultaneously accessible.

    Construction is ongoing at the mill.

    LNXT:

    Chris is checking on the cost of a service contract for the generator.

    NMAD:

    Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).

    AZTN: Replace XLAK clock.

    CERI:

    The DBB firewall installed and operating. Only specific ip's are allowed through. You need to use the route command to manually add a route. Evertying behind the firewall is the same D ip number but with 10.156.1 instead of 141.225.156. Remaining items include:


    ANSS SM

    Ownership, Operation, and Maintenance of the four Charleston, SC stations (C1SC, C2SC, ADSC, and TRSC) will be turned over the Univ. of SC at Columbia possibly by September 1. John will need to give Phillip Crotwell a show and tell at each station. Data flow will need to be redirected to Columbia (firewalls and etc taken care of). Then those feeds will get added to the CO network export to CERI. The stations will keep the same name but the network code will change (from NM to CO).

    Year3 upgrades
    stationyr3 hardwareroutermss100
    RDSTYYN
    MCARYNN
    CUETN?Y
    MKARYYY
    CBHTNNN
    COLTNNN
    GILTNNN
    NAITNNN
    NMEMNNN
    TUMTNNN

    Replace foam rubber plugs at all year 1 stations (need to notch it to accomodate the GPS pods): ASAR, CBHT, CUET, CVTN, HDBT, LNXT, MCAR, NAIT, SFTN & TUMT.

    NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.

    CVVA is upgraded but there remains an unresolved issue with the firewall Steve is working with Merritt Reagan NGC to get them resolved. Merritt says their it security has been slow to respond but they're still working on this.

    Al and Jim have all the issues resolved and we're now able to download 200 sps triggers. These are the guidelines.

    HDBR

    The aztnworm upgrade broke the disk sharing with the win98 box. Mitch sent a budget and justification to Sharam. This project needs to be either upgraded to replace the substandard PMD dataloggers, or it needs to be decommissioned (see HDBR notes from ).

    Recorder #5 has a bad disk bay.
    No contact with #7

    HDAR

    Bad CMG5 E/W.

    Not getting broadband feed.

    HDBT

    Replace two bad mss100's.

    Bad 100' accelerometer.


    ETSZ

    Dynamic ETDown list

    Down
    TVNC/John
    GRBT/John- sporadic
    CCNC/John- see image
    TRYN/John-reinstall after tower is rebuilt.

    StrongMo Down
    ADSC/Steve changing ip issues?
    CVVA/Steve firewall issues?
    ATTN/John

    Drifting
    ETT.EHG/John - same as before see image

    Gain Channel Problems
    SMNC/John - same as before see image

    Unknown
    RBNC/John - interference
    VHTN-EHN/John - Same as before w/ new image
    WSNC/John - see image
    CMGA
    CSTN/John got a new lantronix card and is working now, but is very gappy

    FPAL is about 10 degrees warmer than the Arkansas stations so some shade or forced ventilation may be in order.

    Mitch needs to get off his butt and select the 3 non-ARRA ET stations for upgrade.

    STAR Node

    BHT. Chris reports that it is time to seriously pursue another site as access is even difficult by 4-wheeler. David reports moving it will take all the fun out it, what's the worst that could happen? John said that folks at Cumberland Trail State Park said we can put the station at another spot on Black Mtn. which is just a couple of miles NE of the present location. It is accessible by a paved road. We can pursue that or I can look for something that will put it further NE to get it more equidistant between DYTN and WMTN. Right now it is awfully close to DYTN and BCRT. Mitch likes the idea of splitting the difference between DYTN and WMTN. John will dig out his old notes and starting looking around.

    BRBC Node

    Power upgrades: TVNC, GFM (done), RBNC, and TRYN.

    GFM The NC state park service is purchasing GFM. We already have a couple permissions from them and John initiated the paperwork for the GFM permission.

    TRYN: Hardware is removed from the tower and will be reinstalled when the new one is complete.

    There seems to be a strange, repeating signal on many of the stations, most notably on WSNC/RBNC. It repeats about every 14 minutes and lasts about 1 sec. Example data are in /gaia/home/mwithers/public_html/STAFF/Analog/2010/20100910_051500_MAN. Click this picture of it on CCNC. This from Jim:

    The 5 sec. noise bursts appear simultaneously approx. every 14 minutes on the stations ( RBNC, WSNC, MGNC & CCNC) from receive site #2. We should check the phone cable shield ground in the receiver cabinet and rack. Also look at the Balun/TSP card in the rack. It might be RF interference coming in via the receivers?

    MTTN Node

    Greentop is getting a new tower too. There will not be internet up at the mountain, so we need to find a place to transmit to. We should be able to hit just about anything in Knoxville, Maryville or Pigeon Forge; is there a preference?

    A new prof at UNC-CH, Lara Wagner, might be interested in co-locating some of here equipment (Nanometrics broadbands & digitizers), with as at VHTN or LRVA. She is setting up a transect array of 10 stations in western NC.


    GPS

    James has a very nice SOH website at http://www.ceri.memphis.edu/people/jpdavis/gama/results.html.

    PTGV was losing satellites indiscriminately. There another sensor package transmitting right at the gps stations that could be causing interference. The hosts assured James that they aren't the source of the problem though is seems to be clearing up as their experiement completes and more of their sensors go offline.

    Most monuments need a new paint job and weed control. Mineral Area Colege is the top priority.

    MAIR is currently wireless. We have a cat5 surge arrestor and permission to run the cable into the office (less than 50' though it does need a hole in the wall).

    There may be an issue with the NetR8 stopping satellite tracking at temperatures below -5C. Ain't that special. Presumably those in the west are applying heat and there will be a fix forthcoming from trimble.

    Need to replace older GPS-100 supply systems.

    These stations still need to be steinerized: CVMS, MACC, HCES, MAIR, NWCC.

    STLE, NMKM, and RLAP all need a charger upgrade. They are currently working (and have been for a long time) so this is not necessarily a top priority.


    AG Network

    vent all cabinets.

    Install network WatchDog Timers. Meremonte uses iboot. Is he talking about the ones from dataprobe? http://dataprobe.com/iboot-remote-reboot.php


    ARRA Upgrades

    Rough-ins are complete for all NM broadbands except LPAR and GNAR.

    GNAR needs a custom cabinet on stilts. Can also remove the computer and put both GSAR and GNAR behind a router pulled directly into scream and rtpd at ceri.

    Near-term tentative schedule:

    1. LPAR roughin, 9/14-15
    2. GNAR complete week of 9/20
    3. mkta node week of 10/4
    4. lnxt node week of 10/18 ??
    5. nmad node week of 10/25 ??
    6. CPCT week of 11/1 ??

    SWET is done.

    Need to test rtpd on wintel. Scream/scream2ew will be replaced with rtpd/reft2ew.

    We are approved for the following in approximately the same order as we hope to complete them:

    1. Upgrade 2 ET broadband stations with new 6-c DAS, new sensor, and accelerometer (CPCT and SWET).
    2. Upgrade 11 NM broadband stations with new 6-c DAS, new sensor, and accelerometer.
    3. Upgrade 3 ET shortperiod analog stations with new 6-c das, digital telemetry, and accelerometer (BCRT, GMG, and GTTN).

    Note that using other funds, we're also upgrading 3 additional ET stations to digital with episensor and FPAL to the same hardware as the other broadbands.

    Mitch hung a chart in the war room to track our progress (we're required to make quarterly progress reports and to meet the proposed schedule).

    We need to be very careful to keep AG, ARRA, and CERI hardware separate and be certain that hardware purchased by each project goes only to that project.

    All replaced DM24's and CMG40T's are to go to south campus and taken off inventory. This hardware is not to be redeployed anywhere in the CERI seismic network nor FDL. Mitch is checking whether they can be donated to Bob's Argentinian colleagues.


    FDL

    Guralp is in Tigerbuy so we'll be able to pay them for the repair of the two CMG6TD's.


    Computer Room Power

    Need to review power loads and line balance after adding the new AQMS servers. Review needs to consider addition of new landmark servers, mac server, and seisnet deep waveform buffers. The Earth Science Library (ESL) in 3892 is moving to the main branch and has a liebert. We should consider carving out rack space in that room (though backup power is an issue). Mitch will set up a meeting with Jim B., Steve B., and Bob D. We will have to move into an era of not having all servers (e.g. landmark servers) on the generator.


    NSMP

    We're now officially providing maintenance support for NSMP stations in the NMSZ area. There are 22 stations. SLU has 11 to the north and CERI has 11 in AR and TN. They are all dial-up K2 or Etna. Data continue to go to NSMP in Menlo who will continue to monitor SOH. They will email or call when there is a problem. We're the boots on the ground.

    NSMP Station List.

    Due to remodeling, we pulled K2 1738 from Firestation #2 in Dyersburg, TN in February 2009. It is in the shop with accessories.

    We pulled the k2 at Obion (station 2460) because the building it was located in is being torn down.


    Metadata

    The neic has a new metadata website at https://geohazards.usgs.gov/smds/index.php?message=1. John is finished populating it with info from our stations.

    ISTI was contracted by USGS to support INV. Mitch is working with Sid to get initial dataless SEED going. John will be our INV czar. (though now I hear rumblings that there may be a different ANSS supported tool for this).


    MISCELLANEOUS

    The seisnet group on umdrive is at http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password.

    Rotational sensor is feeding an earthworm but still has IRIG crosstalk problem to fix. Its fine on an Oscope hooked to the output of the mux and looks fine in LabView. Its bad when viewed with wave_viewer. Need to see what the sac files look like to make sure it isn't just a wave_viewer feature.

    Chris continues to hound the supply to try to get Parker's old Suburban onto govdeals.

    The pink palace display now includes a heli connection for MPHB. Need to retrieve any remaining hardare (e.g. antenna and reciever on roof).


    CERI

    3890 Power Chris is going to make a diagram for 3890. Good luck son.

    3876 Power Chris is going to make a diagram for 3876. Good luck son.

    Should we have a trouble ticket system? Yes, all we need is someone with enough time to decide which to use, acquire it, install it, learn it, and train everyone else how to use it. O.T. Guy won the nomination.