Tech Staff Meeting Notes January 4, 2012
The top priority is always repair of down stations.
NM problem stations
ET problem stations
January 16 is MLK Day and is a paid university holiday.
NetOpsVI is focused on field ops. It will be in Berkeley, CA March 27-28, 2012. Mitch and David will attend on seisnet funds and Jim and John will apply for USGS travel support.
There are several stations that have a problem with arbitrarily hanging. Mitch sent an email to the netops list but the resulting suggestions didn't really help to solve the problem.
GTTN/CPRT. These are two refteks behind a single router. They use udp to transmit data. The router opens a port when a broadcast is initiated by the reftek. The rtpd on achilles acknowledges back to the reftek on that port. When the refteks creates a new connection for whatever reason, sometimes the router port used by the router isn't the same as what the reftek or rtpd thinks it is. Steve has a "fanless pc" in house with ubuntu installed and he is working on rtpd/rtp2rtpd and on Scream. Jim updated the firmware on the GNAR reftek and its been behaving ever since (even though we were previously at a newer release than what reftek support said was needed to fix this problem). So while we were going to test the fanless pc at GNAR first, we can now go straight to GTTN/CPRT (though new building construction is underway there now).
GNAR. Jim updated the firmware and its been behaving ever since (even though we were previously at a newer release than what reftek support said was needed to fix this problem).
RDST and MCAR are also prone to hang. They're now on gcf2ew along with a script that forces a kill and restart when necessary.
FPAL did the hanging trick too. Mitch did all the resets possible. Tom power cycled everything. Mitch reinstalled updated firmware and still it didn't work. All along we were able to get a direct connection to the reftek via rtcc but not an rtpd connection and acquisition start was on but the rtpd connection was down. Also tried stopping and restarting rtpd. Mitch changed the delay on the stream details button and uploaded the new parameters and it magically started working. I don't think this is the solution to get it working but is instead coincidental. The real problem is using udp for long haul ip comm. Mitch updated the firmware here too and just like GNAR, its been behaving ever since (even though we were previously at a newer release than what reftek support said was needed to fix this problem).
EDIT, NNAR & CHNM: NNAR and CHNM repeat through EDIT and the received signal is very bad due to trees. NNAR and CHRM are very noisy and fade out quite a bit. EDIT is now located in the Chicksaw National Wildlife Preserve so cutting trees is not an option. There isn't a clear path to a repeater on the Edith forest tower either. Need a new plan (maybe go directly to the node?).
WALK: Chris and Manning were unable to reach WALK up in the KY bend. Mr Lynn indicated that the station was completely underwater but it is still up? (yes still up).
Reinstall the rest of HDBT when internet connectivity is established (either the connection to Boswell becomes live, or we bail and move the Boswell dsl to Mud Island).
Search for quieter site for CTCT.
Original list based on NM event 20111229_150610_MAN
Down
ASAR - probably just needs a reset
HCAR
EBZ
PGVM.EHN
Unk. Problem
HDAR (pulled for repair)
PEBM.HHN - drifting
Isis Oxidation
GLST.EHE (rebuilding hardware that came out of LEPT to install at GLST)
Muted Amplitude
BETM.EHN
NHIN.EHE
BLAR.EHN
LRAR.EHE
Weak Battery/Solar
There were 10 stations down for this event. I'm guessing that for a
majority of them it was just the battery.
/gaia/smeagol/dat2/Working/NM/201112/20111224_124800_MAN
BOAR, BVAR, CPAR, HCAR, HTAR, LFRT, LVAR, TNMT, TYAR, and WADM.
DBB
MKTA
LNXT:
Chris will purchase the 1 year term planned service agreement with Cummins.
Steve will replace the lnxtworm computer.
NMAD:
G and D and the local EMA installed a 25kW generator to power that side of the building. If there's room power-wise would they let us power the rack from it too? We could share the maintenance of the generator. We'd also have to install a transfer switch.
Need to install an iboot and discontinue POTS.
AZTN:
CERI:
Twisted pair from Richardson is extremely noisy causing problems with
the analog backbone. Richardson is also on the UM 5-year capital
improvement plan to be replaced. We got a tour of Wilder and its doable.
We need to see if we can get a path to CERI so we know whether we just
need to move what's on Richardson now, or whether we need to turn Wilder
into an entirely new node (i.e. new mudworm). Then we can get Planning
to help us with a design and then we can get permission.
We could just move the firewall from 3904/218 to Wilder and get the DBB
on the campus internet that way (if there's no aironet line of site).
There are good telephone pairs from from the Wilder roof to the basement
and we could move the ceri node/mudworm to the basement. Otherwise we'd
have to find and trust connections between the basement and CERI.
We have line of site from the southeast corner of 3904 except there are powerlines in the way. Can we put up a tower on the east side of 3904 to get over the trees and power lines on central?
Board Shop was decommissioned.
LNXT: We have a Basalt to replace the cmg5td at LNXT. This will require some additional engineering. We need to use rock2slink to transfer data (like the granites). Mitch will determine if we can keep the RS422 and rock2link via serial or whether we have to convert everything to IP.
Ownership, Operation, and Maintenance of the four Charleston, SC stations (C1SC, C2SC, ADSC, and TRSC) was turned over the Univ. of SC at Columbia. At Phillip's request, we letting the data flow through CERI for the time being. Eventually it will get switched to Columbia and the network codes will then change from NM to CO.
RDST, CBHT, and MCAR are coming in via gcf2ew on anssworm. CBHT occassionally loses its connection but is overall, ok. MCAR loses its connection about twice as much so while its mostly functional, it should be better. At RDST, a datapacket sneaks through a couple times per day. So, there must be hardware problems. RDST should be fixed or removed (though if there is a hardware problem, it can't be redeployed).
Replace foam rubber plugs at all year 1 stations (need to notch it to accomodate the GPS pods): ASAR, CBHT, CUET, CVTN, HDBT, LNXT, MCAR, NAIT, SFTN & TUMT.
NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.
CVVA. John contacted DMME again. He's also looking into the possibility of a freewave link to UVA.
We need a single document (access database excel spreadsheet) to document information on all freefield strongmotion stations to include contact(s), current IP's(eternal/external), what ports are allowed in from what IP's here at CERI, telnet yes or no, trigger levels etc. etc.
HDBR
Granites are in house and powered up. Steve assigned ip's (.158.50-57) and David installed rock2slink on them. Mitch is trying to get slink2ew importing data.
HDAR
Bad CMG5 E/W.
Not getting broadband feed.
On hold for Boswell comm.
HDBT
Replace two bad mss100's.
Bad 100' accelerometer.
On hold for Boswell comm.
If we can't get the Boswell comm worked out by the Jan 31, we'll remove bowlworm and discontinue the dsl at Boswell. Steve will investigate if we can get dsl at either the weight station or mud island. Otherwise, we'll make a wireless link from the weigh station to AZTN and look into getting additional bandwidth at AZTN. The building and bridge channels don't need to be exported continuous to CERI but the mud island and weigh station channels do. Though we may wish to have a couple of the AZTN and bridge channels on helicorders for SOH.
Original list taken from ET event (Ohio) 20111231_200500_MAN
Down
GTTN
RCGA
LRVA.EHE
TRYN
WSNC- bad L4
CMGA-periodic down-possible weak battery
Sporadic Down
TVNC
SMNC
Misc Problems
SWET.HHE - drifting - see image
DYTN - gain channel is down - see image
RBNC - gain channel and *.EHE channel problem - see image
Strong Motion problem
CSTN - down at the time of this email
CVVA- internet connection
Only the DYTN upgrade remains.
See map.
Forest service has draft agreement.
STAR Node
BHT move. Latest update from John:
- We had planned on moving BHT from it's present location because access has gotten so bad.
- I have been working on getting a permission to move it about 3 miles to the E NE to Black Mtn. There is some issue with the permission, but my contact with the Cumberland Trail State Park says he believes we can work it out.
- If this station is ever upgraded to a RefTek, it is doubtful we would be able to make the 45 mile radio shot from Black Mtn. to Starr Mtn. so we would have to find a new place to transmit to. It will not repeat through any other station, because it is either blocked by mountains or too long a shot.
- Previously, we had discussed trying to find another site to relocate BHT to in order to center it between WMTN and DYTN.
- I found a site in the Catoosa Wildlife Area called Pilot Knob Mtn, (36 deg. 322'N, 84 deg. 45'W). It is about 18 miles NE of BHT. It is at an old fire tower site and apparently there is still a good road up to it.
- I have also found a couple of potential sites to transmit to where we could get internet.
So, before I proceed any further, we need to decide if a replacement station for BHT should get a RefTek and if so, do we want to put the station at Black Mtn. or at another site.
BRBC Node
Power upgrades: RBNC, and TRYN.
TRYN: Hardware is removed from the tower and will be reinstalled when the new one is complete.
MTTN Node
Greentop is getting a new tower. When? Now--station went down on 12/21 possibly from a cut cable or something. John is investigating.
STLE landowner has asked
that we move eq. cabinet and protect antenna monument with steel barricade.
Antenna mount is done.
The VLF power supplies were installed CJTR and PIGT. Still needed
at NMKM and RLAP.
MAIR is now connected directly to the Water District internet. We can
recover the wireless hardware.
Granites are in house. Dave installed rock2slink on them. Mitch is working on slink2ew.
All 11 CMG6TD's are back, checked, serviced, and ready to go.
What equipment in the FDL is no longer needed. Mitch will call a meeting of FDL users to discuss culling the equipment. Also, the check out/in process is largely ignored--what system could be implemented that people will actually use.
The seisnet group on umdrive is at http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password.
The pink palace display now includes a heli connection for MPHB. Need to retrieve any remaining hardare (e.g. antenna and reciever on roof).
