Tech Staff Meeting Notes April 5, 2012
The top priority is always repair of down stations.
NM problem stations
ET problem stations
NetOps was useful and informative. We're going to use the ASL job ticketing system.
Fishing trip is April 19-20.
There are several stations that have a problem with arbitrarily hanging. Mitch sent an email to the netops list but the resulting suggestions didn't really help to solve the problem. Many of the reftek stations mysteriously cleared up after reloading the firmware.
GTTN/CPRT. These are two refteks behind a single router. They use udp to transmit data. The router opens a port when a broadcast is initiated by the reftek. The rtpd on achilles acknowledges back to the reftek on that port. When the refteks creates a new connection for whatever reason, sometimes the router port used by the router isn't the same as what the reftek or rtpd thinks it is. Steve has a "fanless pc" in house with ubuntu installed and he is working on rtpd/rtp2rtpd and on Scream. Jim updated the firmware on the GNAR reftek and its been behaving ever since (even though we were previously at a newer release than what reftek support said was needed to fix this problem). So while we were going to test the fanless pc at GNAR first, we can now go straight to GTTN/CPRT (though new building construction is underway there now).
RDST and MCAR are also prone to hang. They're now on gcf2ew along with a script that forces a kill and restart when necessary. Need a fanless pc.
EDIT, NNAR & CHNM: NNAR and CHNM repeat through EDIT and the received signal is very bad due to trees. NNAR and CHRM are very noisy and fade out quite a bit. EDIT is now located in the Chicksaw National Wildlife Preserve so cutting trees is not an option. There isn't a clear path to a repeater on the Edith forest tower either. Need a new plan (maybe go directly to the node?).
WALK: Chris and Manning were unable to reach WALK up in the KY bend. Mr Lynn indicated that the station was completely underwater but it is still up? (yes still up).
Search for quieter site for CTCT.
Data taken from the 4.0 Oklahoma event. Here:///gaia/smeagol/dat2/Working/NM/201204/20120403_073430_MAN/
Down
TNMT - frequent extended drop outs.
HCAR - flooded, need to verify that well stays dry.
PGVM.EHN
RELT.EHZ
Unk Problem
MLDM - see image
NNAR -interference? see image
Strong Motion station problems
PARM.HN* - see image
HDAR
HENM.HNZ - muted amplitude
RDST
LNXT HN* NM 00 (that's the CMG5TD, LNXT HN* NM 01 is the RT147 and is operating normally).
Isis Oxidation
GLST.EHE
Muted Amplitude
WALK.EHN
LRAR.EHE
BLAR.EHE
BETM.EHN
MCAM.EHZ
WADM.EHN
Mag. 4.0 gain range problems
These stations had problems with the gain channel during the Magnituded 4 Missouri
event on Feb 21, 2012. It isn't clear to me whether they data channels saturated
or whether there was excessive chatter in the gain channel.
See image.
CHRM
DWDM
MATM
PGVM
FLPT
LFRT
HTAR coax problem.
Weak Battery/Solar
There were 10 stations down for this event. I'm guessing that for a
majority of them it was just the battery. Chris fixed 4 of them. HCAR was
flooded and HTAR is a likely coax problem. The remaining 4 are listed
/gaia/smeagol/dat2/Working/NM/201112/20111224_124800_MAN
LFRT, LVAR, TNMT, and WADM.
DBB
MKTA
LNXT:
Power converter failed and was replaced.
CMOS Battery is bad. Steve will order enough to replace them at all nodes.
DSL modem is bad. Steve is getting a replacement.
Partical strongmo conversion to Basalt started. Needs to be finished.
Replaced fan--need to do that at all the nodes.
Jim is getting a quote to replace the generator blockheater, muffler and exhaust manifold, and vaporizor solenoid.
Mitch needs to modify env_alarm to customize for lnxt generator alarms:
Here are what the new Lenox node status alarm channels will be when the new generator interface module is installed. As was the case before, all outputs are normally high and go low to indicate an alarm condition. High is defined as +2.5 VDC or approximately +1024 counts.
- Ch2 Generator Failure, will go low when generator tries to start and fails (was AC power failure)
- Ch3 XLAK GPS IRIG time code generator alarm (no change)
- Ch4 Generator online, will go low when the building AC is transferred to the generator (was Moisture alarm)
- Ch5 Dyersburg Elec./Building AC fail, will go low when Dyersburg Elec. AC lines into transfer switch are dead. (was Intrusion alarm)
- Ch6 Under temp. <50 F (no change)
- Ch7 Over temp. > 90 F (no change)
NMAD:
Steve will cancel phone line.
G and D and the local EMA installed a 25kW generator to power that side of the building. If there's room power-wise would they let us power the rack from it too? We could share the maintenance of the generator. We'd also have to install a transfer switch.
AZTN:
CERI:
Blew surge arrestors on two of the three tone bundles for the ABB. Repaired.
Twisted pair from Richardson is extremely noisy causing problems with
the analog backbone. Richardson is also on the UM 5-year capital
improvement plan to be replaced. We got a tour of Wilder and its doable.
We need to see if we can get a path to CERI so we know whether we just
need to move what's on Richardson now, or whether we need to turn Wilder
into an entirely new node (i.e. new mudworm). Then we can get Planning
to help us with a design and then we can get permission.
We could just move the firewall from 3904/218 to Wilder and get the DBB
on the campus internet that way (if there's no aironet line of site).
There are good telephone pairs from from the Wilder roof to the basement
and we could move the ceri node/mudworm to the basement. Otherwise we'd
have to find and trust connections between the basement and CERI.
We have line of site from the southeast corner of 3904 except there are powerlines in the way. Can we put up a tower on the east side of 3904 to get over the trees and power lines on central?
Ownership, Operation, and Maintenance of the four Charleston, SC stations (C1SC, C2SC, ADSC, and TRSC) was turned over the Univ. of SC at Columbia. At Phillip's request, we letting the data flow through CERI for the time being. Eventually it will get switched to Columbia and the network codes will then change from NM to CO.
RDST, CBHT, and MCAR are coming in via gcf2ew on anssworm. CBHT occassionally loses its connection but is overall, ok. MCAR loses its connection about twice as much so while its mostly functional, it should be better. At RDST, a datapacket sneaks through a couple times per day.
Replace foam rubber plugs at all year 1 stations (need to notch it to accomodate the GPS pods): ASAR, CBHT, CUET, CVTN, HDBT, LNXT, MCAR, NAIT, SFTN & TUMT.
NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.
CVVA. John contacted DMME again. He's also looking into the possibility of a freewave link to UVA.
We need a single document (access database excel spreadsheet) to document information on all freefield strongmotion stations to include contact(s), current IP's(eternal/external), what ports are allowed in from what IP's here at CERI, telnet yes or no, trigger levels etc. etc.
HDBR
HDAR
Bad CMG5 E/W.
Not getting broadband feed.
HDBT
Replace two bad mss100's.
GTTN move is rescheduled for May 14. Chris or John will be present to guide the move and make sure its working afterward.
Data taken from: /gaia/smeagol/dat2/Working/ET/201204/20120402_174120_MAN and /gaia/smeagol/dat2/Working/ET/201204/20120403_073430_MAN
Down
TRYN
WSNC
CMGA -- roughly daily dropouts (signal fades then goes flat) probably not solar related, need to camp out with a spectrum analyzer to identify the interference.
GMG: no comm with Reftek at the pit.
BCRT -- this one looks like a flakey reftek affecting all six channels, John
has requested a replacement RT-130 and RT-147 from the depot.
Gain Channel Problems - see images
ETT -- see image, gain channel out and drifting EHN channel
DYTN
Unk Problem
GRBT -- Spikes on vertical see image
RBNC.EHE - see image
Strong Motion station down
CVVA - no internet
STN -- data gaps
Only the DYTN upgrade remains.
See map.
Agreement is complete: clear to launch.
STAR Node
BHT move. John has a verbal permission from the Catoosa WMA to move it about 9 miles to the E NE to Pilot Knob Mtn. This likely will need to be a written permission. The new location does not have a path to Star Mountain and we don't currently have hardware for a digital upgrade. Need a new plan. John is looking into getting the road repaired to BHT.
BRBC Node
Central Receive: Still have interference from the ham operator. John will pursue this with the FCC.
TRYN: New tower and building are in place, O.K. to install anytime.
WSNC: Fixed transmitter but had to pull the sensor to repair the E-W channel.
CMGA: need to camp out with a spectrum analyzer to identify the interference.
MTTN Node ASTN: Change to AC power. We have permission and can install anytime.
RLAP, CVMS, HCES, PIGT are out as of 04/05/12.
STLE landowner has asked that we move eq. cabinet. Chris is looking for options. They have not done any building yet (04/04/12).
LCHS and HCES will get upgrades to NetR9. *** Done, having problems with HCES, plan to swap back to the NetRs that was there next week. Also plan to run the conduit.
We need a better plan for end-to-end O&M of the GPS network in the post JPD era.
The VLF power supplies were installed CJTR and PIGT. Still needed at NMKM and RLAP.
Chris is trying to find out if we have them or if we need to purchase.
Need to recover the wireless hardware.
Granites are on a 192 network. Steve is setting up a worm to pull data over the wireless and do an end-to-end test for two weeks prior to installation.
The rotation sensors leaked. PMD no longer makes them and isn't willing to repair or replace. New sensors are not in the budget.
HDBT enclosure is reinstalled.
There is about 50' of water in the 100' borehole. The sensor package leaked and is now trashed. It is not repairable and there's no budget for a new sensor.
CMG5TD is ready for installation at HDAR. Need to reinstall hardware for link to AZTN.
All 6 ag stations got a visit, LCHS and FCAR were fixed. Next visit plan on new batteries. Want to do at least two stations/year. Horton has funds.
Chris is looking for suitable bags to protect them during burial. First experiment with a bag under water failed.
What equipment in the FDL is no longer needed. Mitch will call a meeting of FDL users to discuss culling the equipment. Also, the check out/in process is largely ignored--what system could be implemented that people will actually use. Jer Ming is having trouble with some of the old equipment.
We have one bad CMG-6 Should it go Guralp for repair? Who pays for it?
All hardware is now on a 4s gain range delay. That is, once it gain ranges, its will stay low gain for at least 4 seconds before going back to high gain. Each cycle that exceeds the threshold resets the counter. So it will return to high gain 4 seconds after the last peak that exceeded the threshold.
We'll start with the upgrades to obtain parts for other repairs. MORT and LEPT are first up.
Completed maintenance visit to Newport,Paragould & Corning.
Newport needs a phoneline.
Obion has a gps problem, plan to be in area next week.
Paris Tn plan a visit whem looking for Middle tn Site.
Tiptonville Tn hope to visit next week.
Are the 40T's going to Argentina or ASL?
Need a new door locking scheme.
The seisnet group on umdrive is at http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password.
