Tech Staff Meeting Notes May 3, 2010
The top priority is always repair of down stations.
NM problem stations
ET problem stations
NSMP station maintenance is now added to the Duty Table (Data still go directly to NSMP in Menlo Park and NSMP still does the SOH). Note also that everyone is still needed to work on essentially all tasks.
Fiscal Fish Fry is scheduled for July 9.
Everyone should sign up for the DiSC Personality Profile class at http://uofmpt.memphis.edu. It will help us work even better as a team. The class is on June 10 (a thursday) from 1pm to 4 in AD 177B.
Chris and Dave continued their campaign of pulling canisters and continue
to chip away at Holly's list.
Holly's NM list:
My list of high priority stations to fix: RDGT (done), PWLA (done), HOVM, CTCT (pulled), WYBT (done), and MARM
Down
CTCT - cannister pulled
MPH.LLZ
HOVM
Sporadic Outages
BACM
BRNM
Muted Amplitude
BETM
BROM
CATM.EHE
DWDM.EHE
LRAR.EHE
MARM.EHE
MARM.EHN
MATM.EHN
WALK.EHN
WADM.EHN
Isis Oxidation problem
SJBM.EHN
Gain channel problems
CHRM
MORT
Unusable for v. small events
PLAR
CHRM
Add BRNM, which also knocks out BACM, to the list. It's been having problems for the last week. Looks like an intermittent power problem, most likely a bad SR2.
Node reset instructions are at
/gaia/smeagol/home/seisadm/ew/node.notes/phone.reset.proc
Note that using the pc reset option with the GX620 system can cause it
to hang. Only power cycles are recommended. The reset option has been
disabled at most nodes.
new batteries were installed at lnxt, nmad, sftn, and richardson. mkta is on deck.
MKTA:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
Mitch met with Bob Woodward and Bob Busby of IRIS about EarthScope. Bob Busby said he might be able to loan us some WildBlue hardware to test. Assuming we can get sufficient bandwidth without violating shared use maxima, this could be a viable option if dsl is unavailable.
LNXT:
The generator major failure and a fan. Chris and Dave replaced the fan, and shaft, and bearings, put on new belts, antifreeze, gear oil, thermostats, gaskets, yada, yada. Still has interlock problems on startup, needs a load test and should have a complete "physical" by a certified tech. Chris is looking into it.
LNXT now has a phone reset (see note above).
The DBB is currently stable and there is a second computer at lnxt on the dsl. Its second ethernet on the 10.156 is currently disabled.
NMAD:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
New AC unit installed. Old one is back for repair.
New Low Voltage cutout is installed.
AZTN: Upgrade PC and replace XLAK clock
CERI:
Need to gather up a contact list (e.g. everyone's home phone, power companies, etc) and put it on webworm (password protected). Jim has a draft of this that so far has all the node contacts. Mitch suggested that the umdrive seisnet space might be a better place than webworm for it.
The DBB firewall installed and operating. Only specific ip's are allowed through. You need to use the route command to manually add a route. Evertying behind the firewall is the same D ip number but with 10.156.1 instead of 141.225.156. Remaining items include:
I lost track of what stations need what part of the yr3 upgrade. Based on a recent email from Jim we have these stations waiting completion:
| station | yr3 hardware | router | mss100 |
|---|---|---|---|
| RDST | Y | Y | N |
| MCAR | Y | N | N |
| CUET | N | ? | Y |
| SEAR | Y | Y | Y |
| MKAR | Y | Y | Y |
NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.
CVVA is upgraded but there remains an unresolved issue with the firewall Steve is working with Merritt Reagan NGC to get them resolved.
Al and Jim have all the issues resolved and we're now able to download 200 sps triggers. These are the guidelines.
HDBR
There was a very successful trip under the bridge on April 13. Jim reported:
We got six recorders back online, had to pull #5 due to a hard drive failure. Of all the options I think I like this the best. It looks like we can get a DC powered "industrial" (-40 to 80 deg. C) 4 port switch which supports TX/RX multimode fiber w/ ST connectors for around $200. I'll look into it further tomorrow.Steve found out that there's a second ide controller that he may be able to use to easily fix the #5 hard drive problem.
Recorder #7 needs a site visit to correct a non-fatal error in the bat file.
It isn't realistic to move the antennas so we're not likely to significantly improve the telemetry under the current circumstances. The part that causes the problems though are not the recorders but the ringbuf program. This pulls data using the netbui remote mounts into a buffer on the windows 98 box at aztn. Note that this is not really a proxy for a real-time feed and never has been. And we don't really need continuous data. We can pull data as needed from the recorders when there's an event. And instead of Al sinking enormous time and frustration into keeping ringbuf going, he can instead just download 15 minute (or some several minute time segment) at regular intervals (say once per month or week) to verify that its working. Steve and Al are working out this new procedure.
We also discussed the need to have the data vetted. No qualified structural engineer, to Mitch's knowledge, has looked at bridge data and verified that its good. Mitch processed a day of continuous data that Steve saved on October 27, 2005 (/gaia/home/mwithers/Projects/I40Bridge/hdbr/segy/20051027). The instrument response was removed, data were converted to segy, and a gzipped tar ball is available. Mitch looked at it and it looks reasonable but it should still be vetted by a structural engineer who knows what to look for.
HDAR
Bad CMG5 E/W.
Not getting broadband feed.
HDBT
Replace two bad mss100's.
Bad 100' accelerometer.
Holly's ET list: (this is an old list due to storm interference
at the time of the meeting)
Down
CCNC (fixed)
CMGA (fixed)
LRVA.EHE
Sporadic Outages
TRYN
TVNC (interference)
SMNC (interference)
Strange signal
ATTN (see attached)
ATTN_GMG-April2010.docx
Unusable
GMG- horizontals (fixed)
Strong Motion stations down
ADSC (networking issues)
CVVA (networking issues)
FPAL is happier since the upgrade to a Ref Tek, Trillium and Episensor. It is staying about 10 degrees warmer than the Arkansas stations so some shade or forced ventilation may be in order.
Mitch needs to get off his butt and select the 3 non-ARRA ET stations for upgrade.
STAR Node
CPCT had bad timing. John reported that there is no damage to any hardware at Cooper Cave. While he was checking the GPS pod he saw 2 satellites briefly. It looks like we are just dealing with an old dumb GPS receiver with bad sky view. John cut down some trees to help. Al put it back in the tank and so far it seems to see 6 sats and is holding the correct time.
BHT. Chris reports that it is time to seriously pursue another site as access is even difficult by 4-wheeler. David reports moving it will take all the fun out it, what's the worst that could happen? John said that folks at Cumberland Trail State Park said we can put the station at another spot on Black Mtn. which is just a couple of miles NE of the present location. It is accessible by a paved road. We can pursue that or I can look for something that will put it further NE to get it more equidistant between DYTN and WMTN. Right now it is awfully close to DYTN and BCRT. Mitch likes the idea of splitting the difference between DYTN and WMTN. John will dig out his old notes and starting looking around.
BRBC Node
Many of the NC station problems are probably because of insufficient power. John will find out what's needed to increase solar and battery to help make it through the winter storms. The ET stations have the same issues but problems are not as evident because they are, in general, at much lower elevation (less snow). CMGA has had a power upgrade to two 100 aH batteries and a 64W solar panel. Other stations in need of power upgrade are: TVNC, GFM, RBNC and TRYN.
GFM The NC state park service is purchasing GFM. We already have a couple permissions from them and John initiated the paperwork for the GFM permission.
TRYN: the forest service wants to remove the tower. There will be a new 180' tower and we have permission to be on it. We also have permission to be in the building (rack mount only). They want this complete before winter.
There seems to be a strange, repeating signal on many of the stations, most notably on WSNC/RBNC. It repeats about every 14 minutes and lasts about 1 sec.
MTTN Node
Greentop is getting a new tower too. There will not be internet up at the mountain, so we need to find a place to transmit to. We should be able to hit just about anything in Knoxville, Maryville or Pigeon Forge; is there a preference?
A new prof at UNC-CH, Lara Wagner, might be interested in co-locating some of here equipment (Nanometrics broadbands & digitizers), with as at VHTN or LRVA. She is setting up a transect array of 10 stations in western NC.
James has a very nice SOH website at http://www.ceri.memphis.edu/people/jpdavis/gama/results.html.
HCEX primary power is out. Most likely, it's blister pack has wriggled free from the plugin. It's a quick fix if someone's in the Troy area.
CVMS NetRS receiver is developing a tracking problem. It'll need to be replaced with one of our new wiz-bang NetR8 receivers. I need to get with Chris or Dave and go over the NetR8 and how to install it. The old receiver will need to be returned here so we can RMA it.
Most monuments need a new paint job.
Need to do weed control and most stations.
Access points have undersized power supplies and need to be replaced.
Need to replace older GPS-100 supply systems.
These stations still need to be steinerized: CVMS, MCAM, HCES, MAIR, NWCC.
STLE, NMKM, and RLAP all need a charger upgrade. They are currently working (and have been for a long time) so this is not necessarily a top priority.
Comm line from maintenance shed to park service office at CCAR was restored but our cable to the wireless access point on the outside of the building was accidentally cut when the siding was being being replaced.
This is Jim's list of hanging chads to finish up the AG network installation. After that, we enter O&M mode.
Jim and Greg worked out a plan for the communications that won't make the towers at the nodes fall over.
Jim began ordering supplies. Radios may need to go out for bid.
Chris has the NM permissions done.
We are approved for the following in approximately the same order as we
hope to complete them:
Note that using other funds, we're also upgrading 3 additional ET stations to digital with episensor and FPAL to the same hardware as the other broadbands.
Mitch hung a chart in the war room to track our progress (we're required to make quarterly progress reports and to meet the proposed schedule).
We need to be very careful to keep AG, ARRA, and CERI hardware separate and be certain that hardware purchased by each project goes only to that project.
All replaced DM24's and CMG40T's are to go to south campus and taken off inventory. This hardware is not to be redeployed anywhere in the CERI seismic network nor FDL.
Two CMG6TD's are at guralp depot in Menlo for repair.
Need to review power loads and line balance after adding the new AQMS servers. Review needs to consider addition of new landmark servers, mac server, and seisnet deep waveform buffers. The Earth Science Library (ESL) in 3892 is moving to the main branch and has a liebert. We should consider carving out rack space in that room (though backup power is an issue).
We're now officially providing maintenance support for NSMP stations in the NMSZ area. There are 22 stations. SLU has 11 to the north and CERI has 11 in AR and TN. They are all dial-up K2 or Etna. Data continue to go to NSMP in Menlo who will continue to monitor SOH. They will email or call when there is a problem. We're the boots on the ground.
Due to remodeling, we pulled K2 1738 from Firestation #2 in Dyersburg, TN in February 2009. It is in the shop with accessories.
From Arnie Acosta regarding Sta. 2460 Obion TN building closing:
I received a call today from assistant Fire Chief Mr. James Evans with the Obion Fire Department alerting us that the building that K2 1867 is located in will be torn down in 30 days.
Mr. Evans says we may be able to reinstall the K2 in the new fire station about 21/2 blocks away, so if You can arrange one of Your technicians a site visit to relocate the accelerograph that would be great. Mr. Evans phone number is (731) 335-0988, if You any questions please contact me.
The neic has a new metadata website at https://geohazards.usgs.gov/smds/index.php?message=1. John is finished populating it with info from our stations.
ISTI was contracted by USGS to support INV. John will work with them to get initiall dataless SEED going. John will be our INV czar.
Dave will ask Melody about replacing our cell phones with more current models.
What to do with the Steiner brain dump DVD's?
Mitch made a seisnet group on umdrive. Go to http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password. There are several things on there including John's database. We can put contact info there too when its available. Only people in the seisnet group have access to this area. If you download something for update, don't forget to upload it again so that we all have the same version of the files.
There is now a ceri_seis email group that includes all the seismic networks staff.
Rotational sensor is feeding an earthworm but still has IRIG crosstalk problem to fix. Its fine on an Oscope hooked to the output of the mux and looks fine in LabView. Its bad when viewed with wave_viewer. Need to see what the sac files look like to make sure it isn't just a wave_viewer feature.
Chris continues to hound the supply to try to get Parker's old Suburban onto govdeals.
The pink palace display now includes a heli connection for MPHB. Need to retrieve any remaining hardare (e.g. antenna and reciever on roof).
3890 Power Chris is going to make a diagram for 3890. Good luck son.
3876 Power Chris is going to make a diagram for 3876. Good luck son.
Should we have a trouble ticket system?