Tech Staff Meeting Notes January 5, 2010
The top priority is always repair of down stations.
NM problem stations
ET problem stations
The task list updated February 6, 2009 needs to be updated again. Note that everyone is still needed to work on essentially all tasks. There is however, a single person who is primarily responsible for each one. Mitch of course, is ultimately responsible for everything.
Repair down stations and routine maintenance
Assemble hardware for AG Network
NM broadband ARRA upgrade prep
Our USGS cooperative agreement for the next funding cycle is approved and is for five years.
Three servers are on the way from Golden to host the CISN software to be installed in Memphis to process all regional data. Everyone is tired of calling it the CISN Software and the name Cheetah was not well received. The NIC decided on ANSS Quake Monitoring System (AQMS).
The open research equipment tech I position will be moved to the new networks contract and will be advertised in January. The goal of filling it by February 1 may slip but not by much.
Greg is now on stimulus funds to can help with those efforts.
Classes begin January 14.
Chris is arranging annual safety training for himself, David, and Jim. Unfortunately none of the staff were able to attend so Chris will arrange another session.
Its likely that some of these station problems are at the nodes and can be fixed by checking the discriminators in detail.
Holly's NM list:
DOWN
MORT
Muted Amplitude
LRAR.EHE
DWDM.EHE
TNMT.EHN
GOBM.EHN
MARM.EHE
MIST.EHE
MSAR.EHN
WADM.EHN
Unusable
CHRM
PLAR
Gain Channel Problem
CHRM
FLPT (doesn't gain range)
Drifting
KEWM.EHE
Isis Oxidation
HCAR.?
SJBM.EHN
MARM.EHN
MCAM.EHN
WYBT.EHZ
Jim noted that sungle component problems are non-trivial to fix. He asked if some stations could be decommissioned and the hardware resinstalled at bad stations. Mitch asked whether new hardware could be built and used to replace aging systems--the design is sound but some hardware is nearly 15 years old.
Several of the stations can get improved telemetry by transmitting directly to the node. The original configuration changed enough so that there is room in the band.
Jim asked if the outage list could be prioritized. Holly and Mitch will work together to do a mass relocation to get Hypo statistics on which stations are used more. They'll use this info help set priority stations (though any down stations is still undesirable).
MKTA:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
LNXT:
ssh via lnxtdsl is now available from smeagol and other trusted hosts.
Still need to tie generator status alarms into lnxtworm
Now that we have a phone line for the dsl, we can replace the adam6060 with the simpler style phone reset.
NMAD:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
Over the holiday break, Mitch had to reboot nmadworm because it was suffering from the serial mouse freakout problem. When it came back the broadbands didn't. Scream2ew was not getting data from Scream. Previously we had scream2ew configured to get udp from port 1567 with no Server configured. Mitch couldn't get this to work nor could it get it to work on the loop back. Its now getting it from the 10.156 address on port 1567. I don't know why this is changed (maybe because the ip is now "unroutable"?) but it might be more reliable and may be necessary on other machines behind the firewall.
AZTN: Upgrade PC and replace XLAK clock
CERI:
The battery on the primary generator failed and was replaced by P-plant.
Chris tested the old generator and it smoked and ran very rough. He contacted P-plant who then put in a service request with Cummings Mid-South.
Need to gather up a contact list (e.g. everyone's home phone, power companies, etc) and put it on webworm (password protected). Jim has a draft of this that so far has all the node contacts. Mitch suggested that the umdrive seisnet space might be a better place than webworm for it.
The DBB firewall installed and operating. Only specific ip's are allowed through. You need to use the route command to manually add a route. Evertying behind the firewall is the same D ip number but with 10.156.1 instead of 141.225.156. Remaining items include:
For some stations (most frequently SEAR) data feeds get dropped. Scream sits there "connecting" but never connects. The mss100 is fine. A disconnect then connect on scream fixes the problem (or rather fixes the symptom--the problem is unknown). Mitch will look into implementing reboot_mss_ew.
NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.
CVVA is upgraded but there remains an unresolved issue with the firewall Steve is working with Merritt Reagan NGC to get them resolved.
Al is making a practice run on the strongmo trigger download for data from the 3.4 near Dyersburg on Nov 6. Mitch sent these instructions are archving triggered strongmotion data:
HDBR
Recorders 1, 6, and 7 are repaired, configured, and awaiting installation.
Al reports that of the remaining four, one is not associating.
Resets are not working well and tones are degraded. Likely cause is telemetry between the reset unit on the AZTN roof and the DAS's on the bridge. If its ice, it will self-correct. Otherwise we'll check it when we get time and a lane closure to install the 3 repaired units.
HDAR
DSL is running at Boswell and we can see the PC there (bowlworm). But we still can't see the hardware at the weigh station (nor mud island) on the VLAN. Steve reports that Lylah is having trouble tracking down the problem.
Need to determine what import machine and wsv all these data should be on. How do we trigger?
Bad CMG5 E/W. (on hold for vlan comms)
HDBT
Mud Island vertical array is operational. Not really, functional is a better
word until we can get it online to qc/vet data (see HDAR).
Replace two bad mss100's. (on hold for vlan comms)
Bad 100' accelerometer. (on hold for vlan comms)
Holly's ET list:
DOWN or SPORADIC
CCNC
CCRT
LRVA.EHE
RBNC
SMNC
TRYN
TVNC
WSNC
DOWN
FPAL - broadband
FPAL hung again. Strongmo is still sending data. Can't telnet to the terminal server. Jim sent a uds 2100 to John to configure and take to FPAL. If its the sensor though, repair will have to wait for the new reftek and trillium.
Many of the NC station problems are probably because of insufficient power. John will find out what's needed to increase solar and battery to help make it through the winter storms. The ET stations have the same issues but problems are not as evident because they are, in general, at much lower elevation (less snow).
STAR Node
BHT. Chris reports that it is time to seriously pursue another site as access is even difficult by 4-wheeler. David reports moving it will take all the fun out it, what's the worst that could happen? John said that folks at Cumberland Trail State Park said we can put the station at another spot on Black Mtn. which is just a couple of miles NE of the present location. It is accessible by a paved road. We can pursue that or I can look for something that will put it further NE to get it more equidistant between DYTN and WMTN. Right now it is awfully close to DYTN and BCRT. Mitch likes the idea of splitting the difference between DYTN and WMTN. John will dig out his old notes and starting looking around.
BRBC Node
GFM The NC state park service is purchasing GFM. We already have a couple permissions from them and John initiated the paperwork for the GFM permission.
TRYN: the forest service wants to remove the tower. There will be a new 180' tower and we have permission to be on it. We also have permission to be in the building (rack mount only). Can we convert this station to AC?
NCCU: Gordana wants to get this working. John will schedule with her to come in and try to get it back up.
James has a very nice SOH website at http://www.ceri.memphis.edu/people/jpdavis/gama/results.html.
James serviced PIGT and checked the battery (its good).
CVMS needs routine service and battery check.
CJTR is not online. All GPS are stations are now on the internet. Well done James!
STLE, NMKM, and RLAP all need a charger upgrade. They are currently working (and have been for a long time) so this is not necessarily a top priority.
Both receivers were replaced at HCES. Coax for Mattioli's receiver is trashed--awaiting new cable from them.
All 6 stations are roughed in.
Cables were returned to Reftek for replacement with cables that works with the Sensor Control Board. Six SCB's are on order.
We are approved for the following in approximately the same order as we
hope to complete them:
ASL is sending the 13 Reftek's and 13 Trilliums. ASL ordered cables. No word on accelerometers nor additional 3 refteks.
Note that using other funds, we're also upgrading 3 additional ET stations to digital with episensor and FPAL to the same hardware as the other broadbands.
Mitch hung a chart in the war room to track our progress (we're required to make quarterly progress reports and to meet the proposed schedule).
We need to be very careful to keep AG, ARRA, and CERI hardware separate and be certain that hardware purchased by each project goes only to that project.
All replaced DM24's and CMG40T's are to go to south campus and taken off inventory. This hardware is not to be redeployed anywhere in the CERI seismic network nor FDL.
Needs to be cleaned up in preparation for receipt of 11 crates from PASSCAL for Jer-Mings experiment.
Greg's advice: I can not find any transient voltage suppressor that i like better for the computer room. I suggest buying the APC model PMF-4S-A. List price is $2325. Purchasing should be able to find a distributor with a little bit of a discount. The unit is pretty bare bones except for a surge counter which will at least tell you if an event occurred. The unit should be connected either directly to the output of the transfer switch, or to a 100A breaker in the essential loads panel next to the transfer switch. You will probably need less than $100 of conduit and fittings to make it work. Chris should be bable to install it, or i can come down to install it. You can go to apc.com to look at it. look under TVSS datacenter/120/208 v 160,000 KA rating. for pictures etc..
Installation complete.
Our cell phones have extremely poor performance in outlying areas. They should be replaced.
Tom Brackman at NKU has a shiny new CMG6TD that he wants to install and get integrated with ANSS. He'll run scream and Steve will help him get a minimal earthworm going to export data (only need startstop, statmgr, scream2ew, and export_scnl).
What to do with the Steiner brain dump DVD's?
Mitch made a seisnet group on umdrive. Go to http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password. There are several things on there including John's database. We can put contact info there too when its available. Only people in the seisnet group have access to this area. If you download something for update, don't forget to upload it again so that we all have the same version of the files.
There is now a ceri_seis email group that includes all the seismic networks staff.
The OGS connection is complete (they're running an earthworm on 4 different machines because of the way the DAS processes are spread around). Mitch will make a separate OGS heli page and verify NEIC is getting them.
Woops, not so fast. Connecting their scream with ours somehow clobbered their kiosk display so they turned off the feed. Steve will help them, simplify things and use an earthworm export instead. New hardware may be needed.
Rotational sensor is feeding an earthworm but still has IRIG crosstalk problem to fix. Its fine on an Oscope hooked to the output of the mux and looks fine in LabView. Its bad when viewed with wave_viewer. Need to see what the sac files look like to make sure it isn't just a wave_viewer feature.
Chris continues to hound the supply to try to get Parker's old Suburban onto govdeals.
The pink palace display now includes a heli connection for MPHB. Need to retrieve any remaining hardare (e.g. antenna and reciever on roof).
3890 Power Chris is going to make a diagram for 3890. Good luck son.
3876 Power Chris is going to make a diagram for 3876. Good luck son.
Should we have a trouble ticket system?