Tech Staff Meeting Notes October 5, 2009
The top priority is always repair of down stations.
NM problem stations
ET problem stations
The task list updated February 6, 2009 needs to be updated again. Note that everyone is still needed to work on essentially all tasks. There is however, a single person who is primarily responsible for each one. Mitch of course, is ultimately responsible for everything.
Repair down stations
Expansion Space reorganization
SM sample rate fix/yr 3 hardware upgrade
AG Network
There will be a Chilli pot luck on October 30. See Jenny to contribute $2 for spoons and bowls, or to contribute a pot of spicy goodness.
Chris is arranging annual safety training for himself, David, and Jim. Unfortunately none of the staff were able to attend so Chris will arrange another session.
Its likely that some of these station problems are at the nodes and can be fixed by checking the discriminators in detail.
Holly's NM list:
Down
GOBM.EHN
LRAR.EHE
Isis Oxidation problem
SJBM.EHN
JHAR.EHE
WYBT.EHZ - and drifting
MARM.EHN
Gain Channel problem
MORT.EHG
CHRM.EHG
Muted Amplitude
BLAR.EHN
DWDM.EHE
MATM.EHN
MCAM.EHN
Unusable
CHRM
Timing Problem ?
TMAR
Drifting
HICK.HHE
Several of the stations can get improved telemetry by transmitting directly to the node. The original configuration changed enough so that there is room in the band. Jim will investigate.
MKTA:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
LNXT:
DSL is running at lnxt but for an unkown reason we can't see it from Memphis. Steve will call tech support to help get the problem resolved.
Still need to tie generator status alarms into lnxtworm
NMAD:
Need alternate comm path. Telecom says dsl isn't available (though we once had ISDN).
AZTN: Upgrade PC and replace XLAK clock
CERI:
Need to gather up a contact list (e.g. everyone's home phone, power companies, etc) and put it on webworm (password protected). Jim has a draft of this that so far has all the node contacts. Mitch suggested that the umdrive seisnet space might be a better place than webworm for it.
CVTN optical isolation. Need to upgrade to year 3 hardware. Done.
RDST needs upgrade to year 3 hardware with router. Done.
NAIT: Couldn't get the NWS cable modem to cooperate at NAIT (firewall?). Dean was going to ask Comcast about it and maybe get a new modem. Steve will follow up.
Need to complete firmware upgrade on CVVA. The facility changed hands. They are willing to let us piggy back on their internet and John is working out the technical details with them (i.e. which side of the firewall we'll be on, hopefully the outside).
Now that CMG5 triggers actually contain data, we need to tune the trigger level. Mitch sent explanatory info and guidelines to Al and Steve to tune. Need to periodically test the data download function. Triggers need not be saved for any earthquakes less than magnitude 4 (we will already have data from nearly any event). Jim tuned things reasonably well during the upgrade but we need Al to keep an eye on them.
SLU had trouble getting triggers from the 10sps datastream. They are having success creating 200sps triggers off the 100sps channel without excessive triggers (though I wonder what their levels are). The two newer CMG5TD's we got from ASL also don't trigger well off the 10sps datastream. Jim will investigate the feasibility of switching to the 100sps tap as the trigger source for the 200sps upgrade. Then Al will retune as above. Done.
Most of the strong motion stations can run at 100 sps without any modifications. We will install a router on the few that have problems (RDST will be the test case). These (eleven of them) will also need year 3 upgrades. CVTN and RDST are upgraded to year 3 and RDST has the router. But there have been two hangs at RDST in the past month. The best laid plans...
HDBR
Recorders 1, 6, and 7 are repaired and awaiting Steve to reconfigure.
After that we can ask Bob for the next lane closure for reinstallation.
Al reports that of the remaining four, one is not associating.
HDAR
DSL is running at Bosewell and we can see the PC there (bowlworm). But we still can't see the hardware at the weigh station (nor mud island) on the VLAN. Steve reports that Lylah thinks its a wiring problem and will call her for an update.
Need to determine what import machine and wsv all these data should be on. How do we trigger?
Bad CMG5 E/W. (on hold for vlan comms)
HDBT
Mud Island vertical array is operational. Not really, functional is a better
word until we can get it online to qc/vet data (see HDAR).
Replace two bad mss100's. (on hold for vlan comms)
Bad 100' accelerometer. (on hold for vlan comms)
John and Chris had a very long and successful week in the field just before labor day and Chris provided this report:
Thanks guys!SWET- found several problems, some may have been self inflicted. We installed a new solar reg. and coax at vault side. After many checks we found a baud rate difference with vault freewave. Changed that and could reach the CMG3 from bell tower. That left the phone lines from tower to Bran Potter's lab. This took a long time but we got telecom "Geno" to trace the pairs and found problems. Once new pairs were connected I was able to connect with Scream. Bran was very happy!! So I think the problem all along was phone lines. Geno will be the one to help us when we upgrade SWET system.
While I was working on SWET John went to FPAL and installed the 40T.
GMG- e/w S-13 was dug up and replaced. I have 2 S-13s Parker sent back that have issues. An inline fuse installed (power feed for pit box) suspect a bear pulled ground cable from battery box, repaired and covered well with stones. I had keys made for USFS lock that we filed down to fit. Parker now has replacement key for USFS gates. GMG still has key broken in tower cab door lock.
ETT- cleared trees at our eq. site. R+r filter on ac. New Baluns on both ends of balanced line. Installed new ground wire to fix water leak at pit box. Replaced broken window in firetower. Another one is broken. Repaired siding that was loose.
CCRT- had bad cable chew at pit box, connections bad at tower side. repaired and dug in pit box with concrete fill. All connections checked and repaired. Suggest balanced line be put in pvc conduit instead of poly tubing. Large chews on poly tubing run.
CPCT- installed new breakout box, pest chaser and cave lights. Cut trees off power lines at meter.
Looks like ETT started having problems on the 8th would like to have Freon and gauges next visit to Starr Mtn.
Holly's ET list:
Stations down
CPRT (needs another solar panel)
SMNC (John has a replacement sensor)
TVNC (John has a replacement sensor)
Sporadic Outages
LRVA
CMGA
Station problems
BCRT
ETT
GBRT - drifting gain channel
Strong Motion Stations
C2SC
CVVA
STAR Node
RCGA charging circuitry at is old and primitive and in need of an upgrade.
GMG - Need battery box solar reg mounting plate and need to fuse the battery. Done.
BRBC Node
GFM The NC state park service is purchasing GFM. We already have a couple permissions from them and John initiated the paperwork for the GFM permission.
TRYN: the forest service wants to remove the tower. There will be a new 180' tower and we have permission to be on it. We also have permission to be in the building (rack mount only). Can we convert this station to AC?
NCCU: Gordana wants to get this working. John will schedule with her to come in and try to get it back up.
James has a very nice SOH website at http://www.ceri.memphis.edu/people/jpdavis/gama/results.html.
Need weed control and routine maintenance visit to all sites.
James missed the meeting but provided this update.
CJTR - Steve Brewer's still working on contacting DIS to get IP addresses reserved for CJTR. We'll keep using dialup until the network link is established. The MoA will be updated once these chages are all made and finalized.
MAIR - The WAPs at MAIR have not been replaced. The problem was in the blister-pack on the receiver side. I changed both blister packs with new ones to be safe. Comms have been restored. The PSWD office defintely has wireless Internet using the same provider as Mid-Valley. We don't know how long they'll be there and already have alot of effort put into Mid-Valley, so we'll stick with Mid-Valley's Internent for now.
PTGV - Comms were down because we'd been disconnected in the Delta Center's LAN room. Our router has a dedicated Cat-5 run to Bus 'D', jack 31, in their LAN room. Comms have been restored. There's alot of fine dust collected in the GPS enclosure. A visit with a vacuum might be a good idea in the future.
STLE - Comms were down because of a loose connection between our router and the Cat-5 arrestor, which was fixed. Comms have been restored. We're now sharing the Cat-5 run Jim installed with a PC and server-blade for the WREG Weathernet3 station at the Steele School Admin building. We're all connected through a 5-port 3-com switch in the maintenance room. The maintenance room in the admin building is now normally locked at all times. Get keys from the office.
STLE, RLAP, NMKM - All sites are still using early versions of Greg's battery charger. They're running fine, but can be upgraded to the new charger when possible.
Vaults, enclosures, and radios are on the way. Need to clean out and reorganize the expansion space to make room.
All six sites are selected and Horton is working with AGS and Park service on permits for them.
We need a reftek how to. There is a class in Dallas but we missed the registration. Mitch is trying to get Mark here but he has a busy schedule (maybe in November).
Jim has a trillium and reftek test setup on the internet. Mitch will install rtpd on a development box and get ref2ew and basic earthworm going.
Greg discovered and confirmed with Drobek and Meremonte that the sensor power from the reftek is not as clean as the trillium and episensor requires. When Mitch gets him some data to look out, he'll come up with a mitigating design.
Greg determined that current draw should be about 675mA at 12.0 VDC assuming a 70% transmit ratio for the freewave. Two 55-60 watt panels should suffice and to allow for temps down to 10 degrees F, we should use four 105ah batteries to get 14 days backup power.
We are approved for the following in approximately the same order as we
hope to complete them:
Note that using other funds, we're also upgrading 3 additional ET stations to digital with episensor and FPAL to the same hardware as the other broadbands.
Mitch will make a chart to hang in the war room to track our progress (we're required to make quarterly progress reports and to meet the proposed schedule).
There are problems with the Generators and wiring in 3904/218. Symptoms include surges on circuits not backed up by the generators during the transfer test every other monday. This resulted in some damaged hardware on September 30. Also there are the ups problems and rebooted machines from the June storm power outage. Greg guesses there could be loose neutrals. Chris will reconfigure the biweekly test to not transfer power and Mitch will see about getting Greg down here to spend some quality time reviewing and rehabbing the entire mess.
Chuck has a funded project to install a 19 station array somewhere in NW TN. The passcal equipment (40T's with refteks I think) will be available in November. There will be no telemetry. A student will make monthly service runs. The array has a footprint of about 300 meters and is composed of 3 element subarrays of about 30 meters. Chuck will likely try to rent a field. He would like to install them in the pipe vaults like we currently use for the NMSZ permanent broadbands. Hopefully he'll have money for a bobcat for both the installation and decommmission (otherwise they'll be a bear to remove even to just below the plow line). Presumably there will be a meeting in the near future. to work out the details.
What to do with the Steiner brain dump DVD's?
Mitch made a seisnet group on umdrive. Go to http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password. There are several things on there including John's database. We can put contact info there too when its available. Only people in the seisnet group have access to this area. If you download something for update, don't forget to upload it again so that we all have the same version of the files.
The OGS connection is complete (they're running an earthworm on 4 different machines because of the way the DAS processes are spread around). Mitch will make a separate OGS heli page and verify NEIC is getting them.
Rotational sensor is feeding an earthworm but still has IRIG crosstalk problem to fix. Its fine on an Oscope hooked to the output of the mux and looks fine in LabView. Its bad when viewed with wave_viewer. Need to see what the sac files look like to make sure it isn't just a wave_viewer feature.
Supply is checking the condition of Parker's old Suburban, Chris doubts it will run after a year sitting over there. He told her we would try to crank it up if needed. McGoldrick needs to stop worrying about whether or not it will be adopted into a good home and get Old Blue off our lot ASAP. David pointed out that Chris already sold one vehicle this year (his van) and parting two might be too much to bear. Mitch says tough noogies dude, get rid of the damn thing.
The pink palace display now includes a heli connection for MPHB. Need to retrieve any remaining hardare (e.g. antenna and reciever on roof).
We're not allowed to store dead batteries in the open on the pallet outside Chris's garage. We need to get rid of them within 3 or 4 days and/or keep them (short term only; no long term storage) in a labeled enclosure (does it need to be vented?). Chris will look into what's required and make it so.
Chris called P Plant electrical shop. They filled the generators soon after the outage. He said when there is a power failure as long as the generator is running that powers fuel tank pumps on campus they will bring fuel to keep us online. Chris will get all contact information posted near transfer switch and added to Jim's list.
3890 Power Chris is going to make a diagram for 3890. Good luck son.
3876 Power Chris is going to make a diagram for 3876. Good luck son.
Should we have a trouble ticket system?