Tech Staff Meeting Notes October 2, 2024

The top priority is always repair of down stations.
NM problem stations
ET problem stations

Duty Table updated September 2023


Priorities


News & Highlights

ET status
We lost contact with most stations during the day on Friday; the High Peak node at 15:15 UTC on 9/27 (11a eastern). As of Saturday 9/28 we had a peak of nine Helene related outages in the ET network. N4 stations on solar and satellite remained live. Both LRVA and TVNC2 were also out (and remain so) but were out for other reasons prior to the storm. A maintenance trip was in the works to repair them. Both VHTN and GFM came back Saturday 9/28. The remainder came back gradually until as of Wendesday, only two Helene outages remain. Both are a result of not contact with the High Peak node. Both LRVA and CCNC are live but not sending data because of a failed seedlink server, most likely due to a failed GPS. A site visit to replace the GPS antenna. The rest appear to be undamaged, though it's difficult to know without a visit. There may be funds for network hardening. It will be weeks or months until we can get into western NC.

The USGS may be placing at tech at CERI to assist ASL with N4 maintenance. The primary location may be McGoldricks garage. We would need a shed for all the tools currently in the garage and mostly likely put it next to the expansion space.

The pace on the multiple actions necessary to fill the open tech position are proceeding at a very slow pace. But progress is not zero and it is progressing.

There was discussion on MPHB. Mitch wants to close it. Can leave the sensor in the hole for now, but Mitch is concerned with old hardware getting stuck in a good hole. This led to a discussion on the orphaned sensors and obsolet DM24s on either side of the bridge. Only one remains operational and there is no funding except for the ANSS surface installations. Should remove the old hardware as time permits and close the epochs for the obsolete channels.

The MT deployment is complete. It will be pulled in about a month but the student should be able to that on his own assuming he can recruit a helper.

STAM radio test failed, upgraded DLAR to digital instead. STAM will take more work and will be one of the 3 2024 DMs.

Crotwell visit went very well if a bit soggy. The field component of the visit was rained out by Francine. We later found out there is an FDSN web service for AQMS and ISTI will install it on our ubuntu port they're working on. Need to follow up with a design. Model it after Comcat (though comcat doesn't have waveforms, helis, or nice station maps and metadata).

Planning was in favor of a fenced compound and offered a couple locations. Mitch recommended the one east of the long building though more could be in the grass and less in the parking lot. Not sure where it's going.


2024 DM

We have an index and are ready to go. All spending must be completed by January 31, 2025.

Continued SLU adoptions O&M.

Upgrade three NM analog stations (cascadia PH w/Centaur from ASL). We selected these three stations: CACT, ARPT, STAM. This is progressing fast. GFE is in house and most supplies, about $26k worth, are in house or on order.

New node routers

NMSZ

The move to the new building at Luxora is imminent.

have equipment and supplies for two more non-GFE analog upgrades: NWCT and CACT. These are done and represent the 20th upgrade if you count both upgrades at RDGT individually. Mitch owes the staff lunch.

RDST2 needs an IP change though James would like to have someone onsite or available for immediate departure when he makes the change in case something goes wrong and we lose contact.

The big liebert is in need of replacing. Does it need to be a liebert or is there a more cost effective way to cool that room. Is humidity control so important anymore?

We need hot-spare hardware for the nodes. Clocks, radios, etc, and especially spare atoms. There should be 3 spare atoms with at least one in the rack updated and running. There was a discussion on how best to install node specific configs quickly when needed. Using the rear/duply imaging is a bit cumbersome for this application. Might be easiest to configure on the fly, James says it takes a couple hours. There should be detailed instructions on the shiny new confluence wiki so that James isn't the only one able to do it.

CVTN/CVMS. A contractor, A2H, was hired to design and build the new school. They're aware we're there and are willing to work with us to restore those stations when the time comes.

MPHB continues to hang; it's an old serial to inernet DM24 system. The sensor hasn't been calibrated in three decades and has been in the borehole for nearly two of them. It is not being archived at IRIS and there is no metadata (Mitch has the original Guralp documentation though it is likely no longer accurate). We discussed whether to turn it off. It doesn't cost anything though isn't useful either. It is a nice borehole in a very noisy location so also not particularly useful from a monitoring perspective. It can be obsolesced as time permits.

Both UALR pipe vaults were full of water. Will replace them with a CERI style surface vault.

The City of Cape Girardeau sold the building housing sm station CBMO. Mitch has been unable to get in touch with the real-estate agent. Dave discovered the buyer is a philanthropist possibly converting it to a museum. He'll periodically check on it when in the area to keep track of changes and establish a new agreement when the time comes.

The property owners at FVM are constructing a new tower. We're on a cell modem so don't really need the tower. They're moved out of the old building and are letting us stay there. Need to move the cell antenna from the tower to the building.

NMDown


Data taken from:
/dat1/seisadm/Working/NM/202409/20240930_225200_MAN
/dat1/seisadm/Working/NM/202409/20240929_091914_MAN
/dat1/seisadm/Working/NM/202410/20241002_120505_MAN

NM
Down
GSAR.HN*
TWAR -kudzu
NHIN.EH* & .HN*
LVAR
WADM - Intermittent dropouts
WADM.EHZ - see image

Misc Problems
MSAR - spikes - see image
RVAR -spikes - see image

AG
Down
WLAR.HH* &.HN*

Misc Problems
X702.HNN -spikes

ETSZ

FPAL keeps going up and down. Router or local internet?

There are several stations in ET with power issues. Need to assume power upgrades were not completed and do site visits to each one to assess current status.

ETDown

Data taken from:
/dat1/seisadm/Working/ET/202410/20241001_070650_MAN
/dat1/seisadm/Working/ET/202409/20240928_161320_MAN

Down
ASTN.HN*
ATTN
DSNC (Helene)
MGNC (Helene)
TVNC2
CCNC
LRVA
GMG.EHN - see image
VHTN.EHE - see image
DYTN.EHN - see image

The Dalton link is dead and we're currently using a cell modem. It works but it would be nice to get the Dalton link working again. Jim has made multiple attempts to contact them but they don't return calls.


GPS

GPS status page

All GPS stations except Crestview (tornado damage awaiting rebuild of school) are online and operational! Well done!


AG Network

BAT3 will become the permanent Batesville station and BAT1 and BAT2 will be removed. Though we note BAT3 is not actually supported by AG.

Dave Johnston will replace the router at HHAR.


FDL

The old FDL wiki is here: https://umwiki.memphis.edu/display/FDL/CERI+FDL

The new FDL wiki is here: https://uofmwiki.memphis.edu/books/ceri-fdl-migrated/page/ceri-fdl


MISCELLANEOUS

The seisnet group on umdrive is at http://umdrive.memphis.edu/mwithers/seisnet You will need your uuid and password.

The seisnet wiki uses confluence at https://www.atlassian.com/