Wheel Meeting Minutes - Saturday 2020-04-18 14:00 ================================================= VENUE: https://meetings.ucc.asn.au/ *Meeting opened 14:06* ## Attendance - Present - \[TPG\] - \[BRD\] - \[BOB\] - \[CFE\] - \[333\] - \[MPT\] - \[MTL\] - \[NTU\] - \[SJH\] - \[LE@\] ### Apologies - None ### Absent - None ## Schedule next meeting - Schedule/delegate reminders of next meeting - \[TPG\]: Monthly is working well - \[BOB\]: Saturday May 23rd suggestion for next meeting - Scheduled for 2020-05-24 14:00 ## Standing items (brief) ### SWS - Done ### Status Check: Regular Updates - eg. Debian Oldstable 9 "Stretch" --> Debian Stable 10 "Buster" - Discord-irc.ucc.asn.au could use a rebuild, it's Jessie - OcsInventory, uccmonitor (see https://wiki.ucc.asn.au/MissionControl) for an overview - Out-of-date servers: - meersau (stretch) - discord-irc (jessie) - gitlab () - unisfa-koha (stretch) - *Contact Blair, Chase, or Felix from UniSFA to arrange downtime* - salmon (? container, hard to upgrade) - mooneye (working on it) - *Dead disk, needs replacing or new machine STAT* - mussel (stretch) - murasoi (stretch) - motsugo (stretch) - Tuesday 21 Apr upgrade event - "Let's Break UCC" ### Status Check: Password/Key Rotations - DEFER - https://en.wikipedia.org/wiki/Pro_re_nata ### Status Check: Backups - \[NTU\]: We should have done that offsite file-restore demo - Communal decision to remove off mollitz webcam-archives - Full disks - \[TEC\]: If we can clean the old warnings, include webcamrestore, we can get timely new uccmonitor alerts - Dead disks - Mollitz (Dell 2950) can't have disks larger than 2TiB, and is old, so we should look at replacing - Other options: - Motsugo could be replaced - Old Motsugo coule become the new backup server - Worst case: could deploy the Cisco UCS220 or Mudkip - Mudkip has plenty of space drive caddies - Cisco for user server? - ACTION: Get Cisco server to a workable state ## New Matters - \[BRD\] allocating more memory to minecraft2019 - \[RME\] & \[333\]: Migrate to magikarp? - \[NTU\]: Save to molmol NFS or Ceph? (fast to migrate); or - Save to magikarp local Optane dm-cache? (fast to access) - \[333\]: minecraft2019:/, currently hosted on loveday, is 128GB? Could be shrunk? Will take a while to migrate - Run MineOS and a bunch of minecraft servers: maybe 128GB is right-sized? - Check with \[BRD\] & \[MDD\] about the rootfs shrinking? - ACTION: \[333\] to migrate and look at shrinking system disk if necessary - \[333\] Discuss current security of remote management, and whether we need to lock down the 192.168.2.0 subnet further (ie. no access from motsugo) - eg. some management controllers are positively ancient and are unpatched - See this article for an example of a nasty bug (hint: we have 3 HPs): - https://it.slashdot.org/story/19/07/08/007202/critical-bug-last-year-allowed-bypassing-authentication-on-hpe-ilo4-servers-with-29-a-characters - Wheel-only jumpbox or VPN? Log traffic to/from that subnet? - ipmiview tool is handy? (By SuperMicro) - ssh in to some of them - which ones to be decided. Probably runs dropbear - ACTION: \[MTL\] will spin up a jumpbnox VM for testing - ...but a physical host is better than a V< - \[MTL\] mailauesi - imaps, pop3s, submission: currently wheel-only logins - Live testing: imap works, smtp doesn't? - Submission replaces smtp - Deprecate secure.ucc.asn.au for people's mail clients - https:gitlab.ucc.asn.au/ucc-systems-ucc-ansible-soe - \[MTL\] maculatus - new home VM for flame - libc6:i386 installed, old flame MudOS driver runs - listenes via firewall rules to telnet port 4242, and many other ports (like ssh.ucc.asn.au) - possibly no one uses most of those anymore? - webtty - \[MTL\] mailfish - new home for mail services from mooneye and mailman - postfix off switch, similar to postgrey, stop accepting deliveries if AD or NFS is down - Remaining stuff on mooneye - DNS (rearchitect? \[MTL\], see below) - \[MTL\] Folding@Home - Rolled out on Clubroom Desktops - https://stats.foldingathome.org/team/261452 - Ranked 11,724 of 251,202 - Windows Clubroom machines? - Try desktop: Christmas - WSL+OpenSSHd -> running chocolately - can rdesktop in via guacamole/maaxen? - ACTION: \[MTL\] to do above. - \[TEC\]: GPU Server - Bob raised on Facebook group: https://www.facebook.com/universitycomputerclub/permalink/3767237643318123 - \[DBA\] David Adams, Cursor XYZ - \[NTU\] Proxmox Cluster Upgrades and Updates - \[CFE\] We have a virtual cluster we can test upgrading Proxmox PVE v5 -> v6 ``` root@magikarp:~# pveversion pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve) ``` - \[MPT\] 4G backup uplink - Set up: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-April/005323.html - Can be used for outgoing with source-based policy routing - Has CGNAT IPv4 /64 address? - Probably don't want to fail2ban on that if it all comes from the one IP address - Needs a VPN? - Wireguard? - ACTION: \[MSH\] proof-of-concept on murasoi? - TINC? - Cloudflare! All! The! Things! ?!?!?!? - Still Priority #1 - \[MTL\] UCC webservers, etc (was: mussel)_ - More SSL expiries coming in June - If only we can make a CNAME change to ucc.gu.uwa.edu.au? - letsencrypt might see our secondaries? - ACTION: \[NTU\]\[MTL\]\[333\] try it out and extend our expiries - Set up offsite ucc.asn.au nameserver - We've been using and developing iodine - \[MPT\] has moved his domains to Cloudflare Free tier - Using dynamic DNS, via Cloudflare API: token based, not ddclient? ## Matters Arising Previously #### ACTION: Annual Account Locking - if it has not happened by now, it's overdue - Are the rejoining member/password reset/new member account procedures going OK? - Committee: Make it an event! - Account Locking Stream in Charity UnVigil 2020-05-13 through 2020-05-16 #### ACTION: \[333\] to document remote management options for our critical servers - Initial document in place, further testing needs to be done - More details to be added - See /home/wheel/docs/RemoteManagement.org #### ACTION: \[333\] to sort out iDRAC for Mooneye as a priority - Live demo for mudkip! HP iLO - ssh - vsp: Virtual serial port - systemctl enable getty@ttyS1 - power reset - Mail server from scratch on it, and a point a ucc subdomain at it, by the end of this weekend (2020-03-22) - \[333\]: Played with EC2 instance, set up a DNS A record for it, and ran push.sh - Backed up mooneye:/etc/bind/domains to https://gitlab.ucc.asn.au/ucc-systems/ucc-domains.git, cloned it onto cloud-mooneye - Copied the named.config.local across - Commented out mooneye-specific parts (like LetsEncrypt stuff and other referenced secrets only on mooneye) - Tried running zonemake.py, only to find that it needed a package installed - Once package was installed, it still wouldn't work due to symlinks to more stuff on mooneye (like /home/other/www/members.conf) - \[333\] work more independently gitlab vs zonemake.py vs AD getent - \[MTL\] \[NTU\] separate authoritative and recursive - DJB was right! - \[NTU\] has been testing knot-DNS #### ACTION: \[333\]\[THA\]\[TEC\] to buy 1TB SSDs for magikarp + mudkip - Passed by committee on 2020-10-04.txt - Austin Computers: ~$500 Each? MLC, not SLC/QLC https://.www.austin.net.au - Samsung-mz-76q1t0-2-5-1000-gb-serial-ata-iii-v-nand-mlc.html - *But the MLC is a lie!* *Meeting closed 17:04*