UCC Wheel Meeting on 18th April 2020

Wheel Meeting Minutes - Saturday 2020-04-18 14:00

VENUE: https://meetings.ucc.asn.au/

Meeting opened 14:06


  • Present
  • [TPG]
  • [BRD]
  • [BOB]
  • [CFE]
  • [333]
  • [MPT]
  • [MTL]
  • [NTU]
  • [SJH]
  • [LE@]


  • None


  • None

Schedule next meeting

  • Schedule/delegate reminders of next meeting
  • [TPG]: Monthly is working well
  • [BOB]: Saturday May 23rd suggestion for next meeting
  • Scheduled for 2020-05-24 14:00

Standing items (brief)


  • Done

Status Check: Regular Updates

  • eg. Debian Oldstable 9 "Stretch" --> Debian Stable 10 "Buster"
  • Discord-irc.ucc.asn.au could use a rebuild, it's Jessie
  • OcsInventory, uccmonitor (see https://wiki.ucc.asn.au/MissionControl) for an overview
  • Out-of-date servers:
    • meersau (stretch)
    • discord-irc (jessie)
    • gitlab ()
    • unisfa-koha (stretch)
    • Contact Blair, Chase, or Felix from UniSFA to arrange downtime
    • salmon (? container, hard to upgrade)
    • mooneye (working on it)
    • Dead disk, needs replacing or new machine STAT
    • mussel (stretch)
    • murasoi (stretch)
    • motsugo (stretch)
  • Tuesday 21 Apr upgrade event - "Let's Break UCC"

Status Check: Password/Key Rotations

Status Check: Backups

  • [NTU]: We should have done that offsite file-restore demo
  • Communal decision to remove off mollitz webcam-archives
  • Full disks
    • [TEC]: If we can clean the old warnings, include webcamrestore, we can get timely new uccmonitor alerts
  • Dead disks
  • Mollitz (Dell 2950) can't have disks larger than 2TiB, and is old, so we should look at replacing
  • Other options:
    • Motsugo could be replaced
    • Old Motsugo coule become the new backup server
    • Worst case: could deploy the Cisco UCS220 or Mudkip
    • Mudkip has plenty of space drive caddies
    • Cisco for user server?
  • ACTION: Get Cisco server to a workable state

New Matters

  • [BRD] allocating more memory to minecraft2019
  • [RME] & [333]: Migrate to magikarp?
    • [NTU]: Save to molmol NFS or Ceph? (fast to migrate); or
    • Save to magikarp local Optane dm-cache? (fast to access)
  • [333]: minecraft2019:/, currently hosted on loveday, is 128GB? Could be shrunk? Will take a while to migrate
    • Run MineOS and a bunch of minecraft servers: maybe 128GB is right-sized?
    • Check with [BRD] & [MDD] about the rootfs shrinking?
  • ACTION: [333] to migrate and look at shrinking system disk if necessary
  • [333] Discuss current security of remote management, and whether we need to lock down the subnet further (ie. no access from motsugo)
    • eg. some management controllers are positively ancient and are unpatched
  • See this article for an example of a nasty bug (hint: we have 3 HPs):
  • Wheel-only jumpbox or VPN? Log traffic to/from that subnet?
  • ipmiview tool is handy? (By SuperMicro)
  • ssh in to some of them - which ones to be decided. Probably runs dropbear
  • ACTION: [MTL] will spin up a jumpbnox VM for testing
    • ...but a physical host is better than a V<
  • [MTL] mailauesi - imaps, pop3s, submission: currently wheel-only logins
  • Live testing: imap works, smtp doesn't?
    • Submission replaces smtp
  • Deprecate secure.ucc.asn.au for people's mail clients
  • https:gitlab.ucc.asn.au/ucc-systems-ucc-ansible-soe
  • [MTL] maculatus - new home VM for flame
  • libc6:i386 installed, old flame MudOS driver runs
  • listenes via firewall rules to telnet port 4242, and many other ports (like ssh.ucc.asn.au)
  • possibly no one uses most of those anymore?
  • webtty
  • [MTL] mailfish - new home for mail services from mooneye and mailman
  • postfix off switch, similar to postgrey, stop accepting deliveries if AD or NFS is down
  • Remaining stuff on mooneye - DNS (rearchitect? [MTL], see below)
  • [MTL] Folding@Home
  • Windows Clubroom machines?
  • Try desktop: Christmas
  • WSL+OpenSSHd -> running chocolately
  • can rdesktop in via guacamole/maaxen?
  • ACTION: [MTL] to do above.
  • [TEC]: GPU Server
  • [NTU] Proxmox Cluster Upgrades and Updates
  • [CFE] We have a virtual cluster we can test upgrading Proxmox PVE v5 -> v6
root@magikarp:~# pveversion
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
  • [MPT] 4G backup uplink
  • Set up: https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-April/005323.html
  • Can be used for outgoing with source-based policy routing
  • Has CGNAT IPv4 /64 address?
  • Probably don't want to fail2ban on that if it all comes from the one IP address
  • Needs a VPN?
  • Wireguard?
  • ACTION: [MSH] proof-of-concept on murasoi?
  • TINC?

  • Cloudflare! All! The! Things! ?!?!?!?

  • Still Priority #1
  • [MTL] UCC webservers, etc (was: mussel)_
  • More SSL expiries coming in June
  • If only we can make a CNAME change to ucc.gu.uwa.edu.au?
  • letsencrypt might see our secondaries?
  • ACTION: [NTU][MTL][333] try it out and extend our expiries
  • Set up offsite ucc.asn.au nameserver
  • We've been using and developing iodine
  • [MPT] has moved his domains to Cloudflare Free tier
    • Using dynamic DNS, via Cloudflare API: token based, not ddclient?

Matters Arising Previously

ACTION: Annual Account Locking - if it has not happened by now, it's overdue

  • Are the rejoining member/password reset/new member account procedures going OK?
    • Committee: Make it an event!
    • Account Locking Stream in Charity UnVigil 2020-05-13 through 2020-05-16

ACTION: [333] to document remote management options for our critical servers

  • Initial document in place, further testing needs to be done
  • More details to be added
  • See /home/wheel/docs/RemoteManagement.org

ACTION: [333] to sort out iDRAC for Mooneye as a priority

  • Live demo for mudkip! HP iLO
    • ssh
    • vsp: Virtual serial port
    • systemctl enable getty@ttyS1
    • power reset
  • Mail server from scratch on it, and a point a ucc subdomain at it, by the end of this weekend (2020-03-22)
  • [333]: Played with EC2 instance, set up a DNS A record for it, and ran push.sh
  • Backed up mooneye:/etc/bind/domains to https://gitlab.ucc.asn.au/ucc-systems/ucc-domains.git, cloned it onto cloud-mooneye
  • Copied the named.config.local across
  • Commented out mooneye-specific parts (like LetsEncrypt stuff and other referenced secrets only on mooneye)
  • Tried running zonemake.py, only to find that it needed a package installed
  • Once package was installed, it still wouldn't work due to symlinks to more stuff on mooneye (like /home/other/www/members.conf)
  • [333] work more independently gitlab vs zonemake.py vs AD getent
  • [MTL] [NTU] separate authoritative and recursive - DJB was right!
  • [NTU] has been testing knot-DNS

ACTION: [333][THA][TEC] to buy 1TB SSDs for magikarp + mudkip

  • Passed by committee on 2020-10-04.txt
  • Austin Computers: ~$500 Each? MLC, not SLC/QLC https://.www.austin.net.au
  • Samsung-mz-76q1t0-2-5-1000-gb-serial-ata-iii-v-nand-mlc.html
  • But the MLC is a lie!

Meeting closed 17:04