⑥ Robustification
So here, we’ve got more of a smorgasbord of various “improvements”.
Not all of them will be useful to anyone… Myself included…
As some of these… improvements… grow it will make sense to split them out into their own sub-pages.
Package Installs #
We need to install some stuff…. Namely
- nfsclient packages
- lldpd
- shell tools
- monitoring and telemetry tools
- python3
- mailutils
apt-get install bwm-ng cpufrequtils cpuinfo cpuset libconfig9 \
libfreetype-dev libio-pty-perl libipc-run-perl libpng-dev \
libpng-tools libsnmp-base libsnmp40 libtime-duration-perl \
libtimedate-perl lldpad lldpd mailutils moreutils nfs-common \
prometheus-node-exporter prometheus-node-exporter-collectors \
python-is-python3 python3-pip smartmontools snmp \
snmp-mibs-downloader sockperf spi-tools tuna tcputils unzip
Apt will ask you to set up postfix.. The nuances and challenges of setting up and maintaining a mail exchanger are myriad and outside the scope of this document.
it’s cool if you don’t care… I won’t hold it against you. :)
NFSmount /backup #
In my environment, I have a fileserver separate from most of the infrastructure.
Personally, I find it helpful, to have a durable sharepoint I can store ’things for future me’ and have them reside on completely isolated hardware from the thing in question.
This practice has saved my ass more than once, not gonna lie. However, it’s not strictly a requirement.
Setting up, and maintaining an NFS server becomes nuanced as dependencies increase. Guidance for this is outside the scope of this document.
I create distinct lenses into the storage for each context within my network. such that NFS traffic to a specific segment of the environment needn’t traverse a firewall. so, each subnet has its’ own address for the local nfs server..
Your environment may have different needs, and you may choose to deal with this differently. YMMV.
LABEL=writable / ext4 discard,errors=remount-ro 0 1
LABEL=system-boot /boot/firmware vfat defaults 0 1
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup nfs rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.42.23:/Backups/Infrastructure/DNSServer /backup nfs rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.18.40.23:/Backups/Infrastructure/DNSServer /backup nfs rw,relatime,rsize=1048576,wsize=1048576 0 0
#10.0.2.23:/Backups/Infrastructure/DNSServer /backup nfs rw,relatime,rsize=1048576,wsize=1048576 0 0
lldpd #
I found a few sources of “documentation” that were somewhat helpful… So I’ll include 1 them 2 here 3…
So, for the most part, you just sorta install it and move on, but there’s definitely some interesting stuff to explore.
for i in `ls /sys/class/net/ | grep 'eth\|ens\|eno'` ;
do echo "enabling lldp for interface: $i" ;
lldptool set-lldp -i $i adminStatus=rxtx ;
lldptool -T -i $i -V sysName enableTx=yes;
lldptool -T -i $i -V portDesc enableTx=yes ;
lldptool -T -i $i -V sysDesc enableTx=yes;
lldptool -T -i $i -V sysCap enableTx=yes;
lldptool -T -i $i -V mngAddr enableTx=yes;
configure med location address country US
done
IP=`ip addr show eth0 |awk '/inet/ {print $2}'|sed -e 's/\/.*$//'`; lldptool -T -i eth0 -V mngAddripv4=${IP}
lldptool configure med location address county Travis country US country-subdivision Texas building Barn room Barn
lldptool configure med location address county Travis country US country-subdivision Texas building Haus room Office
I spent a fair bit of time dorking around with the information on the calomel.org1 site.
Specifically their network performance page 2.
There’s a lot of nifty stuff there.
However, you might not care too much about tweaking sysctls. This is a starting point tuning similar to what I’ve set up on proxmox. As time goes on I’ll tune this more apropriately for dns servers.
net.core.netdev_max_backlog=13888
net.core.rmem_max=16777216
net.ipv4.tcp_rmem=4096 131072 16777216
net.core.wmem_max=16777216
net.ipv4.tcp_wmem=4096 131072 16777216
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1
net.ipv4.tcp_fack=0
net.ipv4.route.min_adv_mss=536
net.ipv4.tcp_base_mss=1460
net.ipv4.tcp_min_snd_mss=536
Journald is the new-ish systemd logging mechanism..
It’s got some neat configuration knobs.
In our case, the most useful config here is to push syslogs elsewhere.
then setting storage as volatile.
This offloads a large about of unnecessary writes to that MicroSD card you got.
[Journal]
Storage=volatile
ForwardToSyslog=yes
MaxLevelSyslog=info
MaxLevelStore=info
Compress=no
SystemMaxFileSize=2M
SystemMaxFiles=200
SystemMaxUse=400M
RuntimeMaxFileSize=2M
RuntimeMaxUse=400M
RuntimeMaxFiles=200
MaxFileSec=1week
You might knock it….
Hey!
SNMPD
has been around forever🕸️! It’s 🦽old📜 and busted!
🪦 💀
Where’s the 🏎️ new hotness 🏁 monitoring 🔮 shit 🤖 ?
not wrong….
apt-get install snmp-mibs-downloader snmpd
agentAddress udp:161
sysLocation ['30.3295110,-97.7560740']
sysName coredns-01
sysContact Me <coredns@wolfspaw.com>
sysObjectID 1.3.6.1.4.1.50536.3.1
#sysServices 72
sysServices 76
rocommunity "w01f5paw" default
view default included .1
view default included .1.3.6.1.2.1
#view default included .1.3.6.1.2.1.25.1
proc lldpd
proc lldpad
proc sshd
proc coredns
disk / 10000
disk /var 5%
includeAllDisks 10%
load 12 10 5
# Arbitrary extension commands
#
#extend-sh test3 /bin/sh /tmp/shtest
master agentx
extend .1.3.6.1.4.1.2021.7890.2 hardware /bin/cat /sys/devices/virtual/dmi/id/product_name
extend .1.3.6.1.4.1.2021.7890.3 vendor /bin/cat /sys/devices/virtual/dmi/id/sys_vendor
extend .1.3.6.1.4.1.2021.7890.4 serial /bin/cat /sys/devices/virtual/dmi/id/product_serial
extend uptime /bin/cat /proc/uptime
extend .1.3.6.1.4.1.2021.7890.1 distro /usr/local/bin/distro
# This line enables Observium's ifAlias description injection
pass_persist .1.3.6.1.2.1.31.1.1.1.18 /usr/local/bin/ifAlias_persist
# Allow Systems Management Data Engine SNMP to connect to snmpd using SMUX
smuxpeer .1.3.6.1.4.1.674.10892.1
Timesyncd #
[Time]
NTP=timey.wolfspyre.io wimey.wolfspyre.io wobly.wolfspyre.io tick.wolfspyre.io tock.wolfspyre.io
#FallbackNTP=ntp.ubuntu.com
RootDistanceMaxSec=4
PollIntervalMinSec=16
PollIntervalMaxSec=64
disable ipv6 #
place this in
/etc/sysctl.d/9933_WPL_DISABLE_ipv6.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1