Server | diyIT

FreeBSD server with ZFS storage and applications/service running in jails

OS / FreeBSD 14.0

FreeBSD 14.0-RELEASE-p4 diyIT amd64

FreeBSD patch & build

git clone --branch releng/14.0 --depth 1 https://git.FreeBSD.org/src.git /usr/src/

echo -e "include GENERIC\nident diyIT\nmakeoptions WITH_EXTRA_TCP_STACKS=1\noptions TCPHPTS" > /usr/src/sys/amd64/conf/diyIT

freebsd-update fetch

freebsd-update install

svnlite update /usr/src

cd /usr/src

cd /usr/src && make -j8 buildkernel KERNCONF=diyIT && make install kernel KERNCONF=diyIT

reboot

FS / ZFS

Multiple pools based on type of data and access requirements. Allows disks to spin down when not in use, increasing their life and reducing overall power consumption.

  1. zroot - 2 SSD 860 EVO mirror - fast disks, constant RW access, for root file system
  2. ship - 4 Enterprise Capacity raidz1 + log SSD 850 PRO - fast disks, constant RW access, for jails
  3. tank - 2 WD Red - slow disks, constant access, for fragmented torrent data
  4. base - 4 Desktop SSHD raidz1 - slow disks, infrequent access, for transitional video data
  5. data - 4 Barracuda Compute riadz1 + log SSD 850 PRO - high capacity, slow disks, infrequent access, for data and backups.

Key ZFS settings

# set for all zfs pools
zfs set atime=off 

# set for all zfs datasets with compressed data, like video
zfs set compression=off 
zfs set logbias=throughput 

zfs list -o mountpoint,recordsize,sync,logbias,compression,atime
Hardware

Motherboard - Gigabyte MD90

CPU x 2 - Intel E5-2689-V3 - 12 core, 2.5Ghz, 30MB cache

Memory x 24 - 8GB DDR4 2133MHz ECC

Storage Controllers

  • Intel (MB): C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (4 + 2 disks)
  • Intel (MB): C610/X99 series chipset sSATA Controller [AHCI mode] (4 disks)
  • LSI SAS9207-4i4e (PCIe): SAS2308 PCI-Express Fusion-MPT SAS-2 (4 + 4 disks)

Storage Disks

  • 4 x 8TB + 2 x 2TB
  • 2 x 256G + 2 x 512G
  • 4 x 4TB + 4 x 4TB
  • 1 x 8TB usb

Network Controller (MB integrated)

  • Intel: 2 x Intel(R) PRO/1000 PCI-Express Network Driver
loader.conf
sysctl.conf
freebsd-update.conf
OS / FreeBSD 12.0

FreeBSD 12.0-RELEASE-p12 r354847M kernel with FreeBSD 11.2-RELEASE mps(LSI SAS9207-4i4e) driver (to stop the kernel from crashing due to race conditions under load)
Applied the 344061, 344356, D20868, D20869 vnet related patches (to reduce crashes when deleting jails)

FreeBSD patch & build

first time

svnlite checkout https://svn.freebsd.org/base/releng/12.0 /usr/src/

svnlite checkout https://svn.freebsd.org/base/releng/11.2 /root/11.2/

cd /usr/src/sys/dev

mv mps mps.orig

cp -r /root/11.2/usr/src/sys/dev/mps .

echo -e "include GENERIC\nident diyIT\nmakeoptions WITH_EXTRA_TCP_STACKS=1\noptions TCPHPTS" > /usr/src/sys/amd64/conf/diyIT

cd /root

git clone https://git.ahlawat.com/diyIT/FreeBSD.git


first time and subsequent kernel updates

freebsd-update fetch

freebsd-update install

svnlite update /usr/src

cd /usr/src

Apply VNET patches

patch -C < /root/FreeBSD/patches/344061.patch

patch -C < /root/FreeBSD/patches/344356.patch

patch -C < /root/FreeBSD/patches/D20868.patch

patch -C < /root/FreeBSD/patches/D20869.patch

Test patch with -C option - remove to (re)apply patch

cd /usr/src && make -j8 buildkernel KERNCONF=diyIT && make -j8 installkernel KERNCONF=diyIT

reboot


Reference vnet

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235699
https://svnweb.freebsd.org/base?view=revision&revision=344061

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235684
https://svnweb.freebsd.org/base?view=revision&revision=344356

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234985
https://reviews.freebsd.org/D20868
exec_prestop="ifconfig epair0b -vnet ioc-$JAIL"

FreeBSD vnet buglist

Reference mps

https://bugs.freebsd.org/bugzilla//show_bug.cgi?id=224496

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235559

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237937

FreeBSD mps buglist

Notes:

Installation:

  • portsnap fetch extract // the ports directory is mount shared with the poudriere (pkgp) jail
  • during install create user "a" 1001 and user "p" 1000
  • ln -s /usr/local/bin/bash /bin/ # (for scripts searching for bash in /bin/bash)

Setup swap on ZFS volume

zfs create zroot/raw
zfs create -V 64G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/raw/swapvol
swapon /dev/zvol/zroot/raw/swapvol

Install ZSH instead

pkg install zsh zsh-autosuggestions zsh-syntax-highlighting git-prompt.zsh
git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ~/powerlevel10k
echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >>! ~/.zshrc

chsh -s /usr/local/bin/zsh
pw usermod -n a -G wheel -s /usr/local/bin/zsh
visudo and uncomment wheel

$ cat .zshrc
source ~/powerlevel10k/powerlevel10k.zsh-theme

# To customize prompt, run `p10k configure` or edit ~/.p10k.zsh.
[[ ! -f ~/.p10k.zsh ]] || source ~/.p10k.zsh

source /usr/local/share/zsh-autosuggestions/zsh-autosuggestions.zsh

source /usr/local/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
ZSH_HIGHLIGHT_STYLES[suffix-alias]=fg=cyan,underline
ZSH_HIGHLIGHT_STYLES[precommand]=fg=cyan,underline
ZSH_HIGHLIGHT_STYLES[arg0]=fg=cyan
ZSH_HIGHLIGHT_HIGHLIGHTERS=(main brackets pattern cursor)
ZSH_HIGHLIGHT_PATTERNS=('rm -rf *' 'fg=white,bold,bg=red')

#if [ -z "$TMUX" ]
#then
#    tmux attach -t TMUX || tmux new -s TMUX
#fi

Install ZSH with oh-my-zsh "powerlevel10k/powerlevel10k"

pkg install zsh
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting
git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/themes/powerlevel10k

set in ~/.zshrc: plugins=(git osx iterm2 docker zsh-autosuggestions zsh-syntax-highlighting)
set in ~/.zshrc: ZSH_THEME="powerlevel10k/powerlevel10k"

chsh -s /usr/local/bin/zsh
pw usermod -n a -G wheel -s /usr/local/bin/zsh
visudo and uncomment wheel

Packages:

  • pkg install nano bash bash-completion sudo wget rsync git git-extras rpl xorriso htop nut smartmontools zfs-stats zfsnap debootstrap node_exporter py37-prometheus-client

Configurations:

  • crontab -l
00 08,12,16,20 *   *   *   /root/FreeBSD/scripts/zfs_health.sh
00   2    *   *   0   /usr/local/sbin/zfSnap -d -s -S -a 2w -p weekly_ -r zroot ship data base
00   3    *   *   6   /root/FreeBSD/jails/jails-update-pkgs.sh pkgp-only

Updates

  • run - update.sh - this updates the jails and upgrades the packages
  • run - jails-update-pkgs.sh - this only upgrades the packages
Showcased here is a capital and operational cost effective approach, using minimal server and networking hardware with multiple virtualized applications for Home and Business. This solution template can be easily scaled out and adapted for larger Enterprise deployments.
drop by the diyIT Matrix public room at #diyit:matrix.ahlawat.com
if you have any IT questions/feedback or to request pro bono consulting for a nonprofit

message me privately at @sharad:matrix.ahlawat.com
or email me at - sharad@ahlawat.com - pgpkey: 68DD6B89
Networking and Security Technologist.
Engineer and an avid Programmer.

https://sharad.ahlawat.com
strive to learn and pass on the knowledge to the next generation
one day humanity will understand the meaning of life and hopefully it will be more than ASCII 42 = "*" regex for whatever you want it to be,
and destiny is more than just a roll of a pair of dice with 42 dots (Lets nail down Quantum Entanglement)

May you Live Long (Intelligently) and Prosper and work on technology that matters.
© 2024 Sharad Ahlawat

No personal identifying data is collected or any form of analytics/metrics reported to a third-party by this website.