https://pine32.be - © pine32.be 2026
Welcome! - 133 total posts. [RSS]
A Funny little cycle 2.0 [LATEST]

Search 133 posts with 55 unique tags

#1785100296

[share]

[ homelab | k8s | backup ]

I am still working on a hard drive mount that won’t melt, but in the meantime the software side is ready. To nobody’s surprise, even this backup server is running Talos Linux managed by Omni. It is a two-node cluster. One worker node is the ZimaBlade that has the hard drive attached, and the one control plane node is a virtual one that is running on my other Omni cluster. You can run Talos as just a normal container, but not when managed by Omni. So it is actually a full QEMU virtual machine, but still running as a pod. Working with this virtual control plane means that the worker node is 100% stateless (except for the backup drive, of course). This helps make my offsite backup extra robust, because I will not always have physical access to the worker.

Because the worker and control plane are not on the same network, everything will need to be networked through NetBird. Luckily, this is easily done with the System Extensions of Talos. So NetBird is running as an OS process, which allows Flannel and Kubelet to talk over the virtual network. So the nodes can be anywhere, and things will just work, even behind symmetric NAT where KubeSpan won’t work. In NetBird, the Bunker cluster (that is what I am calling this small backup cluster) is fully isolated and has only connections between the nodes and a one-way connection from the worker node to my NAS (pull-based backups).

Next, what is running on the cluster. To start, my usual monitoring stack, but no storage of the metrics this time. I am running the Victoria metrics agent to forward all metrics to my other Talos cluster that has a storage server running. For the backup itself, I am running a basic cronjob with rsnapshot, a wrapper tool on to of Rsync that uses Linux hard links for its ‘incremental’ backups. The folder structure looks like full-sized backups each day, but the hard links keep the storage usage low. Encryption is provided at the drive level using LUKS2. Again this just uses basic Linux components, so if I need to I can just plug the drive into my Linux laptop and recover all my files without any special tools. The transfer is pull based happens over SSH with read only permission. This makes it so my data stays safe (at least 1 copy) even if one of my servers gets compromised. And last, I am running a small DaemonSet with hdparm that makes sure the drive is configured correctly to spin down when idle. The ZimaBlade is passively cooled so the system is fully silent once the disk spins down.

#1784490529

[share]

[ homelab | hardware ]

I thought I was almost done with the hardware side of things. I am setting up another ZimaBlade (the same as my main NAS) with a single 5TB 2.5-inch HDD as an offsite backup (I will cover the software in a later post). I found a 3D-printed case online and all looked good—until I started my first backup test. It is a big one because it needs to transfer all the data the first time.

Things quickly got hot. My hard drive reached its max temp of 60°C and my CPU got to a toasty 95°C. That’s at the limit but manageable, and a backup this big only needs to happen once. But then I noticed the distinct smell of melting PLA plastic. The picture below shows the case with the hard drive removed (the drive was sitting right on top of it). The white and black parts are supposed to sit flush with each other. Back to the drawing board, I guess.

picture of a zimablade with a melty 3d printed case

#1783877625

[share]

[ homelab | hardware ]

I got 2 new (2nd hand, but new to me) Raspberry Pi’s 4 with 8GB ram each. Both have a heatsink “case” that keeps them under 70°C under full load. It’s not a full case which is nice because it doesn’t increase the footprint. But they still offer enough protection to throw them around on my desk without shorting out on something (their is a plate on the bottom as well. I first tried with some microSD cards but they are slow and flaky for running the OS, even when using high speed cards. So I got some USB to SATA adapters (with a too long cable) and some old SSDs. A lot more bulky but at least I can trust them. And it is not too bad when you add some 3D-printed feet and a cable tie.

My main plan is to get my CI (maybe also CD) in order. I want to provide my main projects (like this blog) with multi-architecture container images. Emulation of ARM was way too slow for my liking (over 40 minutes for a clean build!). So to native hardware we go. The 2nd Pi is still up in the air. Maybe a music streaming box attached to some speakers (if I can find some decent software for it). Or maybe just some extra compute, never hurts to be prepared.

2 Raspberry Pi with SSD

#1783447674

[share]

[ odin | dev ]

Odin 1.0 HYPE!! (Odin 2027 actually…)

Fantastic announcement via by gingerBill. Finally HTTP lib, looking forward to testing that. And the main thing is just stability, what is issue is that I have already ran into with the constant updates.

YouTube Thumbnail
Odin 1.0 Announcement
gingerBill

#1782844885

[share]

[ music | scrobble ]

The expansion of my music setup continues with the addition of Koito and multi-scrobbler. Koito is selfhosted scrobbler (like last.fm) and multi-scrobbler routes my scrobbles from multiple sources to multiple clients. In my current setup I have Navidrome and Spotify (for now) as sources, and Koito and Last.fm as clients. I as using Last.fm before and I will continue to use it but I don’t want to depend on it. That is why I also running Koito even though they effectively do the same thing. But Koito gives me full control over my data. The import process to combine all my history from last.fm and Spotify took some time but now I have my full history under my control inside Koito. My public instance is available at scrobble.pine32.be. Now I can just show people this link when they ask what music I listen to, they get real untainted data.

Cover Art
Music Takes You Back
Modjo

#1782332077

[share]

[ music | ai | k8s ]

Got done setting up AudioMuse-AI on my bare metal k8s cluster. For those who don’t know what AudioMuse-AI is, it is a sonic analysis tool with a web interface that can hook into your Navidrome library. This is something I have played around before and always was planning to create myself. But the community beat me to it, more time for other projects I guess. Analysis of my almost 200GB music library took about 12 hours. It was a nice workload for my cluster to show me what it got. It’s a workload that scales perfectly horizontally, it can consume any compute that you can throw at it. On my old NAS it would take days, if not a week.

Below is my current music map: a 2D projection of the track vector space color-coded by genre. TLDR, closer songs sound more simular.

audiomuse-ai music map

Cover Art
Satellite Type 2
Commix

#1781814478

[share]

[ music ]

Still expanding my music library almost at 7000 tracks currently, but it is getting harder to expand. A lot of music is simply not available for purchase in any way, only streaming. So if you really want those track you need to rip them from some streaming service. So no money for the publisher/artist. Kind of their own fault if people can get it legally. Luckily this album is just on Bandcamp.

Cover Art
Tittle Tattle
Nubiyan Twist • CHERISE

#1781553349

[share]

[ homelab | hardware ]

New NAS!!

My previous NAS was an old Fujitsu tower with two Seagate Barracuda 4TB disks running in a ZFS mirror (4TB is plenty of room for my needs). It used about 25 watts at idle. Now I upgraded to a ZimaBlade 7700 NAS kit running an Intel E3950 with 16 GB RAM. A decent jump in performance from 2 to 4 cores but still around the same generation and same amount of DDR3 memory. But I went to 8 watts at idle and it is way way way smaller. Going from a full-size tower to the almost smallest size a 2-disk NAS can be. And it is still low cost.

In terms of software, I wanted away from Proxmox. It was overkill for what I need. My NAS is purely a NAS, all applications run on the Kubernetes cluster. And even if I run any application I want OCI containers, virtual machines are useless to me. So the best option to me seems TrueNAS and it is running very smoothly the first couple of weeks. The setup was also a lot easier comparted to ZFS on Proxmox. And it is better managed and I have actual visibility on my drive’s health. Overall, a huge upgrade.

zimablade nas

#1780951065

[share]

[ golang | wasm ]

In post 1757190021 I talked about the special SQLite lib I am using based around embedded wasm. A couple of months back, the library made a move to a new architecture. No more wasm but still no CGO. The solution: transpile a wasm build of SQLite to pure go code and compile that into your library. This gives us more type safety and better maintainability because the wasm build can be used as a “normal” go module. Also the boot time for applications is better because there is no waiting for the wasm module to load and “boot”. One downside is a clean build getting slower because you are rebuilding SQLite fully (still faster than any Rust build). Also the resulting binary is a bit bigger. But this is still a valid trade off to have no CGO dependency. I will keep using this library.

The transpiler, wasm2go, is also available for general use. I want to try it with libvips for one of my projects. But for now SIMD is not supported, not until the go api for it is stable. I could also build libvips myself with SIMD disabled but that sounds like work and messing with C build systems. So I will just wait, I’m not in a hurry.

#1780255277

[share]

[ homelab | networking ]

I just realized I havent talked about NetBird on this blog. NetBird is the peer-to-peer private network that I use for everything in my homelab. It is similar to Tailscale if you know that. But the main difference is that NetBird is able to be self-hosted, everything is opensource. I know that Headscale is a thing for Tailscale but it is not official and you are still reliant on closed-source clients. And a nice bonus for me is that NetBird is a German/European company.

So for my setup I selfhost NetBird on my VPS based Kubernetes cluster, it needs to have publics IPs to work properly when selfhosting. Then every server and device gets a client to connect. OIDC based for user clients, very nice. I am running a DaemonSet on all my Kubernetes cluster with host networking enabled. So each pod is automatically able to resolve private IPs and with some extra CoreDNS config also the private DNS. These clients on the nodes also function as gateways so all pods and services are available to the private network (with the right RBAC of course). And with some DNS forwarding in NetBird configured I can also resolve hostnames from Kubernetes services on my laptop, for example mb.default.svc.cluster.local. The setup is bit more complicated when using multiple k8s cluster, maybe something for a full-sized blogpost, going over how my network is set up in detail.

#1779738968

[share]

[ music | homelab ]

I really need some form of music “release radar” in my homelab with my move away from Spotify. Forever Live Sessions Vol 2 by Men I Trust has been out for more then a year now! Volume 1 is one of my all-time favourite albums and volume 2 quickly joined that list. Less “fast music” also means that I notice new music less “fast” I guess.

Cover Art
Tree Among Shrubs
Men I Trust

#1779009248

[share]

[ dev | golang | git ]

Last week when moving over to Codeberg (see post 1778177701) I noticed that I am using the github.com domain in my module names. For my projects (like mb) that fine, I just moved it to codeberg.org when moving the repo. But for libraries I can’t expect everybody to change their go.mod file every time I change were I host my code (not that anybody uses my libs but just go with it). Luckily the Go devs have though of this.

When trying to download a module, like github.com/ncruces/go-sqlite3 for example, go will make a http/https request like shown below. The webserver should then provide a special go-import meta tag that where and how it can pull the code.

curl -s 'https://github.com/ncruces/go-sqlite3?go-get=1' | grep go-import
<meta name="go-import" content="github.com/ncruces/go-sqlite3
    git https://github.com/ncruces/go-sqlite3.git">

Most code forges will handle this for you. But you can also deploy your own server on your own domain that handles this. And that is exactly what I have done, setup my own vanity URL. I made a simple stateless webserver for this that just takes a JSON config, source. Currently it will redirect to Codeberg but when I move my repos again I just have to change a basic config file and all modules will keep working as they are supposed to. I just have to make sure that go.pine32.be doesn’t go down. But it is a stateless app running 2 replicas on a highly available Kubernetes cluster, should be fine.

curl -s 'https://go.pine32.be/id?go-get=1' | grep go-import
<meta name="go-import" content="go.pine32.be/id
    git https://codeberg.org/Pineapple217/go-id">

#1778177701

[share]

[ dev | git ]

Lately I was already using my Forgejo instance a lot more the GitHub. Mainly for my private projects or projects where I don’t expect/want contributions. I did this because of all the AI data scraping and I don’t trust GitHub anymore. But I haven’t moved my public projects yet because I didn’t want to push my already niche projects to an even more niche platform. But now that GitHub is struggling to keep actual things online (84.35% uptime over last 90 days as of today) the general public is also looking for alternatives. So for me it was finally time to move my code to Codeberg, MB (this blog you are reading now) being the first project to move. Codeberg e.V.is a German non-profit organization that provides a public Forgejo instance that only allows opensource software. So it checks all boxes for me. And yes, I already have donated. Hopefully it can cover some cost of the GitHub refugees.

#1777583272

[share]

[ homelab | k8s ]

Finally got around to setting up storage on my Dell OptiPlex k8 cluster using Omni. I abandoned the idea of using Rook-Ceph, way to complicated for my needs (maybe skill issue). So back to Longhorn it is. I have set up Longhorn a couple of times so that setup went smooth. But I first needed to get the disks partitioned and setup via Talos OS. That took way to long to figure out. But now I know how to do it and I have a fully declarative config for it, so it was worth it. The full cluster config is just 50 lines! The new disk overview in Omni was a nice help (see img). So now that everything is setup I have 3 TiB of schedulable (sata) ssd storage, so 1 - 1,5 TiB with replication usable. Should be plenty, I only need 200 GB for my music.

Omni node disk view

#1776978473

[share]

[ dev | golang ]

Echo, the web framework I mainly use, recently got a new major release, v5. A big one I like is moving from a custom logger to the default log/slog for the stdlib. Most other breaking changes or not that interesting on their own but they create more room for future expansions. A lot of feature requests where parked until v5 so I hope that we will get a lot of nice things the coming year.

// New type safe API for parsing
id, err := echo.PathParam[int](c, "id")
user, err := echo.ContextGet[*User](c, "user")

#1776283702

[share]

[ music | navidrome ]

All I play is you!! Oh every game I lose!!

Cover Art
Play
Hana Vu

I have been using Navidrome more and more. I think around 50% of my music listening is on Navidrome now. Still need to get an app that works with Android Auto. I still have a lot of music ready that I haven’t listened yet so the percentage will only go up.

When I am using Navidrome I also listen more to full albums. So aside from it being better because I actually own the music I think it also improves the way I consume music in a major way, less “fast music” (as in fast fashion or fast food). Fast music is still fun, but maybe not all the time.

#1775126093

[share]

[ chess ]

I was looking for interesting chess engines to test and I came across Patricia, the most aggressive chess engine ever seen. It is currently rated 3466 on Stefan Pohl’s website (compared to Stockfish 18’s rating of 3910). It is a very strong engine but probably wont ever be the best. But when it wins, it often does so in a brutal way. Bellow is a game played as part of a tournament against RofChade 3.1 with a respectable elo of 3536 (higher then Patricia but still close). The book stats with equal material with a slight position advantage for white. Patricia starts sacrificing early and doesn’t stop until a final rook sacrifice to start the push for mate. White is behind on material the entire game but still wins.

gif of chess game

Engine play starts at move 7

Download PGN

#1773945124

[share]

[ webdev ]

I am giving Tailwindcss one more change. Mainly because I want to try out templui for my next project. Seems handy to get up and running faster and it is a perfect match for HTMX. The setup is not the cleanest but it is working fine up to now. I am using bun to run the Tailwind CLI.

#1773268866

[share]

[ music | rock ]

Cause nothing ever changes

Cover Art
Nothing Ever Changes
Max Baby

#1772925659

[share]

[ mimir | chess ]

Mimir is up to version 1.2 in the mean time and available on Lichess. It is still not super strong but a fun challenge as a human if you give yourself enough time.

https://lichess.org/@/MimirBot

The setup was relatively simple using the lichess-bot adapter. Just plugin in my UCI engine and the API key.

#1772138346

[share]

[ mimir | odin | chess ]

Mimir V1 is done! The last commit for V1 was adding UCI, which took a bit of testing because it needs some small async things. But it is working now, and I immediate put it up against Stockfish 18. of course not the full version, I limited it using the UCI_Elo option to 1600 elo. Still feels like a fair starting point for Mimir’s first game ever. And it won convincingly with no major mistakes. This is of course not a full evaluation, but now that it works with UCI, I can start building my full test suite to get a real Elo value for this first version. This is of course still the start!

gif of chess game

Download PGN

#1771316193

[share]

[ mimir | odin | chess ]

I have been working again on my chess engine writing in Odin, Mimir. It is still far from finished but it is now able to complete a full perft test. This basically means that is able to generate all valid moves (and only valid moves) in every position given. Those move results are validated against known move counts so it is 99,99999 % correct. Next up is doing minimax algorithm and some board eval function. Then it should be able to play its first game. Will also write some more about Odin as I am getting used to it.

#1770402595

[share]

[ homelab | k8s ]

One of my Kubernetes nodes recently went down without me noticing (issue with VPS provider, not my fault). It was nice that all my services stayed online without any hiccups. But I would like to know in the future when my infra goes down. So time to get the classic stack online Grafana (now with build in Alertmanager), Node Expoter, CAdvisor, kube-state-metrics but no Prometheus this time. I am running VictoriaMetrics instead. It is a drop in replacement for Prometheus and is still written in Go but it is made to be more cost effective. It uses 7 times less memory, CPU and storage then Prometheus. And from my testing those numbers seem to be right. Currently I am storing 28.6 billion datapoints using just 4.37 GiB of storage. Below you can see the stats (for everything monitoring) with a current ingest rate at around 20K rec/s.

Resource consumption

#1769462323

[share]

[ homelab | k8s ]

I finally got S3-compatible buckets online, using a Hetzner storage box as the storage. I know they got S3 buckets as well but I am cheap and it will only be used for backups so I don’t care about speed.

For the setup I first used the Kubernetes SMB CSI driver to be able to use my storage box as regular PVC’s. On top of that I am running the Versity gateway. It is a completely stateless S3 server that can use a normal posix filesystem for the storage. It stores everything as plain files, so even without the gateway my files are still fully accessible. I could also spin up a gateway instance wherever I need one, even if my k8 cluster goes down. And I can run it in high availability on my cluster. All these things makes it perfect for backups. And it has nice web interface as a bonus. And of course it is written in Go.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: versitygw-s3
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: smb
  resources:
    requests:
      storage: 30Gi

#1768730464

[share]

[ music ]

I finally have more songs in my offline library then on Spotify. 5859 in my own lib and 5856 on Spotify. I still can’t fully replace Spotify yet but getting closer everyday.

Cover Art
obsessed
wenszy