Doing my cloud exit (kind of)

In a previous post, I shared how I migrated my personal infrastructure to Kubernetes. That transition also involved a shift in underlying hardware. In this post, I’ll explain why I ultimately decided to return to a single baremetal server.

Previous setup: scattered VMs #

While I have largely cut down on running personal and public services in the past few years, I still have a few things to run. I’ve hosted my services on dozens of different providers over the past 10 years, and right now my things are scattered around in various VMs and cloud providers.

For various reasons, I decided recently that I wanted to consolidate this infrastructure as much as I could, in a similar way to what I did a few years ago when I was running everything on a single host, first in Docker containers, then in LXD containers. As explained in my last post, I decided to use Kubernetes, more precisely the k3s distribution, which works nicely for a single-node setup, although this wasn’t my first intention.

My needs #

I need to run a few services, including stateful ones that need volumes and various databases (PostgreSQL, MariaDB/MySQL, Redis, Elasticsearch).

I estimate my overall resource needs as follows:

• About 8 CPU cores, although this would vary widely depending on the CPU performance
• At least 24 GB of RAM
• At least ~150 GB of storage, especially for mstdn.io’s DB which is ~80 GB right now
  • Ideally, the storage should be as fast as possible, NVMe-based, as the impact on database performance is pretty big

Now, I could try to find a single server that matches these criteria, but since I was planning on k8s, I initially considered a multi-node architecture.

Managed k8s offerings #

I started looking around at managed k8s offerings on B2C-friendly cloud companies. Unfortunately, and, I must say, as I expected, they are quite out of my budget.

Here are a few of them with my required resources. I tried to choose the cheapest node possible, meaning no compute-optimized/premium nodes.

These would be my go-to, but out of curiosity I checked more offers:

• Upcloud is free for the control plane, then for 3 nodes of 2CPU, 8GB RAM 80GB storage would be ~75€
• I checked out GCP’s GKE out of curiosity, I can get the compute down to ~$150, but I don’t even want to look at the price of databases 😄
• Civo would be at least $150 as well
• There’s Rackspace Spot where you bid a price for a desired config, which seems a bit crazy

So, overall, managed offerings are not cheap. To that, we would need to add other managed services such as Load Balancers, and ideally managed databases, the latter being usually very expensive (at least 50€ for my use case). Also, volumes.

Well, to be fair, I didn’t expect to find a managed offer that would suit my budget 😄

Back to Hetzner #

So, I dropped the managed k8s idea and decided to self-host.

The only cloud provider with accessible pricing that I know of right now is Hetzner Cloud. I have been a happy customer since 2018.

Multi-node setup #

I started experimenting with multiple k3s nodes and was able to confirm this setup would work for my needs… in theory.

The idea is that I’m going to host everything myself, including databases. Hetzner Cloud doesn’t have a lot of managed services, but offers the cloud primitives that would make this work: VMs, volumes, private network, and load balancers.

Now, looking at the VMs, I would need at least 3 VMs with 8 GB of RAM, probably 4.

Currently, Hetzner has 4 types for compute:

• Shared vCPU
  • Intel
  • AMD
  • Ampere
• Dedicated vCPU
  • AMD

The dedicated AMD option is out of my budget, so I’ll exclude it.

In terms of pricing, Ampere is slightly cheaper than Intel, and AMD is twice as expensive.

When we look at benchmarks, we get the price difference between Intel and AMD!

That doesn’t mean that Intel in general performs worse than AMD, it’s just that they probably chose more recent and powerful CPUs for the AMD offerings. It might also be less overprovisioned?

The Ampere instances are the real outlier here. We get similar performance to the AMD CPU, with a price cheaper than Intel! Considering the good availability and compatibility for ARM nowadays, it seems like a very good deal to me. We are far from the days of Scaleway’s C1 and ARMv7!

Although not strictly necessary, I would like to use a load balancer as well, which adds ~7€.

Then I would like volumes.

The problem with volumes is that you cannot provision less than 10 GB. At €0.0528/GB/month, that means at least 50c for each volume, which is not much but adds up quickly. I need a volume to store my Redis dumps, a volume to store a tiny SQLite database, etc., which use only a few megabytes, not 10 GB.

But the deal breaker is volume performance: 5000 IOPS is quite low! It’s orders of magnitude less than what you could get on NVMe SSDs. My quick pgbench benchmarks showed that performance really suffers from this, which is very unfortunate. I decided to drop this approach considering this.

Overall, I estimate that this setup would cost me around 35 to 40€. It’s definitely not bad and much better than the offerings I was discussing earlier. But can I get a simpler setup with similar pricing and better performance?

Single-node setup #

Let’s consider this. I can live without HA, which means I could make it work on a single-node k8s cluster. This means I can drop the volumes and load balancer, which would make the bill lower. Using the local storage of the VM would provide more performance as well. And the whole setup would be easier to manage, backup, and migrate if needed.

Looking back at the offerings, I would need one of the 32 GB RAM offers. They all have 16 vCPUs.

• CX52 - Intel : 38€ (136 multi-core points/€)
• CPX51 - AMD: 65€ (140 multi-core points/€)
• CAX41 - Ampere - 28€ (329 multi-core points/€)

Looking at the pricing, I’m of course more interested in the Intel and Ampere offerings, which have a much more interesting price-to-performance ratio (2.2x more interesting!)

I think the CAX41 would be a good choice for my use case.

But at that point, I was curious if a baremetal option would fit my use case.

Baremetal #

Unfortunately, the baremetal offering in the <100€ price range is pretty depressing.

It’s very hard to find:

• modern CPUs
• modern RAM (DDR4/5)
• NVMe storage

in the entry range.

A lot of providers such as Scaleway (both Dedibox and Elastic Metal), Leaseweb or OneProvider offer completely outdated hardware in that range such as 10 to 15 year old CPUs, DDR3, old HDDs or SSDs configurations. It’s very surprising.

Even Vultr, which has some pretty powerful, albeit pricey offerings for their Baremetal Compute, has a funny outlier: their cheapest offer is powered by a Xeon from 2011, for only $120/mo 😄

OVH has some pretty interesting offerings in their ECO range, which they consolidated from their Kimsufi and SoYouStart brands. You can find some decent configurations for the price, including not-so-old CPUs and NVMe storage. The downside is that availability is poor (although not as bad as it was a few years ago) and there is a setup fee equivalent to the price of the server.

That leaves us with… Hetzner, as you might have guessed. No wonder they’re the community’s favorite pick!

For my price range, they have the EX line, with Intel CPUs and the AX line with AMD CPUs.

The AMD line is a bit more expensive. Both lines start with desktop-grade CPUs (Intel Core and AMD Ryzen) and have server-grade CPUs for the more expensive SKUs (Intel Xeon and AMD EPYC). It seems that the AMD Ryzen offers come with DDR5 ECC memory, while the current Intel Core i5 offer doesn’t (but supports it).

These offers are great, but a bit too powerful and expensive for what I’m after.

Hetzner has a Server Auction which offers older, refurbished servers for a cheaper price. Most servers don’t have a setup fee (unless they are very recent) and are billed by the hour, which is very handy.

Currently, the cheapest offers seem to feature:

• Intel Core i7-6700 (2015)
• Intel Core i7-7700 (2017)
• Intel Core i7-8700 (2018)
• Intel Core i7-6700 (2015)
• Intel Xeon E3-1275 v5 (2015)
• Intel Xeon E3-1275 v6 (2017)
• Intel Core i5-12500 (2022!)
• AMD Ryzen 2600 (2019)

All for the same price, which is quite funny.

Most of these CPUs are powerful enough, although some are getting a bit old.

The outlier here is the i5-12500, which is very recent. This was the EX43 server. The current EX44 has an i5-13500.

As I said, all of these have the same price, so I picked the “ex” EX43:

• Intel Core i5-12500 (6 performance cores, 12 threads)
• 64 GB of non-ECC DDR4
• 2 × 512 NVMe
• 44€/mo

The nice thing about Hetzner is that they also now offer unlimited traffic for the 1 Gbps port, so one less thing to worry about.

Now, unsurprisingly, the CPU, although “only” an i5, completely destroys the vCPU in Hetzner Cloud, offering 3x (!) the single core performance of their Intel offers and 2x of the AMD offers.

I was very impressed by the performance, and while the CAX41 would have been enough, I decided to go that route. I didn’t choose OVH, as their offerings are slightly less interesting and have a setup fee.

I pay by the hour, which means that if I find a better deal elsewhere, or in the server auction, I can just move. In that regard, I find the flexibility that I liked with rented VMs. I wish we saw more baremetal offerings with a cloud-like management and billing. Hetzner is almost there, but at least they’re clearly the better deal!

I say almost because while the billing is by the hour, their dashboard has clearly not entered the cloud era :D

I’m not sure if the dashboard for the non-auction server is different. It does the job though! To be fair, I don’t really need to look at it once the original setup is done. The main annoying thing for me is that there is no way to see how much you’re paying.

I really wish we could see Cloud Baremetal offerings, but without the considerable markup there is everywhere.

Happy on baremetal #

I talked about my k3s setup in the previous post, so let’s talk about the machine.

In terms of setup I kept it simple. I fought off my desire to use ZFS and kept a simple RAID 1:

root@silo ~# mdadm --detail /dev/md2
/dev/md2:
           Version : 1.2
     Creation Time : Sat Feb 15 11:15:50 2025
        Raid Level : raid1
        Array Size : 498662720 (475.56 GiB 510.63 GB)
     Used Dev Size : 498662720 (475.56 GiB 510.63 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sat Mar  1 16:20:46 2025
             State : active
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : rescue:2
              UUID : 75ea42e6:2b3c9462:a6f4684f:06a110b2
            Events : 1220

    Number   Major   Minor   RaidDevice State
       0     259        7        0      active sync   /dev/nvme1n1p3
       1     259        4        1      active sync   /dev/nvme0n1p3

The NVMe SSDs are simply flying 😄

Now, I have to say, I was biased by the vCPUs and considerably overestimated my CPU needs.

The VM in the screenshot was only running mstdn.io, my biggest workload, but not the only one. The baremetal server is running everything, yet uses only a fraction of the CPU! We can also see a drop in CPU usage on the 28th when I enabled the performance CPU governor. My average CPU usage is now 10%…

Another interesting thing was that I estimated the big variance in CPU usage throughout the day was due to variation in usage and timezones. I wonder if the variation on the baremetal is less visible just because the CPU usage is lower, or if the variation on the VM hid overprovisioning and CPU guest usage.

In terms of RAM, I’m currently using about 32 GB (without cache). I could manage with less than 64 GB, and could optimize further to fit into 32 GB.

At least, I don’t have much risk of having Sidekiq backlogs anymore.

In any case, there aren’t any more interesting deals currently in Hetzner’s server auction, but the good thing about my setup and their no-setup-fee-and-hourly pricing is that I can switch very easily if I find a better deal. I like it!

I will miss some of the VM features such as snapshots, though!

To cloud, or not to cloud #

Leaving the cloud sometimes makes sense. Clearly, for my use-case, managed cloud services don’t make financial sense. Managed compute would work, but as I explained, baremetal wins in performance and price. I also simplified my setup with k3s. So for me, it’s a big win! We’ll see how many years I’ll last on this setup 👀

For companies, I would be more hesitant. I’ve worked for companies fully on AWS and for companies fully on baremetal. One might underestimate the amount of time spent on managing “managed” services and writing Terraform… On the other hand, having worked on an object storage product, one might underestimate the failure rate of hard drives at scale… Pick your poison 😬