Here are the Proxmox servers currently running listed side by side. PBS is the “Proxmox Backup Server,” which is automatically used as the backup destination by the three VM servers.

Hardware:

Firmware:

Software:

Screenshots:

Dokumente / Treiber

ULP-Server Drivers at Dell
LP-Server Drivers at HP
TR-Server Drivers at Gigabyte
PBS Drivers at HP

Der Beitrag <h5>Setup: </h5><h3><b>Proxmox Server</b></h3> erschien zuerst auf flohs blog.

The server has now been in use for some time as needed and has revealed its strengths and weaknesses. Overall, I am very satisfied, although Proxmox and Debian have sometimes been a bit finicky—but only when “fiddling” with them.

Positive aspects:

• The server is really quiet for a computer in this performance class, even under full load.
• Good performance. I run either five Windows 10 or Windows 11 VMs, each with a vGPU and 4 GB RAM, configured with 16 GB RAM and 6 cores.
• I left the RGB on the cooler activated because it sticks out of the top of the case.
• A very powerful system for the price.

Negative aspects:

• Power consumption is high, especially when idle, always consuming over 100 watts. At the limit it’s no surprise that it’s well over 800 watts.
• The rather weak motherboard limits a possible CPU upgrade to 32 cores with SMT (TR 2990WX).
• Nvidia vGPU is unofficial and has already broken several times during system updates. Very time-consuming.

Next time again?

For the current application, anytime. Next time, based on my experience, I would make different decisions on various matters:

• RAM expansion to the limit (even though 128 GB is easily sufficient at the moment), which would then be 256 GB
• Use ECC RAM, which is important for ZFS
• Replace the CPU cooler instead of messing up the case
• Possibly just leave out the GT 710 and run another NVMe SSD in the now free PCIe 2.0 x4 slot

I think that the way the “TR server” is currently configured, it will run unchanged for a long time.

Photos of the current status

Der Beitrag <h5>TR-Server #7: </h5><h3><b>Conclusion</b></h3> erschien zuerst auf flohs blog.

From the outset, I wanted a 4U case to ensure sufficient space. After deciding on a 4U-4129-L from Inter-Tech (called Rosewill outside of Europe)* and receiving it, I realized that the CPU cooler would not quite fit inside. Annoying.

Unusually for me, I opted for the most drastic solution: cutting the cooler out of the cover. This is only acceptable to me because there will be a shelf directly above the server, so the slight protrusion is not a problem.

Otherwise, I didn’t use any of the supplied fans with 4-pin Molex connectors and instead used Pure Wings 2 from BeQuiet* – two 120 mm in the front, three 120 mm in the middle, and two 80 mm in the back.

Der Beitrag <h5>TR-Server #6: </h5><h3><b>The Home of the Hardware</b></h3> erschien zuerst auf flohs blog.

All slots on the motherboard are occupied. The following graphics cards are installed:

⧉ Zotac

⧉ Palit

⧉ Palit

Additional

• A x4x4 bifurcation NVMe card for two additional SSDs
• An Intel X540-T2 network card* with two 10 Gbit ports via copper (RJ45, Dell)

⧉ Dell

Order

The cards are installed as follows:

⧉ Gigabyte

Der Beitrag <h5>TR-Server #5: </h5><h3><b>Expansion cards</b></h3> erschien zuerst auf flohs blog.

Thanks to the many available 3.0 lanes, an above-average number of NVMe SSDs with full bandwidth can be used. Due to the high prices at the upper end of the capacity range, I was keen to experiment with two SSDs and, when I saw a very good offer from Mindfactory at the time of purchase, I was tempted to buy something that was quite unfamiliar to me:

Two MS200s from Mega Electronics (MS200200TTM), a South Korean manufacturer like Samsung that uses Micron or Intel NAND. These PCIe 3.0 NVMe SSDs with 2 TB capacity were repeatedly offered at bargain prices for a while, and the data sheets look good: fast DDR4 DRAM cache, high TBW, good write and read performance – everything as desired.

I’ve read quite a few comments about “Mega Fastro Kaputto,” and the 1 TB version is said to have been prone to failure – but I haven’t noticed anything unusual with my model so far.

It’s important to mention that the MS200 is double-sided and therefore doesn’t fit into every M.2 slot. You have to be careful with mobile devices or expansion cards, and this also plays a role in the compatibility of passive heat sinks. For example, only single-sided M.2 SSDs fit on the expansion card I use.

NVMe-SSDs

• Two MS200 with 2 TB each from Mega Electronics, on the motherboard under the passive heat sinks of the board
• Two Samsung 970 Evo Plus with 2 TB each*, on an x8 expansion card (x4x4 bifurcation*) including JEYI passive heat sinks
• One Samsung 970 Evo Plus with 500 GB*, also on the motherboard

⧉ MEGA Electronics

⧉ Samsung

⧉ JEYI

SATA-SSDs

• Three Samsung SM863a Enterprise SSDs, each with a capacity of 480 GB
• One Samsung 840 with 250 GB for a parallel installation of Windows 11 Professional

⧉ Samsung

⧉ Samsung

HDDs

• Two WD Red drives, each with 4 TB*, 3.5″, SATA – RAID1
• Two Toshiba drives, each with 2 TB*, 3.5″, SATA – RAID1
• One Seagate Barracuda with 500 GB, 2.5″, USB 3.0 internal data HDD for Windows 11
• One Toshiba with 1 TB, 2.5″, USB 3.0 internal – Images and SMB

⧉ WD

⧉ Toshiba

⧉ Sabrent

Der Beitrag <h5>TR-Server #4: </h5><h3><b>Storage – Ola NVMe!</b></h3> erschien zuerst auf flohs blog.

Once the basics are clear, RAM must be procured. ECC would be possible, but it is significantly more expensive and harder to find than normal, unbuffered RAM. So “normal” RAM is used.

The maximum expansion with four memory channels and eight slots would be eight DDR4 modules, each with a capacity of 32 GB. That would give a whopping 256 GB. Desirable, but not financially feasible at the time of my search.

Corsair Vengeance LPX

Because price/performance also plays a role, we opted for DDR4 modules with 16 GB each*, giving us a total of 128 GB for all eight. This means that the four memory channels are fully utilized and the memory controller is not overloaded.

⧉ Corsair

⧉ Corsair

⧉ Corsair

Power supply

I was able to purchase two HX1200s* at a bargain price from a miner in Bavaria. The second one, which has not been used until now, will now be used in this server (the other HX1200 is in “Tamy-PC”). Full accessories, very good condition.

⧉ Corsair

⧉ Corsair

⧉ Corsair

⧉ Corsair

⧉ Corsair

⧉ Corsair

Der Beitrag <h5>TR-Server #3: </h5><h3><b>Memory and PSU</b></h3> erschien zuerst auf flohs blog.

Due to personal preference and a bit of coincidence, I ended up with a first-generation AMD Threadripper, complete with motherboard and cooler. I would describe the purchase price as “unignorable low”; I bought it second-hand via “Kleinanzeigen”.

A old Threadripper wasn’t my first choice, but it was up there. The power consumption, especially when idle, is said to be quite high, which is probably due to the multi-die design of these CPUs. On the positive side, there are plenty of PCI Express 3.0 lanes. The bundle consisted of the following components:

Processor

⧉ AMD

⧉ TPU

⧉ TPU

CPU Cooling

With an emphasis on appearance, the bundle included a powerful Wraith Ripper processor cooler from CoolerMaster. Only compatible with the TR4 socket, it was released specifically for the launch of the second generation Threadripper. I would have bought something flatter (and without RGB) from Noctua, BeQuiet, or Thermalright, but that’s just how it turned out.

⧉ CoolerMaster

⧉ CoolerMaster

⧉ CoolerMaster

Mainboard

I’m a little unhappy with this one – it’s turned out to be an RGB-overloaded Christmas tree, an AORUS Gaming 7 from Gigabyte. Not really the right choice for use in a rack case. From a technical point of view, it will work fine, but I’m turning the RGB off completely.

However, the VRMs are quite weak, so the 1950X (or its successor, the 2950X) is the best you can hope for. With the new UEFI version, second-generation Threadrippers, some of which have a significantly higher TDP (32 cores and 64 threads with the 2990WX -> 250 watts TDP), should also work in theory. Overclocking is not recommended with this board, with any Threadripper variant for TR4. The power supply and its cooling are too weak.

If I had had the choice, I would have gone with MSI or ASUS, as I usually do. Nevertheless, the Gigabyte board is still well suited for its intended purpose.

⧉ Gigabyte

⧉ Gigabyte

⧉ Gigabyte

Let’s move on to the other components, starting with the Memory.

Der Beitrag <h5>TR-Server #2: </h5><h3><b>The Base</b></h3> erschien zuerst auf flohs blog.

After two energy-efficient servers handled the basic services, I wanted a way to perform powerful virtualization of desktop operating systems, especially Windows. The idea of having multiple Windows VMs with functioning vGPUs from Nvidia in order to offer guests suitable places even without compatible Windows was very tempting.

In order to get the most out of the costs, a system with as many PCI Express lanes as possible at a low price makes the most sense. The reason for this is obvious, as everything required runs via PCI Express:

• NVMe SSDs
• Expansion cards for SATA / HBA
• 10 Gbps network card(s)
• Graphics card(s)

The only options here are Intel Xeon from the workstation level up, AMD Threadripper or AMD Epyc.

Workstation?

I wanted to avoid a used OEM workstation and go the DIY route for various reasons. For one thing, these OEM motherboards are always only “as much as necessary” and for another, many of the components used in them are no longer standard-compliant and therefore difficult to maintain/replace/retrofit. Overclocking would also not be possible, but that would not be necessary in this case anyway.

Let’s continue with the base.

Der Beitrag <h5>TR-Server #1: </h5><h3><b>A high-performance server for Proxmox is needed</b></h3> erschien zuerst auf flohs blog.

For me, VMware ESXi was my introduction to the world of virtualization servers a few years ago. This is obvious, as I was and still am an avid user of VMware Workstation on Windows.

When I later read about Proxmox, I was immediately interested:
The lack of restrictions on free use and the developers’ business model are simply better than VMware ESXi. Especially since VMware has since been taken over by Broadcom and the circumstances have deteriorated further. In the end, open source is always the best option.

In order to be independent of the widespread cloud mania, I eventually assembled an “experimental” Proxmox server on my benchtable from discarded components and two identical hard drives, including a boot SSD. Everything worked right away without any problems, even though I had little specific prior knowledge.

After “playing around” a bit, it was clear that I wanted to build and operate a 24/7 Proxmox server.

Which direction should we go in?

Mini PCs, which come in many shapes and designs are among the best hardware for use in a private environment. I recommend refurbished models from Lenovo, Dell, HP, and similar large OEMs. The software support is simply better compared to no-name Far East variants, and the build quality is consistently good. In addition, many of these devices have been running in commercial environments for years, which means that spare parts are readily available.

Why exactly these devices?

• Low power consumption
• Fairly quiet
• Small footprint
• Relatively powerful
• Often socketed CPUs and plugged-in RAM

⧉ Lenovo

⧉ NBB

⧉ HP

One or more mini PCs.

Not in my case: I ended up with hardware that I also considered suitable. First, I got my hands on a 2016 Dell Latitude E5270 subnotebook with a 12.5″ display. It was a well-made device, but it had one major flaw: during the COVID pandemic, it had been stored in a backpack together with a bottle of disinfectant. Over time, this bottle leaked and the contents got into the notebook, especially the touch display. The adhesive throughout the device dissolved, and the backlight in the display was irreparably damaged. Everything was covered in stains, and adhesive was oozing out everywhere around the edges.

As a recycling measure, this notebook, with its display removed, antennas removed, and without a Wi-Fi card, became our first Proxmox server: The hardware wasn’t very powerful, but it was good enough to enable meaningful continuous operation. Two energy-saving cores with hyperthreading, two slots for RAM, and one SATA and one NVMe slot each sounded usable.

Just a few months later, the next patient came to me. After just under two years, it had developed the “cheap notebook hinge breakage” syndrome. An HP 17-by2132ng from 2019. Good specifications on the data sheet, really bad design. Both the workmanship and the layout of everything is not up to par. Unfortunately, this has become increasingly common in recent years.

Here, too, I removed the device’s screen, removed the Wi-Fi and Bluetooth, and installed Proxmox. It was also worth buying a lot of RAM*, which I purchased separately.

Both basic devices were therefore free for me and, as such, made more sense than purchasing mini PCs. Especially since the CPUs in these notebooks only have a TDP of 15 watts each, which is very economical.

⧉ Laptop Markt

⧉ NBB

Applications

At this stage, I had the Dell U dual-core, which is quite weak but still manages to do an amazing amount, and the HP, which is a low-energy quad-core. The more demanding tasks run on this one.

Both “former” notebooks were too weak to virtualize entire Windows environments. I then decided to divide the tasks:

• The Dell notebook will be an “ultra low power server,” with low computing power and low power consumption. The services running on it are mainly for 24/7 monitoring.
• The HP will be the “low power server” to provide all the services I want. Everything that is Linux-based will run here 24/7.
• A thick server with as many PCI Express lanes as possible—to achieve Windows virtualization with good performance, including PCIe passthrough and vGPU. This will only run when needed, as its power consumption is significantly higher than the other two.

I built this high-performance server, how and why is documented in the following posts.

Der Beitrag Virtualization Hardware for Proxmox erschien zuerst auf flohs blog.