Stress test Proxmox setup for minilab

Jump to latest Follow Reply
E

eisa01

Ars Scholae Palatinae
1,090
Subscriptor
  • #1
    Ok, so I've tried Proxmox on a spare Lenovo L480 laptop that I had, which has an 8th generation Intel CPU, 8 GB RAM and a 250GB SSD drive.

    I originally thought to buy a mini PC, but this seems to work fine with 10 W consumption, so might as well start out with it while I figure out if I have more complex needs

    What I intend to do is to set up:
    Home Assistant VM (I have an Apple TV that can act as Thread border router, but maybe I also need a Zigbee / Zwave radio due to lack of Matter devices)
    qBittorrent LXC
    Jellyfin LXC
    OpenMediaVault VM that will host a media drive (one or two SSDs connected via USB) for file sharing to qBittorrent and Jellyfin + my laptop
    Any other services that I may think to test out (Pi-Hole maybe? Then I might also need some more RAM)

    Questions:
    Does this sound like a good idea?

    I also have a 1 TB Nvme SSD but that seems overkill for Proxmox and containers itself? Having data on the boot drive seems like a bad idea, unless I'd use it for something like Paperless and set up backup

    While I will not store anything critical on these machines, it would be nice to have a backup of the Proxmox environment and containers. How do I best do that? Rsync to cloud storage? To my own laptop? If I later upgrade to a mini PC can I just restore things and get up and running?

    Home Assistant is probably the most critical service, they just updated the built in backup system so I can use that

    FYI: I found a nice guide that seems to do slightly similar things that I intend to: https://github.com/mgrimace/Homelab
     
    Last edited:
    K

    koala

    Ars Tribunus Angusticlavius
    8,031
  • #2
    eisa01 said:
    Does this sound like a good idea?
    Click to expand...

    I run Proxmox on 4gb boxes and it's excessively tight. The Proxmox daemons eat a significant chunk of 4Gb, and there's also ZFS memory usage if you decide to use ZFS.

    I suppose it will be OK, but you might want to consider lighter alternatives like Incus.

    eisa01 said:
    I also have a 1 TB Nvme SSD but that seems overkill for Proxmox and containers itself? Having data on the boot drive seems like a bad idea, unless I'd use it for something like Paperless and set up backup
    Click to expand...
    Proxmox is one of the few similar solutions that works well with a single drive or mirror pair.

    My two Proxmox 4Gb hosts use spinning rust and nowadays, compared to other systems I run, they feel glacially slow; applying apt updates is very slow.

    If you can afford it, I'd have "systems" in SSD- both the Proxmox OS filesystem and LXC/VMs non-data filesystems.

    eisa01 said:
    While I will not store anything critical on these machines, it would be nice to have a backup of the Proxmox environment and containers. How do I best do that? Rsync to cloud storage? To my own laptop? If I later upgrade to a mini PC can I just restore things and get up and running?
    Click to expand...
    That is a complex topic.

    I set up all my systems so that "data" is on their own ZFS filesystem. Then I use syncoid to replicate to other disks, including external USB hard drives. ZFS send/recv is ultraefficient and nice.

    Proxmox has its own backup product, but I have not used it.

    In theory with Proxmox you can backup LXC containers and VMs so they can just be restored, but I have not used this either.
     
    steelghost

    steelghost

    Ars Praefectus
    5,456
    Subscriptor++
  • #3
    Proxmox Backup Server (PBS) only backs up VMs and containers. It's good at that because it allows "diff" based backups, but it won't help you with the Proxmox configuration itself. That said, you don't need to use PBS, Proxmox can do backups to local drives, NFS or SMB shares, or whatever else, just fine.

    If you're running a an OMV VM you can set up a share for backups in Proxmox, and then have Proxmox just backup onto that. I have been running the same HA VM for 2+ years, it's been repeatedly migrated multiple times across at least four different physical systems in that time, no problem. Same for my PiHole container and various other bits and bobs. If you decided to get a mini PC at some point in the future, you would just install Proxmox on that and then add the same SMB share to it. Then you would be able to access all the backups from your laptop Proxmox node. A laptop is actually a great candidate for this sort of thing in a lot of ways as it has a built in UPS :)

    Configuration backup for Proxmox is a very highly requested feature, but one that so far doesn't exist outside of people making various scripts on Github, etc. One issue is that Proxmox is essentially "just" some stuff installed on top of Debian, so where are the lines drawn about what is backed up, unless you just backup the whole filesystem?

    I have a TrueNAS VM (which will also boot from the bare hardware in case my Proxmox environment falls over), so that is my ZFS target, I need to work out how to do a backup as @koala describes. For now I just run mirrored drives for everything (no it's not a backup, but it does reduce the chances of a drive failure taking everything down).

    You absolutely can use containers and VMs without Proxmox, it's ultimately "just" a front end to technologies that already exist, but there's a reason why good, well featured front ends exist, they make many operations easier, especially if you're just getting started.

    My observation from fiddling with this stuff for a while is that you can often get away with less RAM than you expect, especially if you're prepared to manage things closely. That said, if you can upgrade the memory in your laptop to say, 16GB, it will make the whole thing more comfortable.
     
    M

    Mental Gear Reduction

    Wise, Aged Ars Veteran
    2,783
  • #8
    eisa01 said:
    Got it down to 7 W power consumption by turning off the screen, so quite efficient :)

    I'm still not sure if I should stick to the 256 GB drive, or switch to the 1 TB drive.
    I guess it's a question of how fast I will use up the storage, and if I then can buy a drive for the same amount I sell it for now or less...
    Click to expand...
    256G is skinny for a VM host. If you can really tightly manage space, it might work all right. But I prefer to make 32G volumes for my Linux guests, and there wouldn't be room for many of those on a 256G drive.

    On a desktop, I'd suggest booting from the 256-gigger and then using the 1TB drive as your VM store. But unless your laptop has a spare NVMe slot (highly unlikely) that's not going to work. Using a USB enclosure could work, but may be unreliable, so I'd probably skip that idea, too.

    edit: of course, the other argument is that with only 8GB of memory, would you ever have enough VMs to exceed 256G? You might not. That's only room for three or four small guests.
     
    steelghost

    steelghost

    Ars Praefectus
    5,456
    Subscriptor++
  • #9
    Run more things as containers rather than VMs, and that 256GB limit is a bit roomier. But of course it depends how much you have ambitions to run on that machine, and eg how much data your Home Assistant host collects. The virtual drive for my HA VM is 100GB, of which about 60GB is used. But I've got loads of add-ons, and I've been accumulating long term data on some things for years.
     
    K

    koala

    Ars Tribunus Angusticlavius
    8,031
  • #10
    It depends massively on what you're doing.

    I decided to run my Proxmox DC on a pair of mirrored 1Tb drives. Currently I have used 607Gb, have 316Gb free.

    / is using 24Gb; but there's 17gb of ISOs and 3.4Gb of backups there.

    • The Nextcloud LXC container eats 2gb of system disk... and 334gb of data (the Nextcloud database is in a different host)
    • I have two K8S single-node VMs; these are eating about 23gb each. They run a few apps, but there's no persistent storage on them.
    • My PostgreSQL LXC container is under a gb of system disk and a gb of data, but it has an absurd about of snapshots that I don't clean eating 160gb of disk.
    • The Clickhouse LXC container storing monitoring data is about 4Gb of disk
    • My workstation LXC uses 26gb of disk; but that has a full graphical desktop, runs some containers (7.2gb), holds a copy of my main mail account (3.5gb), and has source code and compilation artifacts (2-3gb).
    • The rest of LXC containers eat less than a gig each; mostly they have persistent storage on the database containers. These are Miniflux, FreeSWITCH, Nagios, FreeIPA, Ipsilon, SeaweedFS, Bitwarden, Gitolite, and Grafana.
    So like half of my storage is Nextcloud. If you are going to use your machine for personal data storage, the sky's the limit.

    Running K8S does take a significant amount of space for some reason (container images, I guess?), but nothing compared to Nextcloud.

    Non-media data like databases can eat a few gigs of disk.

    Other than that, Linux LXC containers (EL9) are very small.

    LXC with ZFS is really efficient because volume size is basically a quota; the container filesystems are stored on the Proxmox filesystem, so there's no "thin/thick provisioning" to think of...

    If you host media... calculate the size of your media, that can dominate everything else. If you're running Windows VMs, that can also consume a chunk of disk too. Otherwise, storage needs might be modest.

    (Also account if you want to store backups there, ISOs, etc.)
     
    • Like
    Reactions: steelghost
    K

    koala

    Ars Tribunus Angusticlavius
    8,031
  • #12
    That is very true. One of the reasons I picked a 128gb server is to use it also as a lab environment. And I used it a couple of times for stuff that nearly exhausted my resources (building Android...).

    In my current setup, I'm more worried about disk space than RAM (LXC containers sip memory. Really LXC is so efficient...), but mostly because I'm storing pictures and video. CPU I've really never thought about (except when building Android...).
     
    • Like
    Reactions: wobblytickle
    M

    Mental Gear Reduction

    Wise, Aged Ars Veteran
    2,783
  • #14
    eisa01 said:
    I'll go with the 1 TB drive, not worth trying to sell it and then repurchase a larger drive at a later point :)
    Laptop only has 1 Nvme slot

    And yes, I will also try to get some more RAM, want to try to bundle it with another purchase
    Click to expand...
    RAM would be very helpful. If the laptop will take it, 32G would likely be a good target.
     
    • Like
    Reactions: steelghost
    steelghost

    steelghost

    Ars Praefectus
    5,456
    Subscriptor++
  • #15
    eisa01 said:
    I'll go with the 1 TB drive, not worth trying to sell it and then repurchase a larger drive at a later point :)
    Laptop only has 1 Nvme slot

    And yes, I will also try to get some more RAM, want to try to bundle it with another purchase
    Click to expand...
    The speed of the RAM doesn't really matter at all for this use case, as longs as it works and is stable. I mention this because you might be able to get some slower RAM (2400, 2666, maybe even 2133 if it'll run) used for a good price.
     
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply