jrgd

Not that I should even bother, but:

If my recommendations reflect a common consensus in this community, then perhaps there is good reason they are popular, no? In my case, I personally know the person who runs the VPS I rent and do actively trust them. I know that situation isn’t going to be the same for everyone, but if you don’t fully trust your VPS provider, there’s a reason TLS forwarding and ZTT solutions like Tailscale exist.

And sorry my writing isn’t to your expectations, I guess.

My use case has containers and reverse proxy all rootless on a given machine. TLS certs pulled at proxy level. I explicitly group small clusters of relevant services per machine. Authentik and other utilities live on one machine. Things like Jellyfin and Navidrome on another, etc.

The real way to achieve the same using user isolation rather machine isolation (given you’d be having multiple reverse proxies on one machine) is to have a second layer reverse proxy with no TLS or anything, simply handling the routing to the then off-port HTTPS connections published at system level.

For my public-facing stuff, the reverse proxy on my VPS does roughly the same thing, taking my HTTPS connections and either directly forwarding, or decrypting and rewriting the cert to a wildcard one I use if I need packet filtering or other tech for a given service. Having well-defined network ranges and reverse proxies being assigned static addresses (container-level networking or otherwise) helps significantly when working out your trusted proxies config at each layer.

I have a few practices, technologies that I leverage across my systems, network for selfhosting.

First, the principles I follow:

• Least access — Limit exposure of ports, services to as limited an audience as possible.
• Don’t put all your eggs in one basket — Spread your service deployments out across different machines, or at least different users on the same machine. Group relevant services together as seen fit, but don’t deploy everything in the same place.
• Security through obscurity is bad — What I mean is make your process for deploying and maintaining applications easy to read and understand the context. Software versions, customizations, and other tweaks should be easy to find and understand.
• Mind your secrets — Seriously, don’t engage in practices that may lead to exposed secrets. Keep files that may have them out of your git repos. Read your logs before pasting them or feeding them to another application. If it can be done, automate secret deployments where possible.
• Understand your tech — Learning stuff in the context of LAN is perfectly fine and I encourage doing so. For anything that you plan to bring to any wider network though, please do the recommended reading about each component you are using. Security mistakes often come from glossing over something that wasn’t apparent, but was important. Yes, this will slow down your work to a crawl if you’re used to just pulling Docker images and deploying at-will. I still recommend to take the hit in speed in order to do things as right as possible the first time.

Now, onto the actual technologies and practices I use:

First off, as hinted above, I make heavy use of containerized applications. OCI (commonly referred to as Docker) containers are a convenient way for developers to package their applications, but also offer some affordances in security if you leverage the tools well. I personally use Podman as my tool of choice for many reasons, but one of them being the relatively sane rootless setup that is ready to use out of the box. This means that can be started up at user-level (without root permissions and with user-level restrictions), which is great in case of an exploit to escape the sandbox occurs. Combine this with pairing relevant applications together on different users, and an actual semblance of a security model starts to form in mitigating the potential disaster from a rogue container. If you use multiple users on a system, make sure they are isolated users who should not be able to even read another user’s files; don’t give users free sudoers access. I also utilize multiple different machines for different processes, which works as well.

Secondly, I utilize Quadlets, a way to write podman container, pod, network definitions as SystemD units, and be able to hook them in with other SystemD units. I love this approach for many reasons, including having declarative definitions of most of my deployment stack. Certainly makes redeployments, migration, and restoration a breeze when tracked with git. More importantly for security, an easy-to-read set of config files makes managing selfhosting significantly more predictable and boring, which is how it should be. Any question I might have about how something is setup I can find within a minute every time. Less guessing, less searching, more informed decisions; which is crucial to making less mistakes that could have security implications. Practically speaking, there are more than just Quadlets: Docker compose, Kubernetes definitions, Helm charts, and others all exist with their various strengths and weaknesses. Pick what works for you.

With actually exposing stuff to the internet (or at least a broader network), I do limit my machines’ reach where possible. I have the capability to, but do not directly port forward anything from my home router. I rent a small VPS with minimal latency to my location that I use as a forwarding proxy for public services. I have a wireguard tunnel on my home router to the VPS that I let access specific IP ranges that contain my static IP server IPs only, with firewall rules on the router to back the access grant in the tunnel config. From there, I forward traffic to the VPS’s public IP with FirewallD on the edge, or have traffic sit behind a reverse proxy on said VPS.

With most traffic though (especially with services I don’t completely trust to leave exposed), I restrict who and what may gain access to said services. If the model works for you, Zero Trust Tunneling / Mesh Network solutions like Tailscale, Netbird, or Netmaker are of high recommendation. For many of these, they allow you to connect end-user devices and serving devices together over a dynamic Wireguard tunnel (or similar) to share services that normally wouldn’t be as suitable sitting on the open internet. I personally run a selfhosted instance of Headscale which also sits on the edge VPS, and acts as my Tailscale control server and sole traffic relay for connections that can’t be direct peer to peer. Most of my services sit behind the requirement of being in the virtual network, while some I configure dual-offering. I should note that using Tailscale and others that offer direct peer-to-peer will directly expose your devices’ public IPs to others in the network that can interact.

Lastly, I can note that while there have been many efforts with various solutions to try and make selfhosting as mindlessly easy as possible, I don’t recommend running many of them for a very specific reason. Much like how I hate people using LLM chatbots in place of their own brains to try and solve problems, having some container management platform take the initial setup out of selfhosted applications and attempt to abstract away as much of finer details as possible can lead to a fast initial deployment, but certainly not a secure one long-term. I do believe there are many practices and details one should learn about their selfhosting setup (such as networking, firewalls, what each app depends on, how best to upgrade them, when to pin application versions, etc.). Having a platform take that away in the beginning will only lead to problems later down the line when those funny little details start to become more important. Only learning about those details when the problems start to occur becomes a much more monumental hurdle to cross than simply learning the details as you set things up for the first time.

Final Notes

This comment is quite large, even by my standards. Yet, I am sure I have missed something somewhere. I will be happy to reply to any specifics about either technology or practice if I missed anything.

Generally, your auth provider (assuming setup correctly) should be one of the most secure components of your internet-exposed setup. That being said, vulnerabilities can and do occur. Adding a combination of Crowdsec, OWASP CRS, other ModSecurity rulesets, and/or fail2ban will add a degree of migitation of potential exploits by blacklisting known bad actors/active bad behavior.

Above all, you should have a plan to keep aware of updates to your publicly-exposed software and be able to deploy those updates in a timely fashion.

I think the biggest one in recent (past year) would have been the upgrade tree to nvidia-open, which would break 10-series and older support without changing package on affected systems. Definitely a lot of the more recent breakages are not large-scale problems, but they do still exist.

As has been noted in the ArchWiki that is expected to be read by users, one should always check for package-related news before upgrading their Arch system. This is because package changes and breakages that require user intervention do occur, and ignoring them can sometimes leave a broken system. Arch isn’t really hard after getting over the bootstrap hurdle, but simply requires more upkeep effort than most point release distros would.

Based on OP’s edit, that looks correct. Unless OP checks the topology and can make the claim that drives from the main chipset are the slow ones, secondary chipset drives will tend to perform slower, and the ASMedia ASM1061 has several reports online of poor to trash performance compared to baseline.

Best bet if the full performance is necessary out of every drive would be finding a decent quality PCI-E SATA III riser card that doesn’t use a known flawed driver chip rather use the two ‘extra’ ports on the motherboard.

More or less yes, minus the copying files back if the operation was successful. You must be careful shrinking partitions as it is very easy to destroy them, and I’d have to guess the partition layout looks vaguely (EFI System Partition (/boot/efi), Boot (/boot), Root (/), …), which would require shrink and move of the partition before or after /boot. If you’re unfamiliar with shrinking a partition, a bit of reading into how it is done for your filesystem will be required. Different setups, ext4, btrfs, lvm, LUKS, etc. will have different requirements.

Checking the /boot size on my Fedora install, I partitioned out a gibibyte for the 3 kernel plus recovery kernel setup, which takes up about 338 MiB in total. Depending on out-of-tree kernel modules and bootloader modifications installed, your initramfs images could be larger. A few things to look for:

• the size of your current initramfs and vmlinuz image(s)
• any kernel modules you needed to install alongside your system (v4l2-loopback, nvidia, realtek, etc.)
• If there are other large files present in the boot partition

If everything there looks fine and/or is necessary, you might need to expand your /boot partition (either reinstall if new system or offline partition shrinking, moving after a data backup if you have personal files you care about).

You’re likely looking for this docs section for Caddy. The failure is the automated request to populate Caddy’s root CA cert to the host system, but obviously failed as it doesn’t have root permissions. As the docs state, if you intend to use the local HTTPS functionality of Caddy, you can manually run caddy trust privileged in order to populate the Caddy root CA cert manually. If you intend to disable the local HTTPS functionality (such as if you’re running Caddy behind a http reverse proxy), you can ignore the mail message.