Kubernetes storage backends
-
I'm looking for experiences and opinions on kubernetes storage.
I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.
I've looked at linstore or seaweedfs/garage with juicefs but I'm not sure how well the performance of those options is across the internet and how well they last in long term operation.
Is anyone else hosting k3s across the internet in their homelab?Edit: fixed wording
One thing I recently found out is that ceph wants whole drives. I could not get it to work with partitions. I got it to work with longhorn, though I'm still setting things up.
-
I'm looking for experiences and opinions on kubernetes storage.
I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.
I've looked at linstore or seaweedfs/garage with juicefs but I'm not sure how well the performance of those options is across the internet and how well they last in long term operation.
Is anyone else hosting k3s across the internet in their homelab?Edit: fixed wording
Rook-ceph for sure. And echoing another comment, come join homes operations discord, we have a heap of info and people experienced with kubernetes with homelabbing
https://discord.gg/home-operations -
I'm looking for experiences and opinions on kubernetes storage.
I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.
I've looked at linstore or seaweedfs/garage with juicefs but I'm not sure how well the performance of those options is across the internet and how well they last in long term operation.
Is anyone else hosting k3s across the internet in their homelab?Edit: fixed wording
wrote last edited by [email protected]That isn't how you would normally do it
You don't want to try and span locations on a Container/hypervisor level. The problem is that there is likely to much latency between the sites which will screw with things. Instead, set up replicated data types where it is necessary.
What are you trying to accomplish from this?
-
I heard that ceph lives and dies with the network hardware. Is a slow internet connection even usable when the docs want 10 gbit/s networking between nodes?
wrote last edited by [email protected]I’m really not sure. I’ve heard of people using Ceph across datacenters. Presumably that’s with a fast-ish connection, and it’s like joining separate clusters, so you’d likely need local ceph cluster at each site then replicate between datacenters. Probably not what you’re looking for.
I’ve heard good things about Garbage S3 and that it’s usable across the internet on slow-ish connections. Combined with JuiceFS is what I was looking at using before I landed on Ceph.
-
That isn't how you would normally do it
You don't want to try and span locations on a Container/hypervisor level. The problem is that there is likely to much latency between the sites which will screw with things. Instead, set up replicated data types where it is necessary.
What are you trying to accomplish from this?
The problem is that I want failover to work if a site goes offline, this happens quite a bit with private ISP where I live and instead of waiting for the connection to be restored my idea was that kubernetes would see the failed node and replace it.
Most data will be transfered locally (with node affinity) and only on failure would the pods spread out.
The problem that remained in this was storage which is why I'm here looking for options. -
The problem is that I want failover to work if a site goes offline, this happens quite a bit with private ISP where I live and instead of waiting for the connection to be restored my idea was that kubernetes would see the failed node and replace it.
Most data will be transfered locally (with node affinity) and only on failure would the pods spread out.
The problem that remained in this was storage which is why I'm here looking for options.That isn't going to work unfortunately
You need very low latency (something like 10ms or preferably less)
-
I'm looking for experiences and opinions on kubernetes storage.
I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.
I've looked at linstore or seaweedfs/garage with juicefs but I'm not sure how well the performance of those options is across the internet and how well they last in long term operation.
Is anyone else hosting k3s across the internet in their homelab?Edit: fixed wording
I've been using backblaze b2 (via s3fs-fuse container + bidirectional mount propagation to a host path) and a little bit of google drive (via rclone mount + the same mounting business) within kubernetes. I only use this for tubearchivist which I consider to be disposable. No way I'm using these "devices" for anything I really care about. I haven't tried gauging the performance of either of these, but I can say, anecdotally, that both are fine for tubearchivist to write to in a reasonable amount of time (the bottleneck is yt-dlp ingesting from youtube) and playback seems to be on par with local storage with the embedded tubearchivist player and jellyfin. I've had no issues with this, been using it about a year now, and overall I feel it's a decent solution if you need a lot of cheap-ish storage that you are okay with not trusting.
