Some upfront notes - I did this a month ago. At that time, I haven't look into Cloudflare ZeroTrust and tunnel yet. So if I'm doing this now, I probably would do it differently for the networking part.
Let say we have some beefy server with GPU that previously being used for ML works but now sitting idle. A perfect candidate for shared development server. We're currently using Gitpod and while it's been a great experience, not so much for the wallet.
Back in 2010 when we started doing remote work, we also practiced remote dev with a no local code checkout policy. So instead of working on their own computer, developer works on remote EC2 instance. It pretty simple at that time since everyone just use Vim and works inside screen session to get continuous remote sessions. Couple with ssh port forwarding, it has everything that we need.
Fast forward 12 years later, the nature of web development has changed a lot. Frontend tooling use quite a lot of resources and developer also want to use VSCode instead of Vim. Shared server no longer an options.
But what if we can still use shared server with standalone experiences? Enter LXC. People probably knows Docker more than LXC even though they actually using same set of technology such as namespaces and cgroups. The main difference probably we can say docker is used for process container while lxc for OS container, although nothing stop us to use them vice-versa. LXC would give us the traditional VPS thing.
So the idea is that everyone would works inside their own LXC instance.
The Problem
Shared server is not possible because the app being setup in a way that everyone need to run postgresql on port 5432, redis on port 6379, app on port 8000 and so on. Of course we can assign everyone different ports but that's messy.
LXC solve this problem partly but not all. While everyone can run all the services on the ports mentioned above inside LXC, how do we access them from outside?
Nginx for routing
Gitpod give some ideas on how to route external request to your app running inside the container. For example if your app is running at port 8000, your app url will be:-
8000-appname-workspaceid.workspacehost.gitp..
So how do we achieve same url scheme for our dev lxc container? Our nginx config looks like this:-
server {
listen 443 ssl;
server_name ~^(?<port>\d+)-(?<myhost>[^.]+)\.home.lab$;
include snippets/home-lab-ssl.conf;
location / {
resolver 127.0.0.53;
set $target_port $port;
set $target_host $myhost;
rewrite ^(.*)$ "://$http_host$uri$is_args$args";
rewrite ^(.*)$ "http$uri$is_args$args" break;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Origin https://$host;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_pass http://$target_host.lab:$target_port$request_uri;
}
}
This allow us to have url scheme likthis:-
Request to above url will be forwarded by nginx running on the host server to the port 3000 of the container. This solve the routing problem but how about the networking?
SSH Socks Proxy
So the domain kamal.home.lab above is just made up domain that don't really exists. The easiest way to make the domain resolvable is by adding it to /etc/hosts. As mentioned at the beginning of this article, I would probably look into Cloudflare ZeroTrust if I'm doing it now.
My /etc/hosts on the host server looks like this:-
192.168.10.100 8080-kamal.home.lab
192.168.10.100 3000-kamal.home.lab
10.201.33.17 kamal.lab
192.168.10.100 is the host IP address and 10.201.33.17 is the lxc container instance. LXC is configured with bridge networking so the container instances looks like another host in the network. That solve the networking inside the server. But how to access them from my laptop?
This is where ZeroTrust or Tailscale would help but for now I'll just use SSH socks proxy. From my laptop I would do the following:-
ssh server-ip -D 5555
That would create a socks tunnel from my laptop to the server. The next step is to configure my browser (firefox) under the networking settings. Pick Manual proxy configuration and specify SOCKS host 127.0.0.1 and port number 5555. Next is the most important - tick Proxy DNS when using SOCKS v5. This will force Firefox to make dns request through the socks tunnel as well.
DNS Resolving
Our browser already making dns query to our server so we need to something to answer the query, or in other words run a DNS server. Fortunately systemd comes with systemd-resolvd, a lightweight dns server. And by default it will look into /etc/hosts that we enter our records above. Perfect! Just make sure resolvd is running and we're done.
SSL Setup
At this point we can already access any tcp ports inside our lxc instance from our browser but only with plain http. Many apps these days that use websocket will require the connection to be in https, including code-server. Remember our url should be like this:-
We will just use self-signed certificate for this. I'll skip how to generate them as there are plenty of tutorials you can find on now to generate self-signed cert. What's important is for you to add the cert to your browser's certificate list otherwise you'll get the infamous certificate warning.
Installing code-server
First we need to get into our lxc instance. We can use this command:-
lxc exec --user 1000 --group 1000 --env HOME=/home/ubuntu/ -- /bin/bash --login
It's quite long and fortunately lxc provides alias command so we can create alias such as login, that allow us to use shorter command to get into our instance:-
lxc login container-name
For installing coder, just follow the instructions to use the install script. Take notes here, there are 2 projects - Coder and code-server which is part of Coder. Coder is complete development environment similar to Gitpod or Github Codespaces. But we only want code-server here so make sure to install the correct app.
Another notes, you might see instruction to run command such as:-
sudo systemctl enable code-server@$USER
and run into errors. That's because $USER environment variable is not set, so we need to fix command to login above or just use the actual user name. We almost done now. Accessing Github to clone our git repos can be done from code-server itself.