Understanding distro's config mechanism

Ubuntu 20.04 latest ami seems to start using EC2 instance connect by default. And if you want to muck around with it, you can start from /lib/systemd/system/ssh and /lib/systemd/system/ssh.service.d/ec2-instance-connect.conf, not in /etc/ssh/sshd_config.

You can thank me later.

And if you just look at /etc/systemd/system/sshd.service you'll feel kind of magic as well as there's nothing there related to EIC. That is just a symlink to /lib/systemd/system/ssh, which has all the meat there.

Other than sshd.service, syslog.service also symlink to /lib/systemd/system/rsyslog.service.

Ok, so /etc/systemd/system is supposed to be the place where you want to override the system's service unit definition, which is usually in /lib/systemd/system. /etc/systemd/system will have precedence over /lib/systemd/system.

And the correct way to override is not by editing the .service file but instead by creating a directory called servicename.service.d/ directory at the same level and include *.conf file in that directory. Within that .conf file you can override any individual service section attributes such as ExecStart=.

So for example in /lib/systemd/system/ssh.service.d/ec2-instance-connect.conf, ExecStart is overriden with this command instead:-

[Service]
ExecStart=
ExecStart=/usr/sbin/sshd -D -o "AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %%u %%f" -o "AuthorizedKeysCommandUser ec2-instance-connect" $SSHD_OPTS

If you just look in /etc/ssh/sshd_config or /etc/systemd/system/sshd.service, you will feel like a fool because in sshd_config, AuthorizedKeysCommand is commented:-

# Expect .ssh/authorized_keys2 to be disregarded by default in future.#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none#AuthorizedKeysCommandUser nobody

and in /etc/systemd/system/sshd.service, ExecStart looks like this:-

ExecStart=/usr/sbin/sshd -D $SSHD_OPTS

Nothing in there suggested the use of EC2 Instance Connect. And systemd selling proposition is standardization 😊

Let me check how Amazon Linux does it. I think this is very Ubuntu/Debian specific.

ssh provides Include /etc/ssh/sshd_config.d/*.conf as override mechanism but I think a common dilemma faced by package maintainers or distro builders is whether to use the program's specific mechanism or use the system's mechanism.

And I think it's clear package maintainers prefer system mechanisms.

Amazon linux simply use sshd_config file.

AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f AuthorizedKeysCommandUser ec2-instance-connect

Much clearer. no hidden magic.