May 13, 2021

Modern Bastion Hosts

In the modern world of networking, where do bastion hosts fit in? Even in a perfect world of Zero Trust with extremely robust user and device identity based authentication, it would still be risky to have all of your infrastructure publicly accessible. There are still great security benefits to having network layers for defense in depth. Thankfully, modern bastion host technology makes this achievable while being user friendly.

What is a Bastion Host?

a server whose purpose is to provide access to a private network from an external network, such as the Internet.

The ideal “Modern Bastion Host” has the following traits:

Uses SSH certificates
Centrally managed
Cloud platform agnostic
Logs to a Cloud Data Warehouse
Runtime security & alerting

Auth

Traditionally bastions hosts may have used standard static credential authentication, and perhaps directly granted access to resources once on the network. This isn’t great from a security perspective and is difficult to scale. The world is moving toward passwordless authentication methods, and so should our bastion hosts. Introducing identity based authentication solves this problem.

Using your identity provider for authentication & authorization introduces multiple benefits. Combine using your identity provider with using a Certificate Authority for managing SSH certificates, and you’ll have an excellent way to automate user lifecycle management. Rather than manually adding and removing static SSH keys for individual users, you can centrally manage this process with groups within your identity provider.

At Sigma Computing we use Okta for Single Sign-On (SSO) combined with Okta Advanced Server Access (ASA) for SSH certificate and server management. We enforce MFA when authenticating to Okta, and therefore access to our bastion hosts are protected by MFA. With Okta ASA, engineers no longer need to juggle holding onto static SSH keys. The ASA client brokers short lived SSH certificates for the user, based on their identity. With this, our engineers are able to securely log on to our bastion hosts nearly as easily as any SaaS app!

Centrally Managed & Cloud Agnostic

There are some great cloud provider specific products for bastion hosts and SSH management(AWS SSM, GCP OS Login, etc.), but in order to be strategic and avoid vendor lock-in, it is important to remain cloud provider agnostic. Solutions that can be used on any cloud provider, and integrated with any identity provider are best.

This also greatly increases the usability for engineers. Regardless of where the bastion host is, users can access them all through the same method. Okta ASA enables our engineers to easily discover bastion hosts they have access to with one simple command sft list-servers. Bastion hosts from all of our cloud environments are centrally managed, organized into projects, and assigned user entitlements based on Okta groups.

Logging

We need to know who is logging in and when in case of a security incident. The standard linux openssh access logs, normally output to /var/log/auth.logor /var/log/secure depending on the distro, but these can be noisy and difficult to make use of. Okta ASA provides logging that not only shows logins, but also the SSH Certificate credential management actions.

Once users are logged in to the bastion hosts, it is important to be able to audit what commands were ran. Okta ASA provides options for configuring session capture, but in our case we opted in for more tooling to allow runtime security controls on-top of just monitoring.

As logs are collected, we send them to our security cloud data warehouse for long term storage, and of course to leverage Sigma’s capabilities for building dashboard visualization and log searching.

Runtime Security & Alerting

Beyond just monitoring, it is essential that we have the capability to enforce policies and manage SSH sessions in real time. Cmd Control offers a solution, installed as an agent on your Linux SSH hosts. The Cmd console lets us define our own policies and respective actions.

For example, here we can define a policy that triggers whenever an SSH session is about to start that is outside of the US and Canada. When triggered, this will send a slack alert to us, requiring approval within 120 seconds. If approval is not granted, the session will be stopped.

Additional Security Considerations

Patching

Patch the bastion hosts. Everything above will be for a lost cause if you’ve left your bastion hosts vulnerable to known security issues. For bastion hosts running on cloud VMs, its the responsibility of the account owner to patch the OS and packages on the host. For Linux hosts, the unattended-upgrade utility is a great starting point for automating security package updates. The major cloud providers each have their own features for VM OS patch management.

Infrastructure as Code

Using an Infrastructure as Code tool, such as Terraform, modules can be built to represent the bastion hosts as code. This makes it easy to manage and scale. Once the configuration is defined in code, it can be repeatedly deployed to new environments. There should be no manual configuration drift. Configuration as code should be the source of truth.

Zero Trust Principles

“never trust, always verify”

Just because a user is on a bastion host within a private network, does not mean it can be blindly trusted. Want to access the private kubernetes endpoint from the bastion host? Verify your identity again. Want to access a private database? Verify your identity again.

Visualization

To wrap things up, Sigma can easily help build dashboards to convey all the important information, in seconds. Anyone can build a dashboard with Sigma, you don’t need to be a data scientist, or in this case a security wizard. Simply connect Sigma to your security cloud data warehouse, and explore!