Zero Trust for the Home Lab - IPSec (Part 4)

Tenaka
Jun 7
15 min read

Updated: Nov 10

The Road to the World's Most Secure Home Lab....

So far in the pursuit of the World's most secure home lab, the following have been implemented:

Part 2 - VLAN Tagging and Firewalls with pfSense

What's Covered in this Blog

This post covers implementing IPSec using certificates for authentication and data encryption across my Windows Domain. This is not for the faint of heart.

What Is Zero Trust - Recap

Zero Trust is a security framework that assumes no user, device, or network segment is inherently trustworthy, regardless of where it sits in the network. The core principles include:

Verify explicitly – Always authenticate and authorize access.
Use least privilege access – Limit access to only what's needed.
Assume breach – Design as if attackers are already in the network.

How IPSec Addresses Zero Trust Security

Zero Trust assumes the network is hostile, even internal traffic can't be trusted without verification. Every connection must be authenticated, authorized, and encrypted. IPSec (Internet Protocol Security) is a key enabler:

IPSec Overview

IPSec is a suite of protocols designed to secure IP communications by authenticating and encrypting each IP packet. It lends itself to Zero Trust by enforcing confidentiality, data integrity, origin authentication, and replay protection.
When deployed in a Windows environment, IPSec leverages Active Directory and the Microsoft Public Key Infrastructure (PKI), also known a Certificate Authority (CA).

End-to-End Encryption:

All internal traffic should be encrypted. IPSec encrypts IP packets at the network layer, ensuring confidentiality and integrity from endpoint to endpoint, protecting data regardless of the underlying application or network. This can be challenging when accessing services that aren't inherently secure or encrypted, such as the Internet.

Mutual Authentication:

With a CA managing digital certificates, IPSec can perform strong mutual authentication between devices.

Identity Assurance:

The use of a CA provides control over credentials that defines certificate lifespans, whether to issue or revoke them as needed.

IPSec Technical Components

Security Associations (SA):

An IPSec Security Association (SA) are 2 unidirectional agreements between two endpoints defining traffic security, specifying the encryption, authentication methods, keys, and lifetime for that session. These are negotiated automatically via IKE (Internet Key Exchange), ensuring both sides agree on how to protect the data in transit.

AH vs. ESP

IPSec offers two main ways to protect packets:

Authentication Header (AH) – IP Protocol 51

Authenticates the entire IP packet (including headers) to ensure data integrity and source authenticity.
AH does not encrypt the payload, your data remains visible.
Rarely used in practice; largely obsolete in modern IPSec deployments.

Encapsulating Security Payload (ESP) – IP Protocol 50

Encrypts and optionally authenticates the payload, provides:
- Confidentiality (via encryption)
- Data integrity and authentication (via HMAC)
Modes:
- Transport Mode: Encrypts only the payload, IP header is exposed.
- Tunnel Mode: Encrypts the entire original IP packet and adds a new IP header.
Obvious Choice:
- ESP is the standard choice for secure, encrypted communications in Windows IPSec.

IKEv2

Microsoft strongly recommends IKEv2 as the default key management protocol:

Supports certificate and EAP authentication methods.
Built-in NAT Traversal with UDP port 4500 encapsulation.
Robust against network disruptions and supports mobility.
Provides streamlined, secure negotiation of SAs.
Aligns well with 802.1X for device/network authentication.

Windows IPSec implementations starting with Windows 7 and Server 2008 R2 default to IKEv2 for VPNs and IPSec tunnels. Unfortunately, this feature isn't available for standard network traffic using GPO, so we will have to settle for IKEv1. Remember this when configuring swanctl.conf and Linux.

Quick and Main Modes

Phase 1: IKE SA Establishment (Main Mode or Aggressive Mode)

This phase establishes the ISAKMP/IKE Security Association, which provides a secure, authenticated channel for subsequent negotiation. It includes:

Peer authentication using certificates, pre-shared keys, or Kerberos
Diffie-Hellman key exchange to derive shared secrets
Agreement on encryption, integrity, and hashing algorithms

Traffic in this phase uses UDP port 500.

Phase 2: IPSec SA Negotiation (Quick Mode)

Phase 2 leverages the secure channel from Phase 1 to:

Establish one or more IPSec Security Associations (SAs)
Define the protected traffic flows using traffic selectors (source/destination IPs, ports, protocols)
Generate fresh keying material for encryption and integrity

Quick Mode messages are protected by the IKE SA established in Phase 1. The outcome is the creation of ESP SAs for actual packet level protection.

Ports & Protocols:

Crucial! Required Windows firewall rules, both Inbound and Outbound:

UDP Port 500: IKE (initial negotiations)
UDP Port 4500: IKE NAT Traversal (when devices are behind NAT)
IP Protocol 50: ESP (for the encrypted data)
IP Protocol 51: AH (don't use)

Basic CA and GPO Configuration:

Ensure every Windows Domain client trusts the Windows Root CA, and deploy it using GPO if necessary.
Enable auto-enrollment of certificates in Group Policy, certificate templates with the Autoenroll permission will do just that, and enroll automatically.
Crucial! Certificates should not serve multiple purposes. The IPSec certificate must exclude other Application policies to avoid conflicts and processing errors.

Additional Logging

I've enabled the three IPSec auditing policies at the root of the Domain to assist with initial testing and troubleshooting. These settings generate a high volume of event logs, but once IPSec is stable and set to 'Required', logging can be reduced to capture only failures.

Client Certificates

No TPM.....

Not all of my Windows clients and servers are blessed with a TPM, case in point, the Intel Skull Canyon and the Gen 6 NUC - Hyper-V host, they still cling to life with retirement being long overdue. Without the TPM there's no support for Microsoft Platform Crypto Provider.

The fallback option when a TPM isn't available is to use the Microsoft Software Key Storage Provider, targeting the correct provider is managed through Active Directory groups controlling certificate enrollment.

Create the following AD Groups. If you have followed the 802.1x blog, the client groups will have already been created:

For Computer objects that do not support TPM

RG_CA_WksAuthCert_Deny_TPM_Supt
RG_CA_MemberServer_Deny_TPM_Supt

For Computer objects that do support TPM

RG_CA_MemberServer_Allow_TPM_Supt
RG_CA_WksAuthCert_Allow_TPM_Supt

Workstation Authentication Client - TPM Supported

Workstation Authentication Template:

Right-click Certificate Templates and select Manage.
Right-click the Workstation Authentication template and select Duplicate Template.

General Tab:

Template display name: Toyo Workstation IPSec
Validity period: 1 years
Renewal period: 6 weeks
Check 'Publish certificate in Active Directory.'

Compatibility Tab:

Set compatibility levels to Windows Server 2016 and Windows 10

Cryptography Tab:

Provider Category: Key Storage Provider
Algorithm Name: RSA
- RSA is supported for key generation and storage in the TPM.
- ECC (Elliptic Curve Cryptography) isn't generally supported for TPM storage of certificates.
Minimum key size: 2048
Request hash: SHA256
Requests must use one of the following providers:
- Microsoft Platform Crypto Provider
  - Microsoft Platform Crypto Provider is the Key Storage Provider (KSP) that allows certificates and their private keys to be stored in the Trusted Platform Module (TPM).
  - If no TPM is accessible, the certificate will fail to enroll.

Subject Name Tab:

Under 'Build from this Active Directory Information'

Subject name format: Common Name is selected
Include this information in the alternative sub name: Check DNS

Extensions Tab:

Select Application Policies and click Edit....
Ensure Client Authentication is present.
Add IP Security IKE Intermediate > Click OK.
- This isn’t required for 802.1X, but it will be relevant in the next article on IPsec.

Security Tab:

Ensure Domain Computers is removed
Add RG_CA_WksAuthCert_Allow_TPM_Supt and Allow Read, Enroll and AutoEnroll.
Click Apply and OK.