Zero Trust for the Home Lab - IPSec between Windows Domain and Linux using Certs (Part 7)
- Tenaka
- Jul 25
- 13 min read
The Road to the World's Most Secure Home Lab....
So far in the pursuit of the World's most secure home lab, the following have been implemented:
Related Posts:
Part 1 - Zero Trust Introduction
Part 3 - pfSense and 802.1x
Part 4 - IPSec for the Windows Domain
What's Covered in this Blog
This post covers implementing IPSec between my Windows Domain and Rocky Linux.
What Is Zero Trust - Recap
Zero Trust is a security framework that assumes no user, device, or network segment is inherently trustworthy, regardless of where it sits in the network. The core principles include:
Verify explicitly - Always authenticate and authorize access.
Use least privilege access – Limit access to only what's needed.
Assume breach – Design as if attackers are already in the network.
IPSec and Its Back Story
If you haven’t already, start with Part 4, where I implement IPSec in a Windows environment using certificates. And yes, you guessed it, there’s more certificate configuration ahead, wooohoooo, living the dream....
Rocky Linux
Rocky Linux version 10 is today's Linux OS of choice and will be installed onto a Hyper-V platform. Rocky will be deployed as a Wazuh monitoring platform as part of the Zero Trust implementation for the home lab. The installation of Wazuh isn’t covered here, it’ll be the focus of the next article.
Microsoft's SCOM might seem like the obvious choice for me, but there’s a longer-term goal to move away from Microsoft. As the company pivots to a Cloud and AI first strategies, on-prem support and partner benefits are steadily being erased, removing my ability and choice to deploy what and where I want.
PfSense\Managed Switch VLAN
To support the Rocky Linux servers and Wazuh, a new VLAN on the 192.168.90.0/24 subnet will be required. This aligns with the Zero Trust principle of service segregation.
Initially, pfSense is configured to allow unrestricted traffic between VLAN 20, VLAN 30, and VLAN 90 in both directions.
Don't forget to update the managed switch to also allow the new VLAN tag of 90.
A step-by-step guide for setting up VLANs, Firewalls etc for the pfSense is available in Part 2.
IPSec Additional GPO for SSH
An additional GPO exemption allowing SSH (port 22) access between the member server and the Rocky Linux hosts will ease deployment, allowing copy and paste between host and VM.
Current Domain IPSec Settings
Crucial! Windows domain traffic only supports IKEv1, not that Microsoft will make this obvious or configurable via GPO.
Crucial! Make a note of the current IPSec settings any deviation will result in IPSec negotiation failure.
The following IPSec settings are known to work reliably. While some configurations using AES-GCM 128 and 256 are supported, AES-GCM 192 is not supported on Rocky Linux. If you plan to deviate from this setup, be sure to confirm that your chosen ciphers are supported on both Windows and Linux.
Crucial! A step-by-step guide for setting up IPSec in a Windows Domain is available in Part 4. I strongly recommend following that guide before attempting to add Linux to the mix.
IPSec Settings in GPO - Just for info
Open GPO Management and navigate to the IPSec policies and edit
Computer Configuration > Policies > Windows Settings > Security Settings
Right click and Properties on Windows Defender firewall and Advanced Security.
Select the IPSec Settings tab.
Open Main Mode's Customize...
Select and edit the SHA384 integrity policy.
Make a note of all the settings.
Audit Quick Mode
Audit Authentication, which is using the Trusted Root certificate.
DNS
Create a host record for the intended Rocky host.
Linux Packages for IPSec
The latest release of Rocky Linux is installed as a Hyper-V VM, with 6Gb RAM and 250Gb disk and finally, the virtual NIC is set for VLAN 90.
During installation, disable the root account and ensure that the user you create is added to the wheel group to grant administrative (sudo) privileges.
Connection from Server
Once Rocky is properly installed, pfSense should assign it an IP address via DHCP, in my case, it was 192.168.90.100.
Here's the command to set a static IP, gateway and DNS.
sudo nmcli con mod eth0 ipv4.addresses 192.168.90.100/24 ipv4.gateway 192.168.90.1 ipv4.dns "192.168.20.245, 192.168.20.247, 192.168.20.249" ipv4.method manualSSH from PowerShell
I'll be connecting from my Windows server with PowerShell, no more Putty for me.
ssh user@192.168.90.100
Hostname
Update the hostname, match it with the DNS entry earlier.
sudo hostnamectl set-hostname wazuh90.toyo.loc
hostnameUpdates
Start where you mean to finish, apply any updates to ensure stability and security fixes are applied.
sudo dnf install updatesAD Packages
Install the packages that will allow Rocky to be a domain member.
sudo dnf install realmd oddjob oddjob-mkhomedir sssd adcli krb5-workstation -yStrongswan
Install the 2 following packages in order.
sudo dnf install epel-release -ysudo dnf install strongswan -yTime and Timezone
Rocky will be sourcing its time from the DC's, not only to support authentication protocols, but also to ensure that log timestamps are accurate and consistent across the environment.
Search for your locale, mine's London.
timedatectl list-timezones | grep LondonCopy the result and then set the timezone.
sudo timedatectl set-timezone Europe/LondonEnable and start the time sync service.
sudo systemctl enable chronydsudo systemctl start chronydUpdate chrony.conf with the following, so it points at the DC's.
server 192.268.20.245 iburst
server 192.168.20.247 iburst
server 192.168.90.249 iburst
sudo nano /etc/chrony.conf
Restart the time service.
sudo systemctl restart chronydRun the following commands once IPSec is implemented to confirm time and time sync.
chronyc sourceschronyc trackingAD or Not to AD
This step is included in case Rocky needs to join the domain. However, for its intended role as a monitoring solution, it’s best to minimize open ports and limit connectivity between it and the domain to reduce the attack surface.
To Join the Domain
In Active Directory Users and Computers, pre-create the computer object wazuh90 in the required OU, if you don't do this step, Rocky will be added to the AD Computer container.
Discover your domain (use your actual domain name in ALL CAPS).
sudo realm discover TOYO.LOCJoin the domain using an account with permissions.
sudo realm join --user=administrator TOYO.LOCPull password information for an Active Directory user.
sudo getent passwd administrator@ad.company.localPull some domain info.
realm listIPSec Certificate for Linux Preparation
Advanced certificate requests using version 3 templates are not supported through the traditional web enrollment interface (certsrv) unless you're using legacy systems like Windows XP or Server 2003. Clients running Windows Vista or newer are unable to request v3 template certificates via this method due to compatibility limitations.
Microsoft’s recommended approach for handling version 3 templates is to use Certificate Enrollment Web Services (CEP/CES) or leverage Autoenrollment via Group Policy, both of which support modern certificate features and provide a more secure and scalable enrollment process.
I’m not deploying a CES server, that's for another day and another blog, and it’s unnecessary for our needs. CES is mainly used by Windows clients for advanced certificate enrollment. Linux doesn’t require it, since it still supports the legacy method.
New Linux Certificate Template
Let's prep a certificate, open the CA management snap-in, and then right click on Certificate Templates and Manage
Duplicate a Certificate Template
Either duplicate the IPSec (Offline) certificate or the previously created 'Non-TPM' template for server or workstation.
General Tab:
Set the validity period to 1 year
Compatibility Tab:
Set both Compatibility settings to Widnows 2003, failure to do this will mean the template won't be available in the certificate web console.
Request Handling Tab:
Allow the private key to be exported.
Cryptography Tab:
Set the Algorithm to Determined by CSP and key size to 2048
Subject Name Tab:
Set to Supply in the request.
Extensions Tab:
Edit the Application Policies and add in:
Client Authentication
IP Security IKE Intermediate
IP Security Tunnel Termination
IP Security User
Server Authentication
Security Tab:
Add the user or group that will perform the certificate enrollment.
Remove any group that auto-enrolls.
Publish the Certificate Template
Return to the main CA Management snap-in.
Right click on Certificate Templates.
Select New > Certificate Template to Issue > select Toyo Linux IPSec.
Certificate Enrollment
In this section, we’ll walk you through the process of requesting a certificate for a Linux system using the Windows CA web interface.
SSH onto Rocky.
ssh user@192.168.90.100Private Key
A private key is generated locally to ensure it never leaves the system, and a CSR is then created using that key to securely request a certificate from the CA without exposing the key itself.
Create a working directory.
mkdir certscd certsCreate a private key that remains on the host; I'll secure it shortly.
openssl genpkey -algorithm RSA -out ipsec.key -pkeyopt rsa_keygen_bits:2048Create CSR
Create a CSR derived from the Private key.
Update the following with the FQDN of Rocky host.
Copy and paste into the SSH sessions.
openssl req -new -key ipsec.key -out ipsec.csr \
-sha256 \
-subj "/CN=wazuh90.toyo.loc" \
-reqexts v3_req -config <( cat <<EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[ req_distinguished_name ]
CN = wazuh90.toyo.loc
[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth, ipsecIKE, ipsecUser, ipsecTunnel
EOF
)
Cat the CSR, select all the text including the Begin and End Certificate Requests lines, and press Enter to copy to the Windows clipboard.
cat ipsec.csr
Cert Request from CA Web Console
The CSR needs to be copied to the CA Web console to complete the certificate enrolment.
From the Windows Server, open a browser and enter the address to the CA Web server eg https://certs.toyo.loc/certsrv.
Select Request a certificate.
Select Submit a Certificate request by using a base-64-encoded CMC....
Paste the CSR into the Base-64-encoded window.
Select the Toyo Linux IPSec template.
Select Base 64 encoded.
Click on Download certificate.
Open the downloaded certificate with notepad.
Copy the entire contents to clipboard.
Create the Certificate
Return to the SSH session. From this point onwards, every command will require sudo.
Sudo nano FQDN.crt and paste the contents of the Windows clipboard.
sudo nano wazah90.toyo.loc.crt
Ctrl + O to output the contents to file
Ctrl + X to exit Nano.
Copy the Private Key and Certificate to Strongswan
The private key and the certificate are required to be copied or moved to the strongswan directory and configured with the correct permissions.
Copy the Private key.
sudo cp ipsec.key /etc/strongswan/swanctl/private/Set the private key so readable and writable only by the file's owner.
sudo chmod 600 /etc/strongswan/swanctl/private/ipsec.keySet Root as the Owner.
sudo chown root:root /etc/strongswan/swanctl/private/ipsec.keyRepeat the steps to secure the private key in the home directory.
sudo chmod 600 ipsec.keysudo chown root:root ipsec.keyCopy the certificate to strongswan x509 directory.
sudo cp wazah90.toyo.loc.crt /etc/strongswan/swanctl/x509/Set the certificate permissions so the owner can write and everyone else can read.
sudo chmod 644 /etc/strongswan/swanctl/x509/wazah90.toyo.loc.crtTrusted Root CA
The root CA certificate is required on the local host to establish trust in certificates issued by that authority. Without it, the system cannot validate or trust incoming connections or services secured with those certificates.
In the CA web console, click the Home link, then select Download a CA certificate, certificate chain, or CRL.
Select Base 64 and then Download CA Certificate.
Open with Notepad and copy the contents.
Ensure you're in the 'certs' working directory.
cd ~/certsOpen nano and paste the Base64-encoded root certificate from your clipboard into the file.
sudo nano toyo-ca.loc.crtCtrl + O to output the contents to file
Ctrl + X to exit Nano.
Root Trust
Copy the root CA certificate to the trusted anchors directory so the system recognizes it as a valid certificate authority.
Copy the root CA to anchors so the browser trusts sites on my domain.
sudo cp toyo-ca.loc.crt /etc/pki/ca-trust/source/anchors/Refresh the system’s trusted certificate store with the new certificate.
sudo update-ca-trust extractCopy the root CA to the Strongswan x509CA directory.
sudo cp toyo-ca.loc.crt /etc/strongswan/swanctl/x509ca/Set the certificate permissions so the owner can write and everyone else can read.
sudo chmod 644 /etc/strongswan/swanctl/x509ca/toyo-ca.loc.crtFirewalls
The following commands permanently open the required ports and protocols for IPSec traffic. Note that port 4500 and the AH protocol are not needed for non-VPN traffic or this specific configuration.
sudo firewall-cmd --permanent --add-port=500/udpsudo firewall-cmd --permanent --add-port=4500/udpsudo firewall-cmd --permanent --add-protocol=espsudo firewall-cmd --permanent --add-protocol=ahsudo firewall-cmd --reloadSwanctl.conf and not Strongswan
StrongSwan is an open-source implementation of the IPSec protocol suite, used to establish secure, encrypted connections between hosts or networks. It uses IKE (Internet Key Exchange), typically IKEv2, to negotiate and manage security associations. Naturally, Microsoft only supports IKEv2 for VPNs, so we're stuck with IKEv1.
Configuration is handled through swanctl.conf, and the swanctl utility is used to load, manage, and monitor IPSec connections in real time. It supports certificates, EAP, and various authentication methods, making it ideal for inter Domain and subnet traffic, site-to-site and remote access VPNs.
Swanctl is to be used as the IPsec command is deprecated. Swanctl.conf gives a more modular, flexible, and systemd-friendly way to manage StrongSwan.
Backup the original swanctl.conf.
sudo mv /etc/strongswan/swanctl/swanctl.conf /etc/strongswan/swanctl/swanctl.originCreate a new swanctl.conf with nano.
sudo nano /etc/strongswan/swanctl/swanctl.conf Download my swanctl.conf from Github and paste into nano.
Crucial! Update the highlighted values to exactly match your Windows domain, they’re explicit, and any mismatch will prevent the IPSec tunnel from negotiating:
aes256-sha384-ecp384 = Key Exchange (Main Mode)
Integrity algorithm - SHA384
Encryption algorithm - AES-CBC 256
Key exchange algorithm - EC DH P-384
esp_proposals = aes128gcm128 = Data Protection (Quick Mode)
Encryption algorithm - AES-GCM 128
Integrity algorithm - AES-GMAC\GCM 128
connections {
windows-ipsec {
local_addrs = 192.168.90.100
version = 1
local {
auth = pubkey
certs = wazuh90.toyo.loc.crt
id = "CN=wazuh90.toyo.loc"
}
remote {
auth = pubkey
id = %any
}
# Quick Mode – Phase 2 configuration for the IPsec tunnel
children {
# Default phase 2
windows-ipsec {
local_ts = 192.168.90.100/32
remote_ts = 192.168.0.0/16
mode = transport
esp_proposals = aes128gcm128
start_action = trap
dpd_action = restart
rekey_time = 60m
rekey_bytes = 100000K
}
}
# Main Mode – Phase 1 IKE configuration
proposals = aes256-sha384-ecp384
reauth_time = 480m
dpd_delay = 30s
dpd_timeout = 150s
}
}
secrets {
private-1 {
file = /etc/strongswan/swanctl/private/ipsec.key
}
}
authorities {
ca-authority {
cacert = /etc/strongswan/swanctl/x509ca/toyo-ca.loc.crt
}
}In transport mode, only traffic that matches local_ts and remote_ts will be protected by IPSec. Any traffic not matching these rules will pass as normal, unencrypted traffic.
Ctrl + O to output the contents to file
Ctrl + X to exit Nano.
Instruct the Charon daemon to load plugins dynamically, making the setup more flexible and easier to manage across different use cases.
sudo tee /etc/strongswan/strongswan.conf > /dev/null <<EOF
charon {
load_modular = yes
plugins {
include strongswan.d/charon/*.conf
}
}
include strongswan.d/*.conf
EOFStart Strongswan Service
Up to this point, access to Rocky has primarily been from Windows via SSH. The upcoming steps may terminate your session, and any misconfiguration will terminate the connection. With that in mind, you may want to switch to direct console access before proceeding.
Note: Ignore any errors or warnings for sqlite plugin, harmless noise.
Remove the SSH Exemption in GPO
The IPSec GPO exemption for SSH, between the Windows Server and Rocky.
Enable and Start Strongswan
Execute the following commands, any misconfigurations, typos, or incorrect parameters with swanctl.conf will likely to cause prevent service or successfully establish an IPSec connection.
Enable Strongswan service
sudo systemctl enable strongswanStart Strongswan service.
sudo systemctl start strongswanLoad all parameters stored in the swanctl.conf file.
swantcl --load-allStatus of Strongswan
Let’s go through a few configuration steps to verify that IPsec is running correctly and establishing a successful connection to the Windows endpoint.
swanctl.conf does not allow exemptions and communicates exclusively over IPSec with Windows, so I found that using nslookup is a better way to test the initial connection with Windows domain controllers.
Check the status of the Strongswan service, ensure it is running and enabled.
sudo systemctl status strongswan
I prefer retrieving a backdated list of events so use the -n option. This is particularly useful when troubleshooting issues where viewing only the latest events might miss the critical error that triggered the problem.
journalctl -u strongswan -f displays StrongSwan events in real time as they occur.
journalctl -u strongswan -n 50
sudo swanctl --list-conns displays all configured IPSec connections from the swanctl.conf file, including their settings and current status.
sudo swanctl --list-certs lists all loaded X.509 certificates, showing details like subject, issuer, validity period, and key usage.
sudo tcpdump -n esp or udp port 500 captures and displays network packets that are either ESP (IPsec encrypted) traffic or use UDP port 500, which is commonly used for IKE (Internet Key Exchange) in IPSec.
Finally, let’s examine the Windows side of the IPSec connection. Open wf.msc and navigate to either the Main Mode or Quick Mode section. There, you should see the IPSec connection established between Rocky Linux and the Windows Server.
It Never Works First Time....
Windows Firewall
From my experience, the following journalctl messages usually mean that IKE or ESP traffic is being blocked by a firewall, either on the Windows endpoint or by pfSense. If there are no matching log entries on the Windows server, I take it as a sign that the packets never made it through. In that case, I’ll enable or check the firewall logs on pfSense to confirm if it’s dropping the traffic.
journalctl -u strongswan -n 50
wazuh90.toyo.loc charon-systemd[8207]: sending packet: from 192.168.90.100[500] to 192.168.30.61[500] (180 bytes)
wazuh90.toyo.loc charon-systemd[8207]: creating delete job for CHILD_SA ESP/0x00000000/192.168.20.245
wazuh90.toyo.loc charon-systemd[8207]: CHILD_SA ESP/0x00000000/192.168.20.245 not found for delete
wazuh90.toyo.loc charon-systemd[8207]: giving up after 5 retransmits
wazuh90.toyo.loc charon-systemd[8207]: establishing IKE_SA failed, peer not responding
wazuh90.toyo.loc charon-systemd[8207]: creating acquire job for policy 192.168.90.100/32[udp/35655] === 192.168.20.245/32[udp/domain] with reqid {2}
wazuh90.toyo.loc charon-systemd[8207]: initiating Main Mode IKE_SA windows-ipsec[1317] to 192.168.20.245
wazuh90.toyo.loc charon-systemd[8207]: generating ID_PROT request 0 [ SA V V V V V ]
wazuh90.toyo.loc charon-systemd[8207]: sending packet: from 192.168.90.100[500] to 192.168.20.245[500] (180 bytes)
wazuh90.toyo.loc charon-systemd[8207]: creating delete job for CHILD_SA ESP/0x00000000/192.168.20.247
wazuh90.toyo.loc charon-systemd[8207]: CHILD_SA ESP/0x00000000/192.168.20.247 not found for delete
wazuh90.toyo.loc charon-systemd[8207]: giving up after 5 retransmits
Syntax with swanctl.conf
The first three log extracts show that the swanctl.conf file is misconfigured, either a typo or something in the syntax is just plain wrong. Starting the service immediately fails with an exit code, which usually points to a parsing error or a missing/invalid configuration directive.
sudo systemctl start strongswan.service
Job for strongswan.service failed because the control process exited with error code.
See "systemctl status strongswan.service" and "journalctl -xeu strongswan.service" for details.
Running sudo swanctl --load-all gives the same result, confirming that the daemon can’t even load the connection definitions.
sudo swanctl --load-all
Job for strongswan.service failed because the control process exited with error code.
See "systemctl status strongswan.service" and "journalctl -xeu strongswan.service" for details.
Checking journalctl -u strongswan -n 50 reveals that charon-systemd is shutting down with status=22, which typically means there’s a configuration error (e.g., invalid parameters, wrong file paths for certificates, or unsupported options).
journalctl -u strongswan -n 50
wazuh90.toyo.loc systemd[1]: strongswan.service: Control process exited, code=exited, status=22/n/a
wazuh90.toyo.loc charon-systemd[2882]: SIGTERM received, shutting down
wazuh90.toyo.loc systemd[1]: strongswan.service: Failed with result 'exit-code'.
wazuh90.toyo.loc systemd[1]: Failed to start strongswan.service - strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
The final log extract, however, tells a slightly different story. Here, I can see the IKE negotiation starting, but it’s failing with “header verification failed.” This points to either an IKE proposal mismatch (e.g., incorrect algorithms or key sizes), a certificate identity issue, or even corrupted packets caused by a misbehaving firewall/NAT device.
journalctl -u strongswan -n 50
wazuh90.toyo.loc charon-systemd[22855]: 192.168.20.247 is initiating a Main Mode IKE_SA
wazuh90.toyo.loc charon-systemd[22855]: selected proposal: IKE:AES_CBC_256/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/ECP_384
wazuh90.toyo.loc charon-systemd[22855]: generating ID_PROT response 0 [ SA V V V V ]
wazuh90.toyo.loc charon-systemd[22855]: sending packet: from 192.168.90.100[500] to 192.168.20.247[500] (160 bytes)
wazuh90.toyo.loc charon-systemd[22855]: header verification failed
wazuh90.toyo.loc charon-systemd[22855]: received invalid IKE header from 192.168.20.247 - ignored
Thanks for your Time and Support...
Another IPSec and certificate-based blog wrapped up, and just one more to go before my Home Lab’s Zero Trust panacea of perfection is fully implemented.
Honestly, I loved working on this one. My first Linux IPSec deployment in prepping for this blog was Linux-to-Linux, and it was smooth, stable, and just worked. Then I brought Windows into the mix… and suddenly I was questioning my life choices and the tech I’ve devoted my time to. Back in the Vista days, when I was running 100% OpenSuse, I really should have stayed the course.
Next up: installing Wazuh on the Rocky 10 VM I’ve just prepped.
Related Posts:
Part 1 - Zero Trust Introduction
Part 3 - pfSense and 802.1x
Part 4 - IPSec for the Windows Domain
Yet to complete
Part 8 - Monitoring, IPS and IDS
Part 9 - DNS-over-HTTPS
Comments