Part 8: Advanced Operations: Containers, High Availability & Packet Analysis

01: Container Machinery: Docker vs. Daemonless Podman

Containers are not virtual machines. They are isolated Linux processes running directly on the host kernel.

Plaintext

Namespaces (Visibility) + cgroups v2 (Limits) + OverlayFS (Storage) = Container Process

Docker vs. Podman Architecture

Running Rootless Containers with Podman

Bash

# Run an unprivileged Nginx container as a regular user (using high port 8080)
podman run -d --name web -p 8080:80 docker.io/library/nginx:alpine

# Generate a native systemd unit file for the Podman container
podman generate systemd --new --name web --files --restart-policy=always

# Enable lingering so user-scoped systemd services stay running without an active user session
loginctl enable-linger $(whoami)

02: High Availability (HA) & Clustering Fundamentals

High Availability designs eliminate Single Points of Failure (SPOFs). Adding "nines" reduces allowed yearly downtime dramatically:

• 99.9% (Three Nines): Max ~8.7 hours downtime/year.

• 99.99% (Four Nines): Max ~52.5 minutes downtime/year.

Virtual IP (VIP) Failover with keepalived (VRRP)

keepalived uses the Virtual Router Redundancy Protocol (VRRP) to allow two nodes to share a single floating IP address. One node acts as MASTER; if its health check fails, the BACKUP node automatically claims the VIP.

Ini, TOML

# /etc/keepalived/keepalived.conf (MASTER Node)
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    
    virtual_ipaddress {
        192.168.1.100/24
    }
}

Pacemaker, Corosync & Fencing (STONITH)

For multi-resource clusters (managing storage, databases, and IPs together), use Pacemaker and Corosync.

CRITICAL CONCEPT: STONITH ("Shoot The Other Node In The Head"): Fencing forcibly powers off or isolates a non-responsive cluster node. This prevents a split-brain scenario, where both nodes independently believe they are active and simultaneously write to shared storage, causing total data corruption.

03: Packet-Level Network Diagnostics

When application logs fail to reveal why a network request dropped, inspect actual wire packets.

Packet Capture with tcpdump

Bash

# Capture TCP SYN packets (incoming connection requests) on eth0 on port 8443
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0' and port 8443 -w capture.pcap

• -i eth0: Listens specifically on network interface eth0.

• 'tcp[tcpflags] & tcp-syn != 0': Applies a raw BPF filter matching TCP SYN packets.

• -w capture.pcap: Writes raw binary packet captures to a .pcap file for detailed analysis inside Wireshark.

Socket Inspection (ss vs. netstat)

ss queries socket information directly from kernel netlink interfaces, making it much faster than deprecated netstat.

Bash

# List all active listening TCP/UDP sockets with process names and port numbers
ss -tulnp

• -t: Display TCP sockets.

• -u: Display UDP sockets.

• -l: Filter for listening sockets only.

• -n: Show numeric port numbers instead of resolving service names.

• -p: Show the owning process name and PID.

04: The Master Layered Troubleshooting Flowchart

When an internal application fails (e.g., [https://api.internal:8443](https://api.internal:8443)), run this exact 6-step checklist to isolate the failure:

1. DNS Layer: Does the name resolve correctly?

   dig api.internal A

2. Routing Layer: Does the kernel have a route to the IP?

   ip route get <IP>

3. Network Path Layer: Is the path clear?

   mtr -rc 20 <IP>

4. Transport Handshake: Can you complete a TCP 3-way handshake?

   nc -zv <IP> 8443

   • Timeout/Silence: Firewall dropping packets.

   • Connection Refused: Port closed; service is down or bound to the wrong interface.

5. Listening Service: Is the application running on the target server?

   ss -tulnp | grep 8443

6. TLS Layer: Is the SSL certificate valid and complete?

   openssl s_client -connect api.internal:8443 -servername api.internal

By following this layered sequence, you transform guessing into systematic operational engineering.

Congratulations on completing the Linux: Master Series Guide!