HTTPS and TLS 1.3 Handshake: Keys, Certificates, and RTT in Practice

Treating HTTPS as “HTTP wrapped in encryption” misses the two problems the handshake actually solves: how you know the other end is the server it claims to be, and how two parties agree on a key known only to them over a wire everyone can read. Encryption is what follows once both are settled.

1. The Architecture of TLS 1.3: 1-RTT and State Machine Overhaul

TLS 1.3 drastically reduced the handshake latency from 2-RTT to 1-RTT by aggressively deprecating obsolete primitives (RSA key transport, SHA-1) and enforcing Forward Secrecy (FS). In the OpenSSL C implementation, this is driven by the state machine inside statem_clnt.c and statem_srvr.c. A client sends a ClientHello containing pre-computed key_share extensions. The server immediately transitions to TLS_ST_SW_SRVR_HELLO, computing the shared secret, and responds with a ServerHello, EncryptedExtensions, Certificate, CertificateVerify, and Finished messages in a single flight.

2. Advanced State Machine & HKDF Visualization

The following diagram expands the 1-RTT flow to highlight the exact stages where the HMAC-based Extract-and-Expand Key Derivation Function (HKDF) is applied to generate the Handshake and Application Traffic Secrets.

sequenceDiagram
    autonumber
    participant Client
    participant Server
    
    Client->>Server: ClientHello + Key Share (X25519) + ALPN
    Note right of Server: statem_srvr.c: tls_process_client_hello()
HKDF-Extract(0, ECDHE Shared Secret)
HKDF-Expand(Handshake Secret)
    Server->>Client: ServerHello + Key Share
    Server->>Client: {EncryptedExtensions + Certificate + CertVerify + Finished}
    Note left of Client: statem_clnt.c: tls_process_server_hello()
Derive keys, Verify Signature, Verify Finished MAC
    Client->>Server: {Finished} + [Application Data (HTTP Request)]
    Note right of Server: HKDF-Expand(Master Secret) -> Application Traffic Keys
    Server->>Client: [Application Data (HTTP Response)]

3. Mathematical Rigor: ECDLP and AES-GCM Polynomials

Modern TLS 1.3 deployments rely heavily on Curve25519 for the key exchange and AES-GCM for Authenticated Encryption with Associated Data (AEAD).

Elliptic Curve Discrete Logarithm Problem (ECDLP)

The key exchange leverages the Curve25519 Montgomery curve defined by the equation over a prime field \(\mathbb{F}_p\):

\[ v^2 = u^3 + 486662u^2 + u \pmod{2^{255} – 19} \]

The security relies on the intractability of the ECDLP: given a base point \( G \) and a public key \( P = dG \), it is computationally infeasible to find the private scalar \( d \). The shared secret is computed as \( S = d_{client} P_{server} = d_{client} d_{server} G = d_{server} P_{client} \). OpenSSL implements this using highly optimized, constant-time Montgomery ladders to prevent timing side-channel attacks.

Galois/Counter Mode (GCM) Mathematics

Once keys are established, AES-GCM encrypts the application records. The authentication tag in GCM is computed using a universal hash function GHASH over the Galois field \( \text{GF}(2^{128}) \). The field is defined by the irreducible polynomial:

\[ P(x) = x^{128} + x^7 + x^2 + x + 1 \]

Elements of the field are 128-bit blocks. The GHASH function evaluates a polynomial where the coefficients are the ciphertext blocks and the variable is the hash subkey \( H \). In production, this multiplication is massively accelerated using CPU hardware instructions like Intel’s AES-NI (specifically the vpclmulqdq instruction for carry-less multiplication).

4. Production Engineering: OpenSSL and Hardware Acceleration

In high-throughput architectures (e.g., terminating 100k+ TLS connections/sec), CPU overhead is the primary bottleneck. Modern edge load balancers bypass generic C implementations and invoke raw assembly. For example, in OpenSSL’s evp_cipher API, AES-GCM is routed directly to the AES-NI vector units. Engineers tuning Envoy or Nginx for TLS termination must ensure that the OS CPU flags expose aes, pclmulqdq, and avx512f to the user-space process.

Furthermore, OpenSSL’s EVP_DigestSign for ECDSA/RSA certificate signatures is often offloaded to asynchronous cryptographic engines (e.g., Intel QAT) via engine modules, allowing the Nginx event loop to continue serving other epoll events while the hardware computes the ECDLP scalar multiplication.

5. Advanced Tooling: eBPF Traffic Interception

Debugging production TLS issues (like handshake latency spikes or cipher negotiation failures) using TCP dumps is useless since the payload is encrypted. Instead, elite engineers deploy eBPF (Extended Berkeley Packet Filter) to trace OpenSSL dynamically in user-space.

#include <uapi/linux/ptrace.h>
BPF_HASH(start, u32);
BPF_HISTOGRAM(dist);

int probe_ssl_handshake_start(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start.update(&pid, &ts);
    return 0;
}

int probe_ssl_handshake_return(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 *tsp = start.lookup(&pid);
    if (tsp != 0) {
        u64 delta = bpf_ktime_get_ns() - *tsp;
        dist.increment(bpf_log2l(delta / 1000000)); // Log2 histogram in ms
        start.delete(&pid);
    }
    return 0;
}

By injecting this eBPF program into the libssl.so text segment, SREs can visualize TLS handshake percentiles (p99 latency) without modifying the application code or suffering tcpdump context-switch overhead.

6. Post-mortem: Side-Channel Attack Mitigations

Production environments must harden against side-channel vulnerabilities such as Bleichenbacher attacks, Lucky Thirteen, or cache-timing attacks (e.g., Flush+Reload). TLS 1.3 eliminates many of these by removing RSA encryption (PKCS#1 v1.5 padding) and MAC-then-Encrypt CBC modes. To defend against remaining scalar multiplication timing leaks, libraries use Montgomery Ladders and ensure that branch instructions and memory access patterns are strictly independent of the secret scalar bits.

7. TLS Handshake Troubleshooting Matrix

TLS 1.3 issues are often reported as “HTTPS is slow,” “the certificate fails,” or “handshakes sometimes fail.” Those symptoms may come from DNS, TCP, certificate chains, key exchange, ALPN, hardware acceleration, or application retries. The matrix below ties each conclusion to observable evidence so encryption-layer issues are not mixed with lower-layer failures.

FAQ

Does the certificate encrypt application records?

No. The X.509 certificate mathematically authenticates the server via a digital signature (e.g., ECDSA over P-256) over the transcript hash, proving ownership of the public key. Encryption of the application layer is exclusively handled by the symmetric AEAD keys derived from the ephemeral ECDHE exchange via HKDF, guaranteeing Forward Secrecy.

Why not implement TLS from scratch?

Production TLS requires constant-time arithmetic to prevent microarchitectural side-channel data leaks. Rolling your own crypto in high-level languages often introduces cache-timing vulnerabilities, branch-prediction leaks, and memory-safety flaws. Always rely on battle-tested, hardware-accelerated libraries like OpenSSL, BoringSSL, or Rustls.

References

• RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3
• NIST SP 800-38D: Galois/Counter Mode (GCM)