Certificates, CAs, and the Chain of Trust

"Trust is not given; it is constructed, link by link, and verified at every step." -- Bruce Schneier

When your browser throws a full-page warning -- Your connection is not private -- it means the browser tried to verify a digital certificate, walked up the trust chain, and hit a dead end. Understanding exactly what happened and why is the subject of this chapter.

What Is a Digital Certificate?

A digital certificate is, at its core, a signed document that binds a public key to an identity. Think of it like a passport: your government (the issuer) vouches that the photo and name (the identity) belong to you (the subject), and they stamp it with an official seal (the signature) that border agents can verify.

But unlike a passport, a digital certificate is machine-verifiable in milliseconds. Every time your browser connects to a site over HTTPS, it checks the server's certificate using a chain of cryptographic signatures that stretches back to a handful of trusted root authorities embedded in your operating system or browser.

It is more than just a file with a name and a key. It is a structured file with very specific fields. The standard is called X.509, and it has been around since 1988. Every field exists for a security reason that took decades of exploits to discover.

X.509 Certificate Structure: Field-by-Field Breakdown

Every X.509v3 certificate contains these fields. Understanding each one is essential for debugging TLS issues and recognizing misconfigurations.

graph TD
    A[X.509 Certificate] --> B[tbsCertificate<br/>To Be Signed]
    A --> C[signatureAlgorithm]
    A --> D[signatureValue]

    B --> E[version: v3]
    B --> F[serialNumber]
    B --> G[signature algorithm]
    B --> H[issuer DN]
    B --> I[validity period]
    B --> J[subject DN]
    B --> K[subjectPublicKeyInfo]
    B --> L[extensions v3]

    I --> I1[notBefore]
    I --> I2[notAfter]

    L --> L1[Subject Alt Names]
    L --> L2[Key Usage]
    L --> L3[Extended Key Usage]
    L --> L4[Basic Constraints]
    L --> L5[CRL Distribution Points]
    L --> L6[Authority Info Access]
    L --> L7[CT Precert SCTs]

Here is a walk through each field in detail, using actual openssl x509 -text output from a real certificate.

Version -- Almost always v3 (value 2, because it is zero-indexed). Version 3 introduced extensions, which are critical for modern PKI. Version 1 certificates lack Subject Alternative Names, Key Usage, and other fields that modern browsers require. If you ever see a v1 certificate in production, something has gone very wrong.

Serial Number -- A unique identifier assigned by the issuing CA. The serial number must be unique within the scope of that CA. After the 2008 MD5 collision attack against RapidSSL, the CA/Browser Forum mandated that serial numbers must contain at least 64 bits of entropy from a CSPRNG. This prevents attackers from predicting serial numbers and pre-computing hash collisions.

# Inspect the serial number
openssl x509 -in cert.pem -noout -serial
# Serial=0A3C7F8B2E4D6A1C9F0B

Signature Algorithm -- Specifies the algorithm used by the CA to sign this certificate. In 2026, you should see sha256WithRSAEncryption or ecdsa-with-SHA256 (or SHA384/SHA512 variants). If you see sha1WithRSAEncryption, the certificate was issued with a deprecated algorithm. SHA-1 certificates have been distrusted by all major browsers since 2017, following the SHAttered collision attack.

Issuer -- The Distinguished Name (DN) of the CA that signed this certificate. This tells you who vouched for this certificate's authenticity. The issuer field is how browsers walk the chain of trust upward -- they look for a certificate whose Subject matches this Issuer.

Issuer: C=US, O=Let's Encrypt, CN=R3

Each component has meaning: C is country, O is organization, CN is common name. The format is inherited from X.500 directory services, which is why it feels bureaucratic -- because it was designed by standards bodies in the 1980s.

Validity Period -- Two timestamps: notBefore and notAfter. The certificate is only valid between these times. If the current time is outside this window, the certificate must be rejected.

# Check validity dates
openssl x509 -in cert.pem -noout -dates
# notBefore=Jan 15 00:00:00 2026 GMT
# notAfter=Apr 15 23:59:59 2026 GMT

Let's Encrypt certificates have a 90-day validity period. Traditional commercial CAs used to issue certificates valid for multiple years, but the CA/Browser Forum has progressively shortened the maximum validity. As of 2025, the maximum is 398 days, and Apple has proposed reducing this to 45 days by 2027. Shorter lifetimes reduce the window of exposure if a key is compromised.

Subject -- The DN of the entity this certificate represents. For server certificates, this traditionally contained the hostname in the CN field, but modern browsers ignore the CN entirely. They only check the Subject Alternative Name extension. The Subject field is effectively legacy for server certificates but still used for CA certificates and client certificates.

Subject Public Key Info -- Contains the public key algorithm (RSA, ECDSA, Ed25519) and the actual public key material. This is the key that the certificate binds to the identity in the Subject/SAN fields.

# View the public key details
openssl x509 -in cert.pem -noout -pubkey | openssl pkey -pubin -text -noout

For RSA keys, you will see the modulus (the large number) and the exponent (almost always 65537). For ECDSA keys, you will see the curve name (P-256 or P-384) and the public point coordinates.

Extensions (v3) -- These are where the real security semantics live.

Subject Alternative Names (SAN) -- The authoritative list of names this certificate is valid for. Browsers check the requested hostname against this list. Supports DNS names, IP addresses, email addresses, and URIs. A certificate without a SAN matching the requested hostname will be rejected by modern browsers, even if the CN matches.

Key Usage -- Specifies what cryptographic operations this key may be used for. Server certificates typically have Digital Signature and Key Encipherment. CA certificates have Certificate Sign and CRL Sign. This field prevents a server certificate from being used to sign other certificates, limiting the damage if it is compromised.

Extended Key Usage (EKU) -- Further restricts what the certificate can be used for. TLS Web Server Authentication (OID 1.3.6.1.5.5.7.3.1) for servers, TLS Web Client Authentication for mTLS clients. A certificate with only server auth EKU cannot be used for client authentication, and vice versa.

Basic Constraints -- Contains the CA flag (TRUE for CA certificates, FALSE for end-entity) and optionally pathLenConstraint, which limits how many levels of CAs can appear below this one. This is critical: without CA:FALSE on end-entity certificates, a compromised server certificate could theoretically be used to sign other certificates. The 2011 Comodo compromise exploited weaknesses in exactly this area.

CRL Distribution Points -- URLs where clients can download the Certificate Revocation List to check if this certificate has been revoked.

Authority Information Access (AIA) -- Contains the OCSP responder URL for real-time revocation checking, and optionally the URL to download the issuer's certificate (the "CA Issuers" field, which helps clients build the chain if they are missing an intermediate).

Signed Certificate Timestamps (SCTs) -- Proof that this certificate has been submitted to Certificate Transparency logs. Chrome requires at least two SCTs from different log operators for all publicly trusted certificates issued after April 2018.

Inspecting a Real Certificate with OpenSSL

# Pull and display a certificate in full
openssl s_client -connect www.google.com:443 -servername www.google.com \
  < /dev/null 2>/dev/null | openssl x509 -noout -text

Here is what each section of the output tells you:

# Just the subject and issuer -- who is this cert for, who signed it
openssl x509 -in server.crt -noout -subject -issuer

# Validity dates -- when does it expire
openssl x509 -in server.crt -noout -dates

# The public key in PEM format
openssl x509 -in server.crt -noout -pubkey

# The serial number -- unique ID from the CA
openssl x509 -in server.crt -noout -serial

# SHA-256 fingerprint -- useful for pinning and verification
openssl x509 -in server.crt -noout -fingerprint -sha256

# All Subject Alternative Names -- the authoritative hostname list
openssl x509 -in server.crt -noout -ext subjectAltName

# All extensions -- the full picture
openssl x509 -in server.crt -noout -ext basicConstraints,keyUsage,extendedKeyUsage

Run this right now against a site you use daily:

\```bash
echo | openssl s_client -connect github.com:443 -servername github.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 \
    -ext subjectAltName,keyUsage
\```

Note the issuer. Then Google that issuer's name. Who are they? Why does your browser trust them? What root CA sits at the top of that chain? Follow the rabbit hole -- it ends at a pre-installed certificate in your operating system that you never explicitly chose to trust.

The PKI Hierarchy: Who Trusts Whom?

Your browser does not just trust a certificate because it says "I'm google.com." Anyone can create a certificate that says that. The question is: who signed it?

The Three-Tier Model

Public Key Infrastructure uses a hierarchical trust model with three layers. This design is the result of decades of operational experience with the fundamental tension between security (keeping critical keys safe) and availability (being able to issue certificates quickly).

graph TD
    Root["Root CA<br/>(Self-Signed)<br/>Key: Offline HSM<br/>Lifetime: 20-30 years<br/>Example: DigiCert Global Root G2"]

    Inter1["Intermediate CA 1<br/>Key: Online HSM<br/>Lifetime: 5-10 years<br/>Signs: Server certificates<br/>Example: DigiCert SHA2 EV Server CA"]

    Inter2["Intermediate CA 2<br/>Key: Online HSM<br/>Lifetime: 5-10 years<br/>Signs: Server certificates<br/>Example: DigiCert TLS RSA SHA256 2020 CA1"]

    Leaf1["End-Entity: www.example.com<br/>Key: Web server<br/>Lifetime: 90 days - 1 year"]
    Leaf2["End-Entity: api.example.com<br/>Key: Web server<br/>Lifetime: 90 days - 1 year"]
    Leaf3["End-Entity: shop.example.com<br/>Key: Web server<br/>Lifetime: 90 days - 1 year"]
    Leaf4["End-Entity: mail.example.com<br/>Key: Web server<br/>Lifetime: 90 days - 1 year"]

    Root -->|Signs| Inter1
    Root -->|Signs| Inter2
    Inter1 -->|Signs| Leaf1
    Inter1 -->|Signs| Leaf2
    Inter2 -->|Signs| Leaf3
    Inter2 -->|Signs| Leaf4

    style Root fill:#ff6b6b,color:#fff
    style Inter1 fill:#ffa94d,color:#fff
    style Inter2 fill:#ffa94d,color:#fff
    style Leaf1 fill:#69db7c,color:#000
    style Leaf2 fill:#69db7c,color:#000
    style Leaf3 fill:#69db7c,color:#000
    style Leaf4 fill:#69db7c,color:#000

Why not just have the root CA sign everything directly? The root CA's private key is the crown jewel. If it is compromised, every single certificate it ever signed -- and every certificate those signed -- is worthless. So root CAs keep their keys in offline Hardware Security Modules locked in literal vaults. They come online maybe a few times a year to sign intermediate CA certificates. The intermediates handle the daily work. If an intermediate is compromised, you revoke just that intermediate. The root survives.

There are additional operational reasons for the intermediate layer:

Compartmentalization of risk -- Different intermediates can serve different purposes. One intermediate might handle DV certificates, another EV certificates, a third for code signing. If one is compromised, the blast radius is limited.

Policy enforcement -- Intermediates can have Name Constraints limiting them to specific domains. A regional CA might be constrained to only issue certificates for .de domains, for example. If that intermediate is compromised, the attacker can only issue certificates within that constraint.

Agility -- If a cryptographic algorithm needs to be retired (say, SHA-1), you can create new intermediates with the updated algorithm while the root remains stable. Root certificate changes require updating every trust store on every device, which takes years.

Revocation granularity -- Revoking a root is catastrophic. Revoking an intermediate is painful but survivable. Having intermediates gives you a middle ground between "everything is fine" and "burn it all down."

Think of it like a general who never goes to the battlefield. The colonels do the fighting. And if a colonel goes rogue, the general can disown them without the entire army collapsing.

How Chain Verification Works: The Browser's Algorithm

When your browser connects to https://www.example.com, the server sends its certificate and usually the intermediate certificate(s). Your browser then performs chain verification following RFC 5280, section 6.

flowchart TD
    Start([Browser receives server certificate]) --> A{Is the current time<br/>within the certificate's<br/>validity period?}
    A -->|No| Reject1[REJECT: Certificate expired<br/>or not yet valid]
    A -->|Yes| B{Does any SAN entry<br/>match the requested<br/>hostname?}
    B -->|No| Reject2[REJECT: Hostname mismatch]
    B -->|Yes| C{Check Key Usage and<br/>Extended Key Usage:<br/>allowed for TLS server auth?}
    C -->|No| Reject3[REJECT: Wrong key usage]
    C -->|Yes| D{Is the issuer's certificate<br/>available? Check sent chain<br/>or AIA caIssuers}
    D -->|No| Reject4[REJECT: Incomplete chain,<br/>unknown issuer]
    D -->|Yes| E{Verify the cryptographic<br/>signature using the<br/>issuer's public key}
    E -->|Invalid| Reject5[REJECT: Signature<br/>verification failed]
    E -->|Valid| F{Is the issuer certificate<br/>a trusted root in the<br/>local trust store?}
    F -->|Yes| G{Check revocation status:<br/>OCSP stapled response,<br/>CRLite, or OCSP query}
    F -->|No| H{Is the issuer certificate<br/>itself signed by another CA?<br/>Walk up the chain}
    H -->|No| Reject6[REJECT: Chain terminates<br/>at untrusted root]
    H -->|Yes| I{Verify issuer cert:<br/>validity, Basic Constraints CA:TRUE,<br/>pathLen not exceeded}
    I -->|Fail| Reject7[REJECT: Invalid CA<br/>certificate in chain]
    I -->|Pass| E2{Verify issuer cert's<br/>signature with its<br/>issuer's key}
    E2 --> F
    G -->|Revoked| Reject8[REJECT: Certificate<br/>has been revoked]
    G -->|Good| J{Check Certificate<br/>Transparency: are valid<br/>SCTs present?}
    J -->|No| Reject9[REJECT: Missing CT<br/>compliance]
    J -->|Yes| Accept([ACCEPT: Connection trusted])

    style Accept fill:#69db7c,color:#000
    style Reject1 fill:#ff6b6b,color:#fff
    style Reject2 fill:#ff6b6b,color:#fff
    style Reject3 fill:#ff6b6b,color:#fff
    style Reject4 fill:#ff6b6b,color:#fff
    style Reject5 fill:#ff6b6b,color:#fff
    style Reject6 fill:#ff6b6b,color:#fff
    style Reject7 fill:#ff6b6b,color:#fff
    style Reject8 fill:#ff6b6b,color:#fff
    style Reject9 fill:#ff6b6b,color:#fff

# See the full chain a server sends
openssl s_client -connect www.google.com:443 -servername www.google.com \
  -showcerts < /dev/null 2>/dev/null

# Verify a chain explicitly
openssl verify -CAfile root.crt -untrusted intermediate.crt server.crt

The verification is more nuanced than the flowchart suggests. Additional checks include:

- **Path Length Constraints**: The `pathLenConstraint` in Basic Constraints limits how many CA certificates can appear below a given CA in the chain. A root with `pathLen:1` can sign intermediates, but those intermediates cannot sign further sub-CAs.
- **Name Constraints**: Some intermediate CAs are restricted to issuing certificates only for specific domains or IP ranges. A `nameConstraints` extension with `permitted: .example.com` means the CA can only issue for `*.example.com` domains. This is used heavily in enterprise PKI.
- **Policy Constraints**: Certificate Policies and Policy Mappings restrict the purposes a certificate can serve across the chain. Enterprise environments use these to enforce different security levels.
- **Key Usage consistency**: Each certificate in the chain must have appropriate Key Usage for its role. CA certificates must have `keyCertSign`. End-entity certificates must not.
- **Critical extensions**: If a certificate contains a critical extension that the verifier does not understand, the certificate must be rejected. This is the safety mechanism that allows new extensions to be added without older implementations silently ignoring them.

Root Stores: The Foundation of Trust

Who decides which root CAs are in the trust store? That seems like an enormous amount of power -- and it is. It is one of the most important governance questions on the internet. Four organizations effectively decide which CAs the world trusts, and their decisions affect billions of users.

Who Controls the Root Stores?

Mozilla's process is notably transparent. The mozilla.dev.security.policy (now dev-security-policy on Google Groups) mailing list is where trust and distrust decisions are debated. Anyone can participate. When a CA misbehaves, it is discussed openly. When Mozilla debated whether to distrust Symantec's certificates in 2017, the process was entirely public -- hundreds of emails over months. It is messy democratic governance applied to internet trust, and it is arguably the most transparent of the four programs.

The Chrome Root Store is relatively new. Until 2022, Chrome largely relied on the operating system's trust store (Windows, macOS, or Linux). Chrome is now transitioning to its own root store, which gives Google direct control over which CAs Chrome trusts, independent of the OS. This is significant because Chrome has approximately 65% desktop browser market share.

How a CA Joins a Root Store

The process to become a publicly trusted root CA is rigorous and expensive, typically taking one to three years:

1. Build infrastructure -- HSMs for key storage, physically secured data centers, redundant OCSP responders, CT log submission pipelines. The capital investment is in the millions.
2. Write a CP/CPS -- The Certificate Policy (CP) and Certification Practice Statement (CPS) are legal documents describing exactly how the CA operates, how it validates identities, how it stores keys, and how it handles incidents.
3. Pass a WebTrust or ETSI audit -- An independent third-party auditor verifies that the CA actually follows its CP/CPS. This is an annual requirement.
4. Apply to each root store -- Each program has its own application process and requirements. Meeting Mozilla's requirements does not guarantee Apple will accept you.
5. Cross-sign during transition -- New CAs typically have their root signed by an established CA so they can be trusted by older clients that have not yet included the new root.
6. Maintain compliance -- Ongoing annual audits, incident disclosure within 24 hours, adherence to the CA/Browser Forum Baseline Requirements, and responsiveness to root store program inquiries.

Root store operators can and do remove CAs that violate trust. This has happened to:

- **DigiNotar** (2011) -- removed from all root stores after a devastating breach that allowed issuance of fraudulent certificates for Google and other domains
- **CNNIC** (2015) -- removed by Mozilla and Google after an intermediate CA issued unauthorized test certificates for Google domains using a CNNIC-signed intermediate
- **WoSign/StartCom** (2016) -- distrusted by Mozilla, Apple, and Google after backdating SHA-1 certificates to circumvent the SHA-1 deprecation deadline, among other violations
- **Symantec** (2017-2018) -- gradually distrusted by all major browsers after repeated compliance failures spanning years, affecting the largest CA by market share at the time
- **TrustCor** (2022) -- removed by Mozilla after investigative reporting revealed ties to a data-harvesting company with connections to US intelligence contractors

Being removed from root stores effectively kills a CA's business. It is the nuclear option, and it works as a deterrent precisely because it is so devastating.

The DigiNotar Disaster: When Trust Collapses

The DigiNotar breach is the single most important incident in the history of PKI, and it is a masterclass in how not to run a Certificate Authority.

**The DigiNotar Breach (2011)**

DigiNotar was a Dutch Certificate Authority, part of the VASCO group. In June 2011, an attacker -- later attributed to a 21-year-old Iranian hacker calling himself "Comodohacker" (the same person behind the Comodo RA breach earlier that year) -- breached DigiNotar's systems through an unpatched web server.

The attacker gained access to DigiNotar's Certificate Authority signing infrastructure and issued **531 fraudulent certificates** for high-value domains including:
- `*.google.com` -- all Google services, including Gmail
- `*.microsoft.com`, `*.windowsupdate.com`
- `*.skype.com`
- `*.mozilla.org`
- `*.torproject.org`
- `*.wordpress.com`
- Various intelligence agency domains

**Why this was catastrophic:** With a valid certificate for `*.google.com` signed by a trusted CA, an attacker performing a man-in-the-middle attack at an ISP level would show a perfectly valid green padlock. The victim's browser would show zero warnings. Gmail contents, authentication cookies, search queries -- all visible to the attacker in real time.

**How was it discovered?**

A Gmail user in Iran noticed something peculiar. His browser (Chrome) had a feature called **certificate pinning** -- Google had hardcoded the expected certificate fingerprints for Google domains directly into Chrome's source code. When the fraudulent DigiNotar-signed certificate appeared instead of the expected Google-signed certificate, Chrome threw a certificate error. The user reported it on a Google help forum on August 28, 2011.

**The timeline of disaster:**
- **June 17, 2011**: Breach occurs. Attacker gains access to CA infrastructure.
- **June-July 2011**: Attacker issues 531 fraudulent certificates.
- **July 19, 2011**: DigiNotar detects the breach internally through an anomaly in their logging.
- **July 2011**: DigiNotar revokes some fraudulent certificates but does not disclose the breach publicly. They believe they have contained it. They are wrong.
- **August 28, 2011**: The `*.google.com` certificate is detected in the wild by a Chrome user in Iran. Google is notified.
- **August 29, 2011**: Google, Mozilla, and Microsoft begin emergency procedures to distrust all DigiNotar certificates.
- **August 30, 2011**: Google pushes a Chrome update removing DigiNotar from the trust store.
- **September 3, 2011**: Dutch government investigation begins. DigiNotar also ran the PKIoverheid program, which issued certificates for Dutch government services.
- **September 5, 2011**: Fox-IT publishes a forensic analysis revealing the full scope: 531 certificates, multiple CA signing systems compromised, no proper segmentation, outdated software.
- **September 20, 2011**: DigiNotar's parent company, VASCO, files for bankruptcy of the DigiNotar subsidiary.

**The forensic findings were damning:**
- DigiNotar's CA servers were running unpatched Windows software
- The network was not properly segmented -- the attacker moved laterally from a web server to the CA signing infrastructure
- Logging was inadequate, which is why DigiNotar initially thought they had contained the breach
- All CA signing servers were reachable from the compromised network
- No intrusion detection systems flagged the unauthorized certificate issuance

**The real-world impact:** Iranian authorities were performing MITM attacks against Iranian citizens using the fraudulent `*.google.com` certificate. In a country where political dissent can mean imprisonment, torture, or death, intercepting Gmail communications of dissidents was not a theoretical risk -- it was the operational purpose of the attack.

**Lessons:**
1. A single compromised CA can undermine trust for the entire internet
2. Internal detection without public disclosure is worse than useless -- it gives a false sense of containment
3. Certificate pinning (when deployed by Google) caught what the CA trust model could not
4. The consequences of CA failure extend far beyond technology into human rights
5. Network segmentation is not optional for CAs -- the signing infrastructure must be air-gapped
6. Incident response speed matters -- DigiNotar's two-month gap between detection and public disclosure was fatal to their credibility

This is the fundamental weakness of the CA model: you are trusting over a hundred organizations, and any one of them can issue a certificate for any domain. A Malaysian CA can issue a certificate for google.com. A Turkish CA can issue one for your bank. Before Certificate Transparency, the only defense was auditing and the threat of root store removal. DigiNotar proved that was not enough.

Certificate Transparency: Sunlight as Disinfectant

Certificate Transparency (CT) is a system designed to detect misissued certificates quickly. The core insight is simple but powerful: make every certificate publicly visible in append-only logs so that domain owners can monitor for unauthorized certificates issued for their domains.

CT was designed by Google engineers Ben Laurie and Adam Langley in the aftermath of DigiNotar. It became an IETF standard (RFC 6962) and Chrome has required CT compliance for all publicly trusted certificates since April 2018.

How CT Works

sequenceDiagram
    participant CA as Certificate Authority
    participant Log as CT Log Server<br/>(Append-only Merkle tree)
    participant Browser as Browser/Client
    participant Monitor as CT Monitor<br/>(Domain owner's tool)

    CA->>Log: 1. Submit pre-certificate
    Log->>CA: 2. Return SCT<br/>(Signed Certificate Timestamp)<br/>Promise: cert will appear<br/>in log within MMD
    CA->>CA: 3. Embed SCT in certificate<br/>(X.509 extension)

    Note over CA: CA issues the final<br/>certificate with SCTs embedded

    CA->>Browser: 4. Certificate with embedded SCTs<br/>(delivered during TLS handshake)
    Browser->>Browser: 5. Verify SCT signatures<br/>against known log public keys
    Browser->>Browser: 6. Check: enough SCTs<br/>from different logs?

    Note over Log: Log periodically publishes<br/>its Merkle tree root hash

    Monitor->>Log: 7. Poll for new entries<br/>matching watched domains
    Log->>Monitor: 8. Return matching certificates
    Monitor->>Monitor: 9. Alert if unauthorized<br/>certificate found

The system has several key components:

CT Logs -- Append-only data structures based on Merkle trees (the same hash tree structure used in blockchains). Each log is operated by an independent organization (Google, Cloudflare, DigiCert, Sectigo, and others). Append-only means certificates can be added but never removed or modified. The Merkle tree structure means anyone can cryptographically verify that a certificate they see was included in the log and that no entries have been tampered with.

Signed Certificate Timestamps (SCTs) -- When a CA submits a certificate to a CT log, the log returns an SCT: a signed promise that the certificate will appear in the log within the Maximum Merge Delay (typically 24 hours). The SCT is embedded in the certificate as an X.509 extension. Chrome requires SCTs from at least two different log operators.

Monitors -- Software that watches CT logs for certificates matching specific domains. Domain owners run monitors to detect unauthorized certificates. If a CA issues a certificate for example.com without the domain owner's knowledge, the monitor will catch it -- typically within hours.

Auditors -- Software that verifies the integrity of CT logs themselves, ensuring logs are behaving honestly (not removing entries or presenting different views to different clients).

# Search Certificate Transparency logs for certificates issued for a domain
# Using crt.sh (a public CT log search engine run by Sectigo)
curl -s "https://crt.sh/?q=example.com&output=json" | python3 -m json.tool | head -50

# Check if a certificate has embedded SCTs
openssl s_client -connect www.google.com:443 -servername www.google.com \
  < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -A10 "CT Precertificate SCTs"

Set up a CT monitor for a domain you own:

1. Go to [https://crt.sh](https://crt.sh) and search for your domain
2. See every certificate ever issued for it, with issuance dates and CA names
3. For ongoing monitoring, tools like SSLMate's Certspotter provide email alerts:

\```bash
# Install certspotter (Go-based)
go install software.sslmate.com/src/certspotter/cmd/certspotter@latest

# Watch for new certificates
certspotter -watchlist example.com
\```

This is how you would catch a rogue CA issuing unauthorized certificates for your domain. The DigiNotar attack went undetected for over two months. With CT monitoring, it would have been caught within hours.

CT does not prevent misissued certificates -- it makes them visible. But visibility is enormously powerful. Before CT, a rogue CA could issue a fraudulent certificate and it might never be discovered. Now, that certificate will appear in a public log within hours. Combined with automated monitoring, domain owners can detect misuse and demand revocation quickly. CT also creates accountability -- CAs know that every certificate they issue is publicly auditable, which creates a strong incentive to follow proper procedures.

Self-Signed Certificates and Private CAs

Not all certificates come from public CAs. Organizations often run their own internal Certificate Authority for services that do not need public trust.

Why would you run your own CA when Let's Encrypt is free? Several reasons. Internal services on private networks often cannot complete the ACME challenges that Let's Encrypt requires (HTTP-01 needs port 80 accessible from the internet, DNS-01 needs API access to your DNS provider). You might need to issue client certificates for mutual TLS. You might need certificates for non-HTTP protocols like gRPC, MQTT, or internal database connections. Or you might need certificates with custom fields or extensions for internal policy enforcement that public CAs will not include.

Creating a Private CA

# Generate the root CA private key (4096-bit RSA, AES-256 encrypted)
openssl genrsa -aes256 -out rootCA.key 4096

# Create the self-signed root certificate (valid for 10 years)
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 \
  -out rootCA.crt \
  -subj "/C=IN/ST=Karnataka/L=Bangalore/O=Acme Corp/OU=Security/CN=Acme Root CA"

# Verify the root certificate
openssl x509 -in rootCA.crt -noout -text | head -20

# Generate the server's private key
openssl genrsa -out server.key 2048

# Create a Certificate Signing Request (CSR)
openssl req -new -key server.key -out server.csr \
  -subj "/C=IN/ST=Karnataka/L=Bangalore/O=Acme Corp/CN=internal.acme.local"

# Sign the CSR with your root CA
openssl x509 -req -in server.csr -CA rootCA.crt -CAkey rootCA.key \
  -CAcreateserial -out server.crt -days 365 -sha256 \
  -extfile <(printf "subjectAltName=DNS:internal.acme.local,DNS:*.acme.local\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth")

# Verify the chain
openssl verify -CAfile rootCA.crt server.crt

For production internal CAs, consider tools that automate this:

• step-ca (Smallstep) -- Open-source CA with ACME support, short-lived certificates, and easy setup
• HashiCorp Vault PKI -- PKI secrets engine with API-driven certificate issuance
• EJBCA -- Enterprise-grade Java CA for complex PKI deployments
• cfssl (Cloudflare) -- Simple, lightweight CA toolkit

Self-signed certificates and private CAs require careful key management:

- Store the root CA key **offline** and encrypted. Use an HSM if budget allows. The root key should only be used to sign intermediate CA certificates, not end-entity certificates.
- **Never** use the root CA key on a server connected to any network.
- Create an **intermediate CA** for daily certificate issuance, even internally. This mirrors public PKI best practices.
- Document your CA's certificate policy, even if it is internal. When people leave the team, the documentation is what survives.
- Distribute your root CA certificate to all clients that need to trust it via configuration management (Ansible, Puppet, Chef) or MDM for mobile devices.
- **Track certificate expiration dates** -- internal certificates cause outages just like public ones, and they are often less monitored because there is no external service watching them.

Certificate Types: DV, OV, EV

Not all publicly trusted certificates are created equal. There are three validation levels, each requiring progressively more verification of the certificate requester's identity.

Domain Validation (DV)

• What the CA verifies: You control the domain (via DNS record, HTTP file challenge, or email to admin@domain)
• What the certificate shows: No organization name. Just the domain in the SAN.
• Cost: Free (Let's Encrypt, ZeroSSL) to inexpensive ($10-50/year)
• Issuance time: Seconds to minutes (fully automated)
• Use case: Personal sites, SaaS applications, APIs, almost everything

Organization Validation (OV)

• What the CA verifies: Domain control plus the organization's legal existence (business registration, phone verification, physical address)
• What the certificate shows: Organization name in the Subject field (O= and L= fields), but browsers do not display this prominently
• Cost: $50-$200/year
• Issuance time: 1-3 business days
• Use case: Business websites that want verifiable legal identity embedded in the certificate

Extended Validation (EV)

• What the CA verifies: Extensive vetting including legal existence, operational existence, physical address, authorized representatives, and domain control
• What the certificate shows: Previously displayed the company name in a green address bar. Most browsers removed this indicator in 2019-2020.
• Cost: $200-$1000+/year
• Issuance time: 1-2 weeks
• Use case: Debatable since the green bar removal

Browsers removed the green bar because research by Google, Mozilla, and academic institutions showed that users did not notice it, did not understand what it meant, and made no different security decisions based on it. Phishing sites used DV certificates and looked just as legitimate to users. Ian Carroll famously registered "Stripe, Inc" in Kentucky (a different entity from the payment company) and obtained an EV certificate showing "Stripe, Inc" in the green bar, demonstrating that EV did not provide the identity assurance people assumed. Google and Mozilla concluded that the EV visual indicator was providing false security signals and removed it. The certificate still contains verified organization information, but since users cannot see it without digging into certificate details, its practical security value for most scenarios is minimal.

Wildcard and Multi-Domain Certificates

Wildcard Certificates

A wildcard certificate covers a domain and all its single-level subdomains. The wildcard character * matches exactly one label.

Certificate SAN: *.example.com
Covers:      www.example.com      Yes
             api.example.com      Yes
             mail.example.com     Yes
             example.com          No -- bare domain not covered unless also listed
             sub.api.example.com  No -- multi-level subdomain not covered

Wildcard certificates introduce significant security trade-offs:

- **Blast radius**: If the private key is compromised, the attacker can impersonate *any* subdomain. A breach of the marketing blog server would let the attacker impersonate the payment processing subdomain.
- **Key distribution**: The same private key must exist on every server that serves any subdomain. More copies means more exposure points and more complex key rotation.
- **Compliance implications**: PCI DSS 4.0 (requirement 2.2.5) requires that wildcard certificates be justified and that the risk of the wildcard scope is documented. Some auditors push back on wildcards in cardholder data environments.
- **No sub-subdomain coverage**: `*.example.com` does not cover `staging.api.example.com`. You need separate certificates or `*.api.example.com` for that.

For high-security environments, issue individual certificates per service. The operational overhead is manageable with automation tools like cert-manager or certbot.

Subject Alternative Names (SANs)

Modern certificates use the Subject Alternative Name extension to list all valid names. This is the only field browsers check for hostname matching.

# Create a certificate with multiple SANs
openssl req -new -key server.key -out multi.csr \
  -subj "/CN=example.com" \
  -addext "subjectAltName=DNS:example.com,DNS:www.example.com,DNS:api.example.com,IP:10.0.1.50"

# Inspect SANs on an existing certificate
openssl x509 -in cert.pem -noout -ext subjectAltName

Fun fact -- browsers have ignored the Common Name (CN) field since around 2017. They only look at Subject Alternative Names. The CN field is purely legacy. If your certificate has a CN but no matching SAN, modern browsers will reject it. Some older tools like curl on certain Linux distributions still fall back to CN, which can mask the problem during testing.

Certificate Formats and Encoding

Certificates come in several file formats, and the naming conventions are a historical mess that continues to cause confusion:

# Convert PEM to DER
openssl x509 -in cert.pem -outform DER -out cert.der

# Convert DER to PEM
openssl x509 -in cert.der -inform DER -outform PEM -out cert.pem

# Create a PKCS#12 bundle (cert + key + chain)
openssl pkcs12 -export -out bundle.p12 \
  -inkey server.key -in server.crt -certfile intermediate.crt

# Extract certificate from PKCS#12
openssl pkcs12 -in bundle.p12 -clcerts -nokeys -out extracted.crt

# Extract private key from PKCS#12
openssl pkcs12 -in bundle.p12 -nocerts -nodes -out extracted.key

# View contents of a PKCS#12 file
openssl pkcs12 -in bundle.p12 -info -nokeys

Why so many formats? Historical cruft from different platforms standardizing at different times in the 1990s. Java keystores used JKS, then migrated to PKCS#12. Windows loves PFX (which is essentially PKCS#12). Linux and open-source tools prefer PEM because it is human-readable and can be concatenated. You will spend an unreasonable amount of your career converting between these formats. The .cer extension is particularly treacherous because it might be PEM or DER depending on who created it -- you have to inspect the file to know.

Common Certificate Mistakes

**The Intermediate Certificate Gap**

A production TLS error once took four hours to debug because it only affected Android devices. Desktop browsers worked fine. Mobile Safari worked fine. But Android Chrome and the Java HTTP client both failed with "unable to verify the first certificate."

The culprit? The server was sending only the end-entity certificate, not the intermediate. Desktop browsers had cached the intermediate from a previous visit to another site using the same CA. iOS had the intermediate bundled in its trust store. But Android and fresh Java installations had never seen it and could not build the chain.

The fix was a single line in the nginx configuration:

\```nginx
ssl_certificate /etc/nginx/certs/fullchain.pem;  # cert + intermediate
\```

Instead of:

\```nginx
ssl_certificate /etc/nginx/certs/cert.pem;  # cert only -- WRONG
\```

**Always send the full chain** (end-entity + intermediates). Never send the root -- the client must already have it in its trust store. This is the most common TLS misconfiguration encountered in production, and it is insidious because it works on most browsers (which cache intermediates) and only fails on fresh clients or specific platforms.

Test with: `openssl s_client -connect yoursite.com:443 | grep "Verify return code"`

If the result is not `0 (ok)`, your chain is broken.

Other frequent mistakes ranked by how often they cause production incidents:

1. Expired certificates -- No monitoring, certificate quietly expires, outage at 3 AM. This has taken down Slack, Microsoft Teams, and countless smaller services.
2. Missing intermediate certificates -- Works on cached browsers, fails on Android/Java/curl.
3. Hostname mismatch -- Certificate says www.example.com but the request is to example.com (bare domain not in SAN).
4. Wrong chain order -- Intermediates must be concatenated in order: end-entity first, then signing intermediate, then next intermediate. Reversed order causes parsing failures.
5. Key mismatch -- The certificate's public key does not match the private key on the server. This happens during key rotation when the wrong key file is referenced.

# Verify that a certificate and private key match
CERT_MOD=$(openssl x509 -in server.crt -noout -modulus | openssl md5)
KEY_MOD=$(openssl rsa -in server.key -noout -modulus | openssl md5)
echo "Cert: $CERT_MOD"
echo "Key:  $KEY_MOD"
# These MUST be identical. If they differ, the TLS handshake will fail.

# Verify a certificate chain
openssl verify -CAfile rootCA.crt -untrusted intermediate.crt server.crt

# Check certificate expiration for a remote server
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -enddate -checkend 2592000
# -checkend 2592000 exits with code 1 if cert expires within 30 days

The Trust Model's Philosophical Problem

The web PKI is a trust oligarchy. A relatively small number of organizations hold enormous power over internet security. The system works mostly because:

1. The economic incentives are aligned -- CAs lose their entire business if they lose trust
2. Certificate Transparency provides public auditability for every certificate issued
3. Browser vendors act as enforcement -- they will distrust bad CAs regardless of the CA's size
4. The CA/Browser Forum sets minimum technical and operational standards through the Baseline Requirements

But it is imperfect. There are alternative models that have been proposed:

• DANE/TLSA (RFC 6698) -- DNS-based Authentication of Named Entities. The domain owner publishes certificate information in DNS using DNSSEC-signed TLSA records. This removes the need for CA trust entirely -- the domain owner declares which certificate to expect. The limitation is that DANE requires DNSSEC, which has low deployment, and browsers have not adopted it (Chrome explicitly chose not to).
• Web of Trust -- PGP-style, where individuals vouch for each other's keys. This does not scale for the web because it requires humans to verify key fingerprints.
• Trust on First Use (TOFU) -- SSH-style, where you trust the key the first time you see it and alert on changes. This is vulnerable to first-connection attacks but works reasonably well for SSH where the connection model is different.

None of these have replaced the CA model for web browsing. The CA system, despite its flaws, has achieved something remarkable: it provides usable encryption for billions of users who never think about certificates. With Certificate Transparency providing a detection layer, it is good enough -- though "good enough" in security always makes engineers uncomfortable, as it should.

What You've Learned

In this chapter, you explored the certificate infrastructure that underpins trust on the internet:

• X.509 certificates bind public keys to identities through a structured set of fields -- Subject, Issuer, Validity, Public Key, Extensions, and Signature -- each designed to prevent specific attacks discovered over decades of PKI operation
• PKI hierarchy uses root CAs, intermediate CAs, and end-entity certificates to distribute trust while protecting root keys offline; the intermediate layer provides compartmentalization, policy enforcement, and revocation granularity
• Chain verification follows a specific algorithm (RFC 5280) checking validity, hostname matching, key usage, signature verification, path constraints, revocation status, and CT compliance
• Root stores managed by Mozilla, Apple, Google, and Microsoft are the ultimate gatekeepers of internet trust, with governance ranging from Mozilla's public process to Apple's internal review
• The DigiNotar breach demonstrated that a single compromised CA can threaten human lives, and that delayed disclosure is fatal to a CA's credibility
• Certificate Transparency logs make every issued certificate publicly auditable, enabling detection of fraudulent certificates within hours instead of the months it took before CT
• Certificate types (DV, OV, EV) differ in validation rigor, but browsers have largely eliminated visible differences to users
• Practical skills: inspecting, verifying, and creating certificates using openssl commands; understanding format conversions; debugging common chain issues

That browser warning you saw? Your browser was doing exactly what it should. It walked the trust chain, hit an untrusted issuer, and protected you. The system is complex, but when it works, it saves you from attacks you never even see. And when it doesn't work, you get DigiNotar -- which is why the industry builds layers. Trust, but verify. And then verify the verification. Welcome to security engineering.