Configure XtremeCloud Single Sign-On (SSO) for Smart Card Usage

Low Impact Secure Authentication

Introduction

Personal Identity Verification (PIV) Credential

Protecting and keeping government assets secure such as facilities, computers, or information systems, are a fundamental responsibility of government employees. To ensure consistent implementation across the federal government, Homeland Security Presidential Directive12 (HSPD-12) was enacted. HSPD-12 provides for a new standardized federal identity credential that is designed to enhance security, reduce identity fraud, and protect the personal privacy of those issued government identification.

The General Services Administration (GSA) HSPD-12 Managed Service Office (MSO) established the USAccess program as an efficient way for Federal agencies to issue common HSPD-12 approved credentials to their employees and contractors. If an agency has elected to participate in the USAccess Program employees and contractors will receive a USAccess credential. The PIV credential may be referred to as a different name within agencies because some agencies have opted to re-brand their program and credential. Different names include, amongst others: LincPass, Smart Card, and the Common Access Card (CAC) card.

Background

X.509 Client Certificate User Authentication for XtremeCloud Single Sign-On (SSO)

A typical workflow is as follows:

  • A client sends an authentication request over a SSL/TLS channel.

  • During SSL/TLS handshake, the server and the client exchange their x.509/v3 certificates.

  • The container (WildFly) validates the certificate PKIX path and the certificate expiration.

  • The x.509 client certificate authenticator validates the client certificate as follows:

    • Optionally checks the certificate revocation status using CRL and/or CRL Distribution Points.

    • Optionally checks the Certificate revocation status using the Online Certificate Status Protocol (OCSP).

    • Optionally validates whether the key usage in the certificate matches the expected key usage.

    • Optionally validates whether the extended key usage in the certificate matches the expected extended key usage.

  • If any of the above checks fails, the x.509 authentication fails.

  • Otherwise, the client certificate is passed through to the XtremeCloud SSO oAuth2 authenticator and extracts the user’s identity from the Commmon Name (CN) of the digital certificate and maps it to an existing user in the user store.

Note: This Kubernetes annotation in our NGINX Ingress Controller allows the client certificate to pass through one of the upstream XtremeCloud SSO Kubernetes pods: nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: “true”

  • Once the certificate is mapped to an existing user, the behavior diverges depending on the authentication flow:

    • In the Browser Flow, the server prompts the user to confirm identity or to ignore it and instead sign in with username/password

    • In the case of the Direct Grant Flow, the XtremeCloud SSO authenticates the user

Features

Supported Certificate Identity Sources

  • Match SubjectDN using regular expression

  • X500 Subject’s e-mail attribute

  • X500 Subject’s e-mail from Subject Alternative Name Extension (RFC822Name General Name)

  • X500 Subject’s Common Name attribute

  • Match IssuerDN using regular expression

  • X500 Issuer’s e-mail attribute

  • X500 Issuer’s Common Name attribute

  • Certificate Serial Number

Regular Expressions The certificate identity can be extracted from either Subject DN or Issuer DN using a regular expression as a filter. For example, the regular expression below will match the e-mail attribute: ``` emailAddress=(.*?)(?:,|$) ``` The regular expression filtering is applicable only if the Identity Source is set to either Match SubjectDN using regular expression or Match IssuerDN using regular expression.

Mapping certificate identity to an existing user

The certificate identity mapping can be configured to map the extracted user identity to an existing user’s username or e-mail or to a custom attribute which value matches the certificate identity. For example, setting the Identity source to Subject’s e-mail and User mapping method to Username or email will have the X.509 client certificate authenticator use the e-mail attribute in the certificate’s Subject DN as a search criteria to look up an existing user by username or by e-mail.

Important

Please notice that if we disable Login with email at realm settings, the same rules will be applied to certificate authentication. In other words, users won’t be able to log in using e-mail attribute.

Important

Usage of Certificate Serial Number and IssuerDN as an identity source requires two custom attributes - one for serial number and the other for IssuerDN.

Other Features: Extended Certificate Validation

  • Revocation status checking using CRL.

  • Revocation status checking using CRL/Distribution Point.

  • Revocation status checking using OCSP/Responder URI.

  • Certificate KeyUsage validation.

  • Certificate ExtendedKeyUsage validation.

Enable X.509 Client Certificate User Authentication

Enable mutual SSL in our WildFly Application Server using the Kubernetes configMap** that update the application servers **standalone-ha.xml* configuration file:

<security-realms>
    <security-realm name="ssl-realm">
        <server-identities>
            <ssl>
                <keystore path="servercert.jks"
                          relative-to="jboss.server.config.dir"
                          keystore-password="servercert password"/>
            </ssl>
        </server-identities>
        <authentication>
            <truststore path="truststore.jks"
                        relative-to="jboss.server.config.dir"
                        keystore-password="truststore password"/>
        </authentication>
    </security-realm>
</security-realms>

ssl/keystore

The ssl element contains the keystore element that defines how to load the server public key pair from a JKS keystore

ssl/keystore/path

A path to a JKS keystore

ssl/keystore/relative-to

Defines a path the keystore path is relative to

ssl/keystore/keystore-password

The password to open the keystore

ssl/keystore/alias (optional)

The alias of the entry in the keystore. Set it if the keystore contains multiple entries

ssl/keystore/key-password (optional)

The private key password, if different from the keystore password.

authentication/truststore

Defines how to load a trust store to verify the certificate presented by the remote side of the inbound/outgoing connection. Typically, the truststore contains a collection of trusted CA certificates.

authentication/truststore/path

A path to a JKS keystore that contains the certificates of the trusted CAs (certificate authorities)

authentication/truststore/relative-to

Defines a path the truststore path is relative to

authentication/truststore/keystore-password

The password to open the truststore

Enable https listener

  • Add the <https-listener> element as shown below:
<subsystem xmlns="{subsystem_undertow_xml_urn}">
        ....
    <server name="default-server">
            <https-listener name="default"
                        socket-binding="https"
                        security-realm="ssl-realm"
                        verify-client="REQUESTED"/>
    </server>
</subsystem>

https-listener/security-realm

The value must match the name of the realm from the previous section

https-listener/verify-client

If set to REQUESTED, the server will optionally ask for a client certificate. Setting the attribute to REQUIRED will have the server to refuse inbound connections if no client certificate has been provided.

Adding X.509 Client Certificate Authentication to a Browser Flow

  • Select a realm, click on Authentication link, select the “Browser” flow.

  • Make a copy of the built-in “Browser” flow. You may want to give the new flow a distinctive name, i.e. “X.509 Browser”.

  • Using the drop down, select the copied flow, and click on “Add execution”.

  • Select “X509/Validate User Form” using the drop down and click on “Save”.

  • Using the up/down arrows, change the order of the “X509/Validate Username Form” by moving it above the “Browser Forms” execution, and set the requirement to “ALTERNATIVE”

  • Select the “Bindings” tab, find the drop down for “Browser Flow”. Select the newly created X509 browser flow from the drop down and click on “Save”.

Configuring X.509 Client Certificate Authentication

User Identity Source Defines how to extract the user identity from a client certificate.

A regular expression (optional) Defines a regular expression to use as a filter to extract the certificate identity. The regular expression must contain a single group.

User Mapping Method Defines how to match the certificate identity to an existing user. Username or e-mail will search for an existing user by username or e-mail. Custom Attribute Mapper will search for an existing user with a custom attribute which value matches the certificate identity. The name of the custom attribute is configurable.

A name of user attribute (optional) A custom attribute which value will be matched against the certificate identity.

CRL Checking Enabled (optional) Defines whether to check the revocation status of the certificate using Certificate Revocation List.

Enable CRL Distribution Point to check certificate revocation status (optional) Defines whether to use CDP to check the certificate revocation status. Most PKI authorities include CDP in their certificates.

CRL file path (optional) Defines a path to a file that contains a CRL list. The value must be a path to a valid file if CRL Checking Enabled option is turned on.

OCSP Checking Enabled(optional) Defines whether to check the certificate revocation status using Online Certificate Status Protocol.

OCSP Responder URI (optional) Allows to override a value of the OCSP responder URI in the certificate.

Validate Key Usage (optional) Verifies whether the certificate’s KeyUsage extension bits are set. For example, “digitalSignature,KeyEncipherment” will verify if bits 0 and 2 in the KeyUsage extension are asserted. Leave the parameter empty to disable the Key Usage validation. See RFC5280, Section-4.2.1.3. The server will raise an error only when flagged as critical by the issuing CA and there is a key usage extension mismatch.

Validate Extended Key Usage (optional) Verifies one or more purposes as defined in the Extended Key Usage extension. See RFC5280, Section-4.2.1.12. Leave the parameter empty to disable the Extended Key Usage validation. The server will raise an error only when flagged as critical by the issuing CA and there is a key usage extension mismatch.

Bypass identity confirmation If set, X.509 client certificate authentication will not prompt the user to confirm the certificate identity and will automatically sign in the user upon successful authentication.

Adding X.509 Client Certificate Authentication to a Direct Grant Flow

    • Make a copy of the build-in “Direct Grant” flow. You may want to give the new flow a distinctive name, i.e. “X509 Direct Grant”,
  • Delete “Username Validation” and “Password” authenticators,

  • Click on “Add execution” and add “X509/Validate Username” and click on “Save” to add the execution step to the parent flow.

  • Change the Requirement to REQUIRED.

  • Set up the x509 authentication configuration by following the steps described earlier in the x.509 Browser Flow section.

  • Select the “Bindings” tab, find the drop down for “Direct Grant Flow”. Select the newly created X509 direct grant flow from the drop down and click on “Save”.

Client certificate lookup

HAProxy certificate lookup provider

<spi name="x509cert-lookup">
    <default-provider>haproxy</default-provider>
    <provider name="haproxy" enabled="true">
        <properties>
            <property name="sslClientCert" value="SSL_CLIENT_CERT"/>
            <property name="sslCertChainPrefix" value="CERT_CHAIN"/>
            <property name="certificateChainLength" value="10"/>
        </properties>
    </provider>
</spi>

In this example configuration, the client certificate will be looked up from the HTTP header, SSL_CLIENT_CERT, and the other certificates from its chain will be looked up from HTTP headers like CERT_CHAIN_0 , CERT_CHAIN_1, …, CERT_CHAIN_9 . The attribute certificateChainLength is the maximum length of the chain, so the last one tried attribute would be CERT_CHAIN_9 .

Consult the HAProxy documentation for the details of how the HTTP Headers for the client certificate and client certificate chain can be configured and their proper names.

Apache certificate lookup provider

<spi name="x509cert-lookup">
    <default-provider>apache</default-provider>
    <provider name="apache" enabled="true">
        <properties>
            <property name="sslClientCert" value="SSL_CLIENT_CERT"/>
            <property name="sslCertChainPrefix" value="CERT_CHAIN"/>
            <property name="certificateChainLength" value="10"/>
        </properties>
    </provider>
</spi>

The configuration is same as for the haproxy provider. Consult the Apache documentation on mod_ssl and mod_headers for the details of how the HTTP Headers for the client certificate and client certificate chain can be configured and their proper names.

Nginx certificate lookup provider

<spi name="x509cert-lookup">
    <default-provider>nginx</default-provider>
    <provider name="nginx" enabled="true">
        <properties>
            <property name="sslClientCert" value="ssl-client-cert"/>
            <property name="sslCertChainPrefix" value="USELESS"/>
            <property name="certificateChainLength" value="2"/>
        </properties>
    </provider>
</spi>

Note

Consult the NGINX documentation for the details of how the HTTP Headers for the client certificate can be configured. Example of NGINX configuration file :

 ...
 server {
    ...
    ssl_client_certificate                  trusted-ca-list-for-client-auth.pem;
    ssl_verify_client                       optional_no_ca;
    ssl_verify_depth                        2;
    ...
    location / {
      ...
      proxy_set_header ssl-client-cert        $ssl_client_escaped_cert;
      ...
    }
    ...
}

Note

Other reverse proxy implementations

Troubleshooting

Dumping HTTP headers If you want to view what the reverse proxy is sending to XtremeCloud SSO, simply activate RequestDumpingHandler and consult server.log file.

Enable TRACE logging under the logging subsystem

...
    <profile>
        <subsystem xmlns="urn:jboss:domain:logging:3.0">
...
            <logger category="org.keycloak.authentication.authenticators.x509">
                <level name="TRACE"/>
            </logger>
            <logger category="org.keycloak.services.x509">
                <level name="TRACE"/>
            </logger>

WARNING: Don't use RequestDumpingHandler or TRACE logging in production.

Direct Grant authentication with X.509 The following template can be used to request a token using the Resource Owner Password Credentials Grant:

``` $ curl host[:port]/auth/realms/master/protocol/openid-connect/token \ –insecure \ –data “grant_type=password&scope=openid profile&username=&password=&client_id=CLIENT_ID&client_secret=CLIENT_SECRET” \ -E /path/to/client_cert.crt \ –key /path/to/client_cert.key ```

[host][:port]

CLIENT_ID A client id.

CLIENT_SECRET For confidential clients, a client secret; otherwise, leave it empty.

client_cert.crt A public key certificate that will be used to verify the identity of the client in mutual SSL authentication. The certificate should be in PEM format.

client_cert.key A private key in the public key pair. Also expected in PEM format.

Getting Started

Since we are terminating SSL at the NGINX Kubernetes Ingress Controller (KIC), as described here, we have to deal with the client certificate chain.

The NGINX SSL/TLS module does not expose the client certificate chain, so the XtremeCloud Single Sign-On (SSO) NGINX certificate lookup provider is rebuilding it using the contents of keycloak.jks. We will populate the XtremeCloud SSO truststore using the keytool CLI with all root and intermediate CA’s needed for rebuilding the client certificate chain.

Prepare the Java Keystore (JKS) and Kubernetes ConfigMap

We are following the same procedure as outlined in this How-to Guide. Since this Kubernetes ConfigMap is being injected into a running container, this procedure is part of setting up a secure and sanitized Kubernetes Cluster.

Import the DoD Root Certificates

Create the secret for the NGINX Ingress Controller

kubectl create secret generic ca-secret --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=dod-certs/ca.crt

MilConnect for CAC Login

We are using the openssl s_client tool to see what CACs issued by the CAs will be accepted:

[centos@vm-controller certificate-management]$ openssl s_client -showcerts -connect pki.dmdc.osd.mil:443
CONNECTED(00000003)
depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
verify return:1
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify return:1
depth=0 C = US, ST = California, L = Seaside, O = DMDC, CN = dmdc.mil
verify return:1
140537805997968:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1493:SSL alert number 40
140537805997968:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=California/L=Seaside/O=DMDC/CN=dmdc.mil
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
-----BEGIN CERTIFICATE-----
MIIHbzCCBlegAwIBAgIQQ6Xh7EjG2JUAAAAAUO8cNzANBgkqhkiG9w0BAQsFADCB
ujELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsT
H1NlZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAy
MDEyIEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEuMCwG
A1UEAxMlRW50cnVzdCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEwxSzAeFw0x
OTAzMTkxNjAxNTlaFw0yMDAzMTQxNjMxNThaMFYxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIEwpDYWxpZm9ybmlhMRAwDgYDVQQHEwdTZWFzaWRlMQ0wCwYDVQQKEwRETURD
MREwDwYDVQQDEwhkbWRjLm1pbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALPB0eq7ZwTuSubzZx18fA48SvxyvyuObkkf6fTINsVIDS9r9vMzio+0OkZZ
QfMKXnsNFQ4xsz1qDRerJC3Y6Gy7YPC3osqE6/PeZRPNlegtQ+/FMA6PPIo8yPdo
WIrbheEgmkCEar2+hEXi5RTueUjM9mmxB6L3ff83tLDxBq3U3HegDFSzmKg6cBmO
fJHqVKh4MJYWpdLkvvbeIx2Y6XDTBUGmCX25ZexgRnvLg89H/VV8p2VPLxHFBGAk
DDJpQRoqUN7lErWm7yDpABWeDV57phe18yfe40SZIL7MZ6fHZxtv2MwR7U1uBUiB
3B1yT0HT3ElJ1V5fAEcrof2G7SECAwEAAaOCA9IwggPOMIHjBgNVHREEgdswgdiC
CGRtZGMubWlsggoqLmRtZGMubWlsggxkbWRjLm9zZC5taWyCHyoubWlsaXRhcnlm
dW5lcmFsaG9ub3JzLm9zZC5taWyCDCouZG9kdGFwLm1pbIIQKi5kb2RzdXJ2ZXlz
Lm1pbIIJKi5jYXAubWlsgg4qLmRtZGMub3NkLm1pbIISKi5pbnQuZG1kYy5vc2Qu
bWlsgh1taWxpdGFyeWZ1bmVyYWxob25vcnMub3NkLm1pbIIKZG9kdGFwLm1pbIIO
ZG9kc3VydmV5cy5taWyCB2NhcC5taWwwggF+BgorBgEEAdZ5AgQCBIIBbgSCAWoB
aAB1AId1v+dZfPiMQ5lfvfNu/1aNR1Y2/0q1YMG06v9eoIMPAAABaZbLnhkAAAQD
AEYwRAIgUFJok+OTOX7SLGWRr+3nHZ3WY5QXFtrxwTC2JS6pgoYCIF0F2r3GjGvp
2v33I5XIoBdz6dLiEUJzEtI9RQQozbVGAHcAVYHUwhaQNgFK6gubVzxT8MDkOHhw
JQgXL6OqHQcT0wwAAAFplsueMQAABAMASDBGAiEA15JwDQKT4wUHQw2/zl5QPT8f
vUIYxH6gglwCx7/LhLUCIQCckdQrVi/0T2/w+sz9dO33oa4qgJD2ja0QbPxY2J5T
uQB2ALvZ37wfinG1k5Qjl6qSe0c4V5UKq1LoGpCWZDaOHtGFAAABaZbLnlwAAAQD
AEcwRQIgZ83EoR+Jte51HybStrQ//oY5e9ZWoeHQrcFynaG4kMsCIQCDP55f22ia
Du4KD7Wp5giHlQ9e3z7V3yfDK2I32e+C8zAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0l
BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMDMGA1UdHwQsMCowKKAmoCSGImh0dHA6
Ly9jcmwuZW50cnVzdC5uZXQvbGV2ZWwxay5jcmwwSwYDVR0gBEQwQjA2BgpghkgB
hvpsCgEFMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly93d3cuZW50cnVzdC5uZXQvcnBh
MAgGBmeBDAECAjBoBggrBgEFBQcBAQRcMFowIwYIKwYBBQUHMAGGF2h0dHA6Ly9v
Y3NwLmVudHJ1c3QubmV0MDMGCCsGAQUFBzAChidodHRwOi8vYWlhLmVudHJ1c3Qu
bmV0L2wxay1jaGFpbjI1Ni5jZXIwHwYDVR0jBBgwFoAUgqJwdN28Uz/Pe9T3zX+n
YMYKTL8wHQYDVR0OBBYEFAK/zneo+KVmuDK1ppnTdqeF79LEMAkGA1UdEwQCMAAw
DQYJKoZIhvcNAQELBQADggEBAKj9VGgD9LENT6ka3msekpfy0Ty7NqTub9JWTClf
z8y0huxD8MqXz3qQXJ2tOghDGMf/gsQzrTivDq+xO75p7v0SRh+9HZhg4ZJfjFg5
BBRkSsC/yl6YgAWbrh23+a7HKY8o5khkVjZLWdrha9CwTSKtev3o/G2qNNcEPss5
d3HrLdVs8D5/8RPiRpZnROFUEsJdB1wlkV7JQ0XqYo0CjFi42GNtpCEXKIHgu41d
rBkxUg6E8SgBpxAU4+360rIrpeRkTPzQJHGjcCJ88zEYdN0XsUfWSBc1unnOItVk
2VThAoxV9xcKYL/uSUPy/k1/E0mh2kRCNu4DhG8imy9Msqw=
-----END CERTIFICATE-----
 1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
-----BEGIN CERTIFICATE-----
MIIFDjCCA/agAwIBAgIMDulMwwAAAABR03eFMA0GCSqGSIb3DQEBCwUAMIG+MQsw
CQYDVQQGEwJVUzEWMBQGA1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2Vl
IHd3dy5lbnRydXN0Lm5ldC9sZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMDkg
RW50cnVzdCwgSW5jLiAtIGZvciBhdXRob3JpemVkIHVzZSBvbmx5MTIwMAYDVQQD
EylFbnRydXN0IFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBHMjAeFw0x
NTEwMDUxOTEzNTZaFw0zMDEyMDUxOTQzNTZaMIG6MQswCQYDVQQGEwJVUzEWMBQG
A1UEChMNRW50cnVzdCwgSW5jLjEoMCYGA1UECxMfU2VlIHd3dy5lbnRydXN0Lm5l
dC9sZWdhbC10ZXJtczE5MDcGA1UECxMwKGMpIDIwMTIgRW50cnVzdCwgSW5jLiAt
IGZvciBhdXRob3JpemVkIHVzZSBvbmx5MS4wLAYDVQQDEyVFbnRydXN0IENlcnRp
ZmljYXRpb24gQXV0aG9yaXR5IC0gTDFLMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEA2j+W0E25L0Tn2zlem1DuXKVh2kFnUwmqAJqOV38pa9vH4SEkqjrQ
jUcj0u1yFvCRIdJdt7hLqIOPt5EyaM/OJZMssn2XyP7BtBe6CZ4DkJN7fEmDImiK
m95HwzGYei59QAvS7z7Tsoyqj0ip/wDoKVgG97aTWpRzJiatWA7lQrjV6nN5ZGhT
JbiEz5R6rgZFDKNrTdDGvuoYpDbwkrK6HIiPOlJ/915tgxyd8B/lw9bdpXiSPbBt
LOrJz5RBGXFEaLpHPATpXbo+8DX3Fbae8i4VHj9HyMg4p3NFXU2wO7GOFyk36t0F
ASK7lDYqjVs1/lMZLwhGwSqzGmIdTivZGwIDAQABo4IBDDCCAQgwDgYDVR0PAQH/
BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwMwYIKwYBBQUHAQEEJzAlMCMGCCsG
AQUFBzABhhdodHRwOi8vb2NzcC5lbnRydXN0Lm5ldDAwBgNVHR8EKTAnMCWgI6Ah
hh9odHRwOi8vY3JsLmVudHJ1c3QubmV0L2cyY2EuY3JsMDsGA1UdIAQ0MDIwMAYE
VR0gADAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTAd
BgNVHQ4EFgQUgqJwdN28Uz/Pe9T3zX+nYMYKTL8wHwYDVR0jBBgwFoAUanImetAe
733nO2lR1GyNn5ASZqswDQYJKoZIhvcNAQELBQADggEBADnVjpiDYcgsY9NwHRkw
y/YJrMxp1cncN0HyMg/vdMNY9ngnCTQIlZIv19+4o/0OgemknNM/TWgrFTEKFcxS
BJPok1DD2bHi4Wi3Ogl08TRYCj93mEC45mj/XeTIRsXsgdfJghhcg85x2Ly/rJkC
k9uUmITSnKa1/ly78EqvIazCP0kkZ9Yujs+szGQVGHLlbHfTUqi53Y2sAEo1GdRv
c6N172tkw+CNgxKhiucOhk3YtCAbvmqljEtoZuMrx1gL+1YQ1JH7HdMxWBCMRON1
exCdtTix9qrKgWRs6PLigVWXUX/hwidQosk8WwBD9lu51aX8/wdQQGcHsFXwt35u
Lcw=
-----END CERTIFICATE-----
 2 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2
-----BEGIN CERTIFICATE-----
MIIEPjCCAyagAwIBAgIESlOMKDANBgkqhkiG9w0BAQsFADCBvjELMAkGA1UEBhMC
VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW50
cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDA5IEVudHJ1c3Qs
IEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEyMDAGA1UEAxMpRW50cnVz
dCBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzIwHhcNMDkwNzA3MTcy
NTU0WhcNMzAxMjA3MTc1NTU0WjCBvjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVu
dHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwt
dGVybXMxOTA3BgNVBAsTMChjKSAyMDA5IEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0
aG9yaXplZCB1c2Ugb25seTEyMDAGA1UEAxMpRW50cnVzdCBSb290IENlcnRpZmlj
YXRpb24gQXV0aG9yaXR5IC0gRzIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC6hLZy254Ma+KZ6TABp3bqMriVQRrJ2mFOWHLP/vaCeb9zYQYKpSfYs1/T
RU4cctZOMvJyig/3gxnQaoCAAEUesMfnmr8SVycco2gvCoe9amsOXmXzHHfV1IWN
cCG0szLni6LVhjkCsbjSR87kyUnEO6fe+1R9V77w6G7CebI6C1XiUJgWMhNcL3hW
wcKUs/Ja5CeanyTXxuzQmyWC48zCxEXFjJd6BmsqEZ+pCm5IO2/b1BEZQvePB7/1
U1+cPvQXLOZprE4yTGJ36rfo5bs0vBmLrpxR57d+tVOxMyLlbc9wPBr64ptntoP0
jaWvYkxN4FisZDQSA/i2jZRjJKRxAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRqciZ60B7vfec7aVHUbI2fkBJmqzAN
BgkqhkiG9w0BAQsFAAOCAQEAeZ8dlsa2eT8ijYfThwMEYGprmi5ZiXMRrEPR9RP/
jTkrwPK9T3CMqS/qF8QLVJ7UG5aYMzyorWKiAHarWWluBh1+xLlEjZivEtRh2woZ
Rkfz6/djwUAFQKXSt/S1mja/qYh2iARVBCuch38aNzx+LaUa2NSJXsq9rD1s2G2v
1fN2D807iDginWyTmsQ9v4IbZT+mD12q/OWyFcq1rca8PdCE6OoGcrBNOTJ4vz4R
nAuknZoh8/CbCzB428Hch0P+vGOaysXCHMnHjf87ElgI5rY97HosTvuDls4MPGmH
VHOkc8KT/1EQrBVUAdj8BbGJoX90g5pJ19xOe4pIb4tF9g==
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Seaside/O=DMDC/CN=dmdc.mil
issuer=/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
---
Acceptable client certificate CA names
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DoD Root CA 2
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DoD Root CA 3
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD CA-31
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD CA-32
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-33
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-34
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-39
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-40
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-41
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-42
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-43
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-44
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID SW CA-35
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID SW CA-36
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID SW CA-37
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID SW CA-38
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD NPE CA-1
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD NPE CA-2
/C=us/O=U.S. Government/OU=FBCA/CN=Common Policy
/DC=gov/DC=va/OU=Services/OU=PKI/CN=Veterans Affairs User CA B1
/C=US/O=Betrusted US Inc/OU=SSP/OU=Betrusted Production SSP CA A1/CN=Betrusted Production SSP CA A1
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-49
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-49
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-50
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-50
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-51
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-51
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD ID CA-52
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-52
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-54
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-53
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DoD Root CA 5
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-55
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-56
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-57
/C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD SW CA-58
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 8265 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 6473361CD857CBA18C7AF7B782DFB26B267254A0607A523A222DBBA8D0EF82F9
    Session-ID-ctx:
    Master-Key: AFF04D2FD80DAF6C04C4F947E884BFFB4DE0F66ABEF92ACF9D31BC72F237B144C8A54B167D526D35994C932EAEB53C76
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1574279398
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

We’re having an issue with the server certificate not being trusted, so let’s get the certificate block using openssl s_client:

[centos@vm-controller certificate-management]$ openssl s_client -servername sso-f5.eupraxialabs.com:443 -connect sso-f5.eupraxialabs.com:443 </dev/null 2>/dev/null | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            34:be:8f:f6:2f:bf:a9:b4:fd:89:ef:3a:23:f0:d4:a8
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=Acme Co, CN=Kubernetes Ingress Controller Fake Certificate
        Validity
            Not Before: Nov 21 17:04:53 2019 GMT
            Not After : Nov 20 17:04:53 2020 GMT
        Subject: O=Acme Co, CN=Kubernetes Ingress Controller Fake Certificate
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ec:15:74:e3:5f:31:cd:e7:d5:b5:94:04:51:fc:
                    80:c8:92:cd:01:5a:21:c7:de:07:f7:88:45:49:24:
                    24:73:21:43:bd:1b:fa:c5:c2:f8:57:57:e4:cd:40:
                    98:22:a6:f2:37:b4:af:12:2a:08:e7:bc:1d:d7:4a:
                    1f:0f:31:f3:38:bb:d6:4f:8f:ac:5c:cf:b1:22:68:
                    36:eb:67:16:62:15:de:65:8c:c3:97:61:14:78:a7:
                    fe:ef:f6:a1:f0:d4:b3:08:c3:0e:06:53:0c:d1:ac:
                    2f:fa:c9:af:f8:ac:f9:ce:fb:75:6f:11:0a:ff:6e:
                    69:cc:75:5e:b2:2b:1c:99:66:cc:d7:c2:dd:33:91:
                    f0:ec:28:f5:9f:f9:6a:4b:d3:5d:4f:d7:41:c2:93:
                    4b:52:f2:46:c2:08:29:2d:77:1b:d5:31:6f:1b:b9:
                    8b:9a:d0:75:f4:64:24:ea:8a:38:64:a2:bb:44:71:
                    55:83:02:39:d2:e2:59:a8:55:82:c6:2a:aa:00:b7:
                    03:06:50:d5:51:d8:34:b4:f1:8e:24:1a:17:18:25:
                    12:6a:52:fd:e7:e0:44:59:95:98:0b:bc:3c:77:f7:
                    a6:24:a6:61:db:2e:c4:81:c7:dc:12:f7:42:89:f3:
                    64:45:a8:a1:c3:ef:e5:4d:f2:7b:cb:df:22:e9:8e:
                    c2:f9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:ingress.local
    Signature Algorithm: sha256WithRSAEncryption
         00:94:ee:b0:7c:fa:ce:30:09:c2:0b:78:8b:a7:ab:14:17:6d:
         72:22:7d:76:24:17:a4:6f:37:bc:b0:6f:c7:37:d5:b4:93:04:
         70:08:c7:12:35:03:78:ae:b2:d6:be:ca:4a:15:7f:52:03:6c:
         50:c8:cc:b1:39:3c:e6:ee:06:c2:40:87:51:34:4f:a7:0b:43:
         af:e6:ff:e1:38:e5:8f:be:d1:67:a9:67:cc:c7:d9:03:59:a6:
         97:b4:7a:73:37:df:e9:11:40:02:88:be:68:9b:64:6f:0c:db:
         cd:6a:4d:e7:b4:a5:2c:a1:23:e5:76:7b:1b:3d:bc:0d:0b:9f:
         07:23:d6:3c:a9:a3:66:39:95:8e:63:59:c4:4d:28:46:e6:b7:
         f4:6a:69:db:5c:97:f1:0e:d1:6f:68:42:b8:24:df:b5:98:0a:
         c0:cd:ac:23:1a:ed:a9:d4:2b:61:59:e5:22:c9:5d:ac:3e:08:
         fd:f7:42:84:2b:65:02:a7:2e:b2:14:6d:1e:da:3c:6a:be:f9:
         01:2a:f2:e1:e8:13:5e:6b:f4:ca:b3:63:8c:fd:3f:f0:84:8b:
         ee:ba:f4:e3:9f:4a:bb:f7:22:41:81:91:74:8c:d8:f1:81:56:
         1f:ca:87:b9:dc:3d:e7:4c:04:d0:71:b6:8c:2b:b5:2d:09:07:
         40:5a:21:da
-----BEGIN CERTIFICATE-----
MIIDbzCCAlegAwIBAgIQNL6P9i+/qbT9ie86I/DUqDANBgkqhkiG9w0BAQsFADBL
MRAwDgYDVQQKEwdBY21lIENvMTcwNQYDVQQDEy5LdWJlcm5ldGVzIEluZ3Jlc3Mg
Q29udHJvbGxlciBGYWtlIENlcnRpZmljYXRlMB4XDTE5MTEyMTE3MDQ1M1oXDTIw
MTEyMDE3MDQ1M1owSzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJu
ZXRlcyBJbmdyZXNzIENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAOwVdONfMc3n1bWUBFH8gMiSzQFaIcfe
B/eIRUkkJHMhQ70b+sXC+FdX5M1AmCKm8je0rxIqCOe8HddKHw8x8zi71k+PrFzP
sSJoNutnFmIV3mWMw5dhFHin/u/2ofDUswjDDgZTDNGsL/rJr/is+c77dW8RCv9u
acx1XrIrHJlmzNfC3TOR8Owo9Z/5akvTXU/XQcKTS1LyRsIIKS13G9Uxbxu5i5rQ
dfRkJOqKOGSiu0RxVYMCOdLiWahVgsYqqgC3AwZQ1VHYNLTxjiQaFxglEmpS/efg
RFmVmAu8PHf3piSmYdsuxIHH3BL3QonzZEWoocPv5U3ye8vfIumOwvkCAwEAAaNP
ME0wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB
/wQCMAAwGAYDVR0RBBEwD4INaW5ncmVzcy5sb2NhbDANBgkqhkiG9w0BAQsFAAOC
AQEAAJTusHz6zjAJwgt4i6erFBdtciJ9diQXpG83vLBvxzfVtJMEcAjHEjUDeK6y
1r7KShV/UgNsUMjMsTk85u4GwkCHUTRPpwtDr+b/4Tjlj77RZ6lnzMfZA1mml7R6
czff6RFAAoi+aJtkbwzbzWpN57SlLKEj5XZ7Gz28DQufByPWPKmjZjmVjmNZxE0o
Rua39Gpp21yX8Q7Rb2hCuCTftZgKwM2sIxrtqdQrYVnlIsldrD4I/fdChCtlAqcu
shRtHto8ar75ASry4egTXmv0yrNjjP0/8ISL7rr0459Ku/ciQYGRdIzY8YFWH8qH
udw950wE0HG2jCu1LQkHQFoh2g==
-----END CERTIFICATE-----

Here is what an interim certificate, issued by cert-manager (Let’s Encrypt) before a valid certificate is issued:

[centos@vm-controller certificate-management]$ curl -kivL -H 'Host: sso-f5.eupraxialabs.com' 'http://40.87.48.212'|more
* About to connect() to 40.87.48.212 port 80 (#0)
*   Trying 40.87.48.212...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 40.87.48.212 (40.87.48.212) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: sso-f5.eupraxialabs.com
>
< HTTP/1.1 308 Permanent Redirect
< Server: openresty/1.15.8.1
< Date: Thu, 21 Nov 2019 19:48:31 GMT
< Content-Type: text/html
< Content-Length: 177
< Connection: keep-alive
< Location: https://sso-f5.eupraxialabs.com/
<
* Ignoring the response-body
{ [data not shown]
100   177  100   177    0     0   1528      0 --:--:-- --:--:-- --:--:--  1539
* Connection #0 to host 40.87.48.212 left intact
* Issue another request to this URL: 'https://sso-f5.eupraxialabs.com/'
* About to connect() to sso-f5.eupraxialabs.com port 443 (#1)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com,O=cert-manager
*       start date: Nov 20 22:13:46 2019 GMT
*       expire date: Feb 18 22:13:46 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=cert-manager.local,O=cert-manager
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
< Server: openresty/1.15.8.1
< Date: Thu, 21 Nov 2019 19:48:32 GMT
< Content-Type: text/html
< Content-Length: 151
< Connection: close
< Location: https://docs.eupraxia.io/docs/error-cert/certificate-error
<
  0   151    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 1
* Issue another request to this URL: 'https://docs.eupraxia.io/docs/error-cert/certificate-error'
* About to connect() to docs.eupraxia.io port 443 (#2)
*   Trying 13.249.79.76...
* Connected to docs.eupraxia.io (13.249.79.76) port 443 (#2)
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.eupraxia.io
*       start date: Jun 30 00:00:00 2019 GMT
*       expire date: Jul 30 12:00:00 2020 GMT
*       common name: *.eupraxia.io
*       issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
> GET /docs/error-cert/certificate-error HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docs.eupraxia.io
> Accept: */*

Here is what it looks like after Let’s Encrypt issues a valid certificate.

Note: This is a production certificate.

[centos@vm-controller certificate-management]$ openssl s_client -servername sso-f5.eupraxialabs.com -connect sso-f5.eupraxialabs.com:443 </dev/null 2>/dev/null | openssl x509 -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:0e:c0:e0:6a:a0:e1:32:1a:28:c3:a0:72:01:f2:dd:e3:e8
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Nov 21 19:50:39 2019 GMT
            Not After : Feb 19 19:50:39 2020 GMT
        Subject: CN=sso-f5.eupraxialabs.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d8:65:2a:60:16:5f:c2:fc:3e:92:5c:6a:1d:35:
                    d1:34:9b:53:a2:7f:17:ea:a8:55:fc:46:06:c5:df:
                    71:04:9a:d8:9f:d6:4e:72:ac:38:46:37:5a:4e:51:
                    fb:04:9d:38:1d:4b:e4:e7:44:35:ee:b4:bf:56:32:
                    bb:79:da:92:1b:48:61:1f:c4:29:67:4e:32:82:27:
                    94:24:4a:2f:fb:65:64:05:36:05:0f:c3:cf:6f:fa:
                    30:5e:14:e6:0e:37:6c:cd:3c:df:45:6b:69:71:37:
                    93:47:1a:4d:ae:7d:b9:7b:5c:38:b1:65:4d:2a:f4:
                    bc:a1:3a:fe:b5:e2:c6:6d:eb:d2:4f:2a:a7:f7:ce:
                    50:31:e7:e6:09:0a:d8:a9:6d:e0:71:fd:33:18:67:
                    a5:12:45:d2:e3:c9:a4:af:62:98:b9:05:58:b1:e1:
                    1d:72:a2:c9:f7:42:87:d8:e4:46:5d:96:11:e2:3c:
                    df:4c:e3:de:72:f4:fb:58:97:45:96:32:6c:58:2e:
                    3d:be:5e:8c:c3:a9:77:f3:6d:c7:a1:c1:1d:c4:a6:
                    bf:c4:22:1d:bd:54:5e:4a:47:94:09:fb:b6:49:52:
                    9c:5e:67:8f:14:19:f7:bb:9d:ee:63:62:70:ba:d4:
                    99:a8:0f:69:a4:dc:54:82:08:78:cf:f2:19:2e:0d:
                    23:73
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                77:FA:DA:28:C0:72:30:29:86:DA:08:1A:B2:FA:BF:F3:F9:6A:E1:E9
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

        Authority Information Access:
            OCSP - URI:http://ocsp.int-x3.letsencrypt.org
            CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

        X509v3 Subject Alternative Name:
            DNS:sso-f5.eupraxialabs.com
        X509v3 Certificate Policies:
            Policy: 2.23.140.1.2.1
            Policy: 1.3.6.1.4.1.44947.1.1.1
              CPS: http://cps.letsencrypt.org

        CT Precertificate SCTs:
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : 5E:A7:73:F9:DF:56:C0:E7:B5:36:48:7D:D0:49:E0:32:
                            7A:91:9A:0C:84:A1:12:12:84:18:75:96:81:71:45:58
                Timestamp : Nov 21 20:50:39.267 2019 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:44:02:20:5A:C5:41:2F:7D:F2:A6:8D:7D:A1:04:BD:
                            04:36:A6:3C:01:4F:63:95:A3:CB:15:54:F2:E2:D4:3D:
                            14:AA:86:37:02:20:1B:19:14:7B:93:77:D7:5D:81:BB:
                            25:64:23:24:B6:27:00:86:C3:E1:BC:86:ED:E0:4A:4F:
                            3E:E2:84:61:C0:98
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : B2:1E:05:CC:8B:A2:CD:8A:20:4E:87:66:F9:2B:B9:8A:
                            25:20:67:6B:DA:FA:70:E7:B2:49:53:2D:EF:8B:90:5E
                Timestamp : Nov 21 20:50:39.253 2019 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:45:02:20:18:20:86:B0:C0:8E:20:65:84:76:53:17:
                            A3:0C:CD:88:1B:47:69:65:02:71:B1:93:29:3E:63:D9:
                            50:56:10:7B:02:21:00:B2:BC:D1:8C:A0:0C:84:95:16:
                            14:69:A8:25:43:99:BE:DF:65:75:89:7D:D5:2B:28:D5:
                            EF:80:B7:55:28:91:4E
Signature Algorithm: sha256WithRSAEncryption
     75:f7:cd:44:a9:2a:07:ec:7d:6a:c6:b0:47:2b:f9:b1:eb:dd:
     b5:1f:7a:e5:c6:d0:a4:e4:05:64:54:54:b1:c0:3c:ab:97:26:
     02:4c:80:ab:04:46:b2:fa:88:f0:e4:bd:08:eb:cd:c8:9d:96:
     92:af:04:6b:59:50:f5:ed:e2:1a:19:05:8b:aa:92:a7:e5:98:
     4e:8a:56:64:c4:36:ff:c3:92:5b:b7:90:b7:b0:15:ef:7a:98:
     f6:6f:b7:99:b9:68:96:1e:5f:65:4a:48:b8:92:d4:67:22:69:
     e9:49:c7:0e:ea:b3:a9:79:b6:fc:df:d9:90:b9:22:86:45:79:
     14:52:99:23:e6:8c:85:01:4b:a8:94:c4:e9:aa:4e:06:e8:13:
     c0:ff:0f:eb:64:71:c4:78:a2:58:28:21:12:47:3b:bb:c0:12:
     b9:ac:57:96:cb:e1:c8:53:21:f3:b5:c1:53:88:b2:92:15:9e:
     06:b9:0b:5c:96:6b:8e:86:47:cc:e0:5f:36:e9:86:6a:ba:13:
     03:78:f5:4e:74:45:e5:bf:1e:d5:7d:be:0e:21:99:5d:c9:58:
     f8:e8:77:20:51:f7:3f:0b:28:ca:8d:fb:80:99:c5:d0:57:19:
     3a:6c:e4:44:66:a2:d2:08:f7:21:db:35:47:32:ca:d0:fe:36:
     17:81:18:fe
-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgISAw7A4Gqg4TIaKMOgcgHy3ePoMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTExMjExOTUwMzlaFw0y
MDAyMTkxOTUwMzlaMCIxIDAeBgNVBAMTF3Nzby1mNS5ldXByYXhpYWxhYnMuY29t
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2GUqYBZfwvw+klxqHTXR
NJtTon8X6qhV/EYGxd9xBJrYn9ZOcqw4RjdaTlH7BJ04HUvk50Q17rS/VjK7edqS
G0hhH8QpZ04ygieUJEov+2VkBTYFD8PPb/owXhTmDjdszTzfRWtpcTeTRxpNrn25
e1w4sWVNKvS8oTr+teLGbevSTyqn985QMefmCQrYqW3gcf0zGGelEkXS48mkr2KY
uQVYseEdcqLJ90KH2ORGXZYR4jzfTOPecvT7WJdFljJsWC49vl6Mw6l3823HocEd
xKa/xCIdvVReSkeUCfu2SVKcXmePFBn3u53uY2JwutSZqA9ppNxUggh4z/IZLg0j
cwIDAQABo4ICazCCAmcwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF
BwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBR3+toowHIwKYba
CBqy+r/z+Wrh6TAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggr
BgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRz
ZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRz
ZW5jcnlwdC5vcmcvMCIGA1UdEQQbMBmCF3Nzby1mNS5ldXByYXhpYWxhYnMuY29t
MEwGA1UdIARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUH
AgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB
9ASB8QDvAHUAXqdz+d9WwOe1Nkh90EngMnqRmgyEoRIShBh1loFxRVgAAAFuj7su
IwAABAMARjBEAiBaxUEvffKmjX2hBL0ENqY8AU9jlaPLFVTy4tQ9FKqGNwIgGxkU
e5N3112BuyVkIyS2JwCGw+G8hu3gSk8+4oRhwJgAdgCyHgXMi6LNiiBOh2b5K7mK
JSBna9r6cOeySVMt74uQXgAAAW6Puy4VAAAEAwBHMEUCIBgghrDAjiBlhHZTF6MM
zYgbR2llAnGxkyk+Y9lQVhB7AiEAsrzRjKAMhJUWFGmoJUOZvt9ldYl91Sso1e+A
t1UokU4wDQYJKoZIhvcNAQELBQADggEBAHX3zUSpKgfsfWrGsEcr+bHr3bUfeuXG
0KTkBWRUVLHAPKuXJgJMgKsERrL6iPDkvQjrzcidlpKvBGtZUPXt4hoZBYuqkqfl
mE6KVmTENv/Dklu3kLewFe96mPZvt5m5aJYeX2VKSLiS1GciaelJxw7qs6l5tvzf
2ZC5IoZFeRRSmSPmjIUBS6iUxOmqTgboE8D/D+tkccR4olgoIRJHO7vAErmsV5bL
4chTIfO1wVOIspIVnga5C1yWa46GR8zgXzbphmq6EwN49U50ReW/HtV9vg4hmV3J
WPjodyBR9z8LKMqN+4CZxdBXGTps5ERmotII9yHbNUcyytD+NheBGP4=
-----END CERTIFICATE-----

We’re going to test from the command line here:

[centos@vm-controller eupraxialabs-ca]$ curl -vvvvvL https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token --insecure --data "grant_type=password&scope=openid profile&username=&password=&client_id=inventory-app" -E ./jeremy-client.crt --key jeremy-client.key|more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to sso-f5.eupraxialabs.com port 443 (#0)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate from file
*       subject: E=jeremy.estrada@eupraxialabs.com,CN=estrada.jeremy.george.1503395688,OU=Cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
*       start date: Nov 23 21:50:36 2019 GMT
*       expire date: Nov 22 21:50:36 2020 GMT
*       common name: estrada.jeremy.george.1503395688
*       issuer: E=admin@eupraxialabs.com,CN=Eupraxia Labs Certificate Authority,OU=cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com
*       start date: Nov 22 14:25:13 2019 GMT
*       expire date: Feb 20 14:25:13 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> POST /auth/realms/logistics-dod/protocol/openid-connect/token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
> Content-Length: 84
> Content-Type: application/x-www-form-urlencoded
>
} [data not shown]
* upload completely sent off: 84 out of 84 bytes
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Sun, 24 Nov 2019 16:05:21 GMT
< Content-Type: application/json
< Content-Length: 3695
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: INGRESSCOOKIE=1574611521.925.119949.969457; Expires=Tue, 26-Nov-19 16:05:20 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
< Cache-Control: no-store
< X-Powered-By: Undertow/1
< Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/logistics-dod/; HttpOnly
< Pragma: no-cache
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{ [data not shown]
100  3779  100  3695  100    84   5419    123 --:--:-- --:--:-- --:--:--  5417
* Connection #0 to host sso-f5.eupraxialabs.com left intact
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiJkNmYxNGY3MC1jZWIyLTQ1OWUtOWE4OS1jNTNmNDB
hZjlmZDYiLCJleHAiOjE1NzQ2MTE4MjAsIm5iZiI6MCwiaWF0IjoxNTc0NjExNTIwLCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjo
iYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjU2ODNlOGE0LTk3NTktNDA3My04MjFiLWI4YTM
5ZjUzOGM5YiIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmF
nZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZ
lcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmN
vbSJ9.a67WLbamqgvEM48d6p0ogvQMA23kvderQ0ek68MEVF9wDglvwsiOvH1aYVs8sFMFAICTyQW8YAt4_YyZ0nIZhK38wRAuxkaiaXGMiyU8z0BEfmvKaM6qkIyt8UnSoeL64gRd1oQeZjbmOqtyEHOVMuiU8gH4pRkLkAQimUQD9kgbz7
XNCHqYp2TOxaTEe02viLthqUaX6dYLVJyaq-I_CQgs-zR8X_0OEHUrM0O2Q2Olw3DjUhtGh1FQjQKgEcqpUN0onZ2HWQLkfJ5o6P0WcZ2GkQEXfWvn4eYBEEygbUsqtLMD8dQnWRqCBOay9yOHbSQDEt7W1Vcciz8ngiyR6g","expires_i
n":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiIyYjliYTg5Yi03NDl
kLTQyMTktYTZmNS04ZjliNGY2NTMyZTYiLCJleHAiOjE1NzQ2MTMzMjAsIm5iZiI6MCwiaWF0IjoxNTc0NjExNTIwLCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQ
iOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J
5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjU2ODNlOGE0LTk3NTktNDA3My04MjFiLWI4YTM5ZjUzOGM5YiIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9
uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1
haWwifQ.v9gFntd48FEilG826PHpK2v5h9bpqeT9yK39EyAF0lE","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwW
TdPTlZUTVF0V2pzIn0.eyJqdGkiOiIwYmQ4NmMyNi00YzI3LTQwYmUtYTY4YS04YWVmYmU3MDgwZDkiLCJleHAiOjE1NzQ2MTE4MjAsIm5iZiI6MCwiaWF0IjoxNTc0NjExNTIwLCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY
29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0a
F90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiNTY4M2U4YTQtOTc1OS00MDczLTgyMWItYjhhMzlmNTM4YzliIiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlc
m5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.GfROri
6KUMUfDOifq5KqkmSUKIiyG4RIAeJ1SJkpV-UbOo9UVtG-NZ0YtxU88f3NJhi5Y7jQ8MUVp26jE0mEl2WmBGQ4p3ReC7Ce7hoEucnEL43i7wHKhte5jnjCLV981eAdTvUIt0Au3u29x5eQ_3SWoZMylj7HIQfsiAtwyU1Ub5dFUxCyx_6e_w
i2187ufVDwWc6u9nX7wUPOGgqtxcGaWZHdz5Uzgvbyr5uyWereZP1iCjVe-ck0rniQEW_RFDhhv2o7q6Gws6aeEnx43_q9I7nSFFnnmdLWdnkcmskwu_70_1LdHZcZpACmPlwGJTULoSKb0LzlKQsrf286dQ","not-before-policy":15
74610415,"session_state":"5683e8a4-9759-4073-821b-b8a39f538c9b","scope":"openid profile email"}

Let’s get some better looking results using jq, a lightweight and flexible command-line JSON processor:

[centos@vm-controller eupraxialabs-ca]$ curl -vvvvvL https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token --data client_secret=2d60fa91-42a9-4dd0-a81c-31e7ad8f47e1 --data "grant_type=password&scope=openid profile&username=&password=&client_id=inventory-app" -E ./jeremy-client.crt --key ./jeremy-client.key|jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to sso-f5.eupraxialabs.com port 443 (#0)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: client certificate from file
*       subject: E=jeremy.estrada@eupraxialabs.com,CN=estrada.jeremy.george.1503395688,OU=Cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
*       start date: Nov 23 21:50:36 2019 GMT
*       expire date: Nov 22 21:50:36 2020 GMT
*       common name: estrada.jeremy.george.1503395688
*       issuer: E=admin@eupraxialabs.com,CN=Eupraxia Labs Certificate Authority,OU=cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com
*       start date: Nov 22 14:25:13 2019 GMT
*       expire date: Feb 20 14:25:13 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> POST /auth/realms/logistics-dod/protocol/openid-connect/token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
> Content-Length: 135
> Content-Type: application/x-www-form-urlencoded
>
} [data not shown]
* upload completely sent off: 135 out of 135 bytes
100   135    0     0  100   135      0     89  0:00:01  0:00:01 --:--:--    89< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Mon, 25 Nov 2019 22:18:05 GMT
< Content-Type: application/json
< Content-Length: 3852
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: INGRESSCOOKIE=1574720285.508.40082.458817; Expires=Wed, 27-Nov-19 22:18:04 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
< Cache-Control: no-store
< X-Powered-By: Undertow/1
< Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/logistics-dod/; HttpOnly
< Pragma: no-cache
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{ [data not shown]
100  3987  100  3852  100   135   2073     72  0:00:01  0:00:01 --:--:--  2073
* Connection #0 to host sso-f5.eupraxialabs.com left intact
{
  "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiI0YWMzYWFiZi05OGViLTQwZWQtOTQ2OC03NzE3MDlhNDdjOTQiLCJleHAiOjE1NzQ3MjA1ODUsIm5iZiI6MCwiaWF0IjoxNTc0NzIwMjg0LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6ImY1MTU3OWI0LTBhODItNGM4NC1hMDRkLTQzYTMxOTRmZDNkOSIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiUk9MRV9VU0VSIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsInVzZXJfbmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwibmFtZSI6IkplcmVteSBFc3RyYWRhIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiZXN0cmFkYS5qZXJlbXkuZ2VvcmdlLjE1MDMzOTU2ODgiLCJnaXZlbl9uYW1lIjoiSmVyZW15IiwiZmFtaWx5X25hbWUiOiJFc3RyYWRhIiwiZW1haWwiOiJqZXJlbXkuZXN0cmFkYUBldXByYXhpYWxhYnMuY29tIn0.Zes4FdZsE1H9gC4P-dYQZF5zlLjXGVHRoxjK569HQaoJ2XMzi-I5ZbZWvaER5ALLdK5y7DSFIwM47qcPuY98Epkfwr8tzEgVX_QEIIS2w5mfCz5-2235DEx_lcXtFgmnedwaK0Orkw2TPH3dq8oIWWFvrUam0n22JyKaj9rDfvPkYoGq1gEaqZ7XQlZv6rL99maXEyyB67AG0VGNo-hk_Cyh38HQaVpMMi9gmVrWsTmhuGdchdyVie8ef2f1XJNdx55yt-9sSPMTAWGux6WPJv5_zwdKq3cZwIu8FTzKVkUmhcp0Q-WoP-2sgSM9YgN0fldteV9ymycxHK0jq9FbQQ",
  "expires_in": 300,
  "refresh_expires_in": 1800,
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiJmMzY0MzY1OC02MjAxLTRhMTUtYjlmOS05NDRkNzkxYjNlMzgiLCJleHAiOjE1NzQ3MjIwODUsIm5iZiI6MCwiaWF0IjoxNTc0NzIwMjg1LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6ImY1MTU3OWI0LTBhODItNGM4NC1hMDRkLTQzYTMxOTRmZDNkOSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1haWwifQ.MO17soEYawdFSefDL8Nr9zMnScZJcNSFovIaxiDjH48",
  "token_type": "bearer",
  "id_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiIzZDQ0YjAyOC0zMzA2LTRjZTEtYmVlZS04OTM1YTI3MWJkNjQiLCJleHAiOjE1NzQ3MjA1ODUsIm5iZiI6MCwiaWF0IjoxNTc0NzIwMjg1LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiZjUxNTc5YjQtMGE4Mi00Yzg0LWEwNGQtNDNhMzE5NGZkM2Q5IiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJ1c2VyX25hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZlcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmNvbSJ9.g9R0qJBeIyBNphKGpg7HVXfbvL-ku39C0aVa-okVCEvn6T8jV_2ERbQaoszirFf6U8piUu29HVY8QjelqPHpxtLf0Yt9fdcIxTX-IVaXlQAEMeEw_rRoPQiN6xKxVRSyp6CO6wfYwOXGTNLQtrIwzB8c-iBWFvlHuXnGfq9SDhSLYYzAEqXfkF5u1rspAWDWY9t0MTg7IXvhnzjOqo2MjY2mZxK73ewrklElnR0VUOp4-XOwPkG9E-1qc_79eJ5ivqiJBZZ7_e7Ni1oc3SCUKeYGNvZ1voaVfZW5jux4jhSsMvNEPitcoeImGApsPg7DDs17LMzNohTrDjD_VsAeWg",
  "not-before-policy": 1574632256,
  "session_state": "f51579b4-0a82-4c84-a04d-43a3194fd3d9",
  "scope": "openid profile email"
}

This is full decoding the JWT:

https://jwt.io/#debugger-io?token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiJkNmYxNGY3MC1jZWIyLTQ1OWUtOWE4OS1jNTNmNDBhZjlmZDYiLCJleHAiOjE1NzQ2MTE4MjAsIm5iZiI6MCwiaWF0IjoxNTc0NjExNTIwLCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjU2ODNlOGE0LTk3NTktNDA3My04MjFiLWI4YTM5ZjUzOGM5YiIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZlcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmNvbSJ9.a67WLbamqgvEM48d6p0ogvQMA23kvderQ0ek68MEVF9wDglvwsiOvH1aYVs8sFMFAICTyQW8YAt4_YyZ0nIZhK38wRAuxkaiaXGMiyU8z0BEfmvKaM6qkIyt8UnSoeL64gRd1oQeZjbmOqtyEHOVMuiU8gH4pRkLkAQimUQD9kgbz7XNCHqYp2TOxaTEe02viLthqUaX6dYLVJyaq-I_CQgs-zR8X_0OEHUrM0O2Q2Olw3DjUhtGh1FQjQKgEcqpUN0onZ2HWQLkfJ5o6P0WcZ2GkQEXfWvn4eYBEEygbUsqtLMD8dQnWRqCBOay9yOHbSQDEt7W1Vcciz8ngiyR6g%22&publicKey=-----BEGIN%20PUBLIC%20KEY-----%0AMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAjK5ej%2B2uyEDuGu6a7fyp%0A0xP69eq5npRrwPqvbpxX9gWXPoAZMyjmgA7G2Wf7ejypDlT%2BSf93OPijfRUZF2Br%0AVQYz04BC21lubn9VMaG6o3pUTUltME2l0pwiLmrVu89YUieFLdMZ162%2F50c4QO2L%0A2vFqiGJRMtoEd97cMFro5bQ8L7WYmXXgopkjCSroQUlUuBq%2B0r0AGaOrCQh3ZPQL%0A1VS%2Bwz0koei6LjbXzKUJ6WeekgqPnYLa4bazD373zCf1fznZYb2UmqJbJkYMUqZW%0AagwMBxi%2Bgj0LCEoIqBoMtsvJfMJnznNjOxOLfDfMWPwWC4p8DWTKeJh7T1QXJboq%0APwIDAQAB%0A-----END%20PUBLIC%20KEY-----%0A

This is a partial of the JWT from the payload:

"jti": "19df0c9d-ad3b-420b-8640-e47cead2f32b",
  "exp": 1574691564,
  "nbf": 0,
  "iat": 1574691264,
  "iss": "https://sso.eupraxialabs.com/auth/realms/logistics-dod",
  "aud": "account",
  "sub": "b09a6104-9f1a-46f0-8ce6-c2aefcc2958a",
  "typ": "Bearer",
  "azp": "inventory-app",
  "auth_time": 0,
  "session_state": "c296a4ed-9b64-4c8f-99d4-5af67729d33b",
  "acr": "1",
  "realm_access": {
    "roles": [
      "offline_access",
      "uma_authorization",
      "user"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid profile email",
  "email_verified": true,
  "name": "Jeremy Estrada",
  "preferred_username": "estrada.jeremy.george.1503395688",
  "given_name": "Jeremy",
  "family_name": "Estrada",
  "email": "jeremy.estrada@eupraxialabs.com"
}

Adding a client secret to the flow:

[centos@vm-controller eupraxialabs-ca]$ curl -vvvvvL https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token --data client_secret=2d60fa91-42a9-4dd0-a81c-31e7ad8f47e1 --data "grant_type=password&scope=openid profile&username=&password=&client_id=inventory-app" -E ./jeremy-client.crt --key ./jeremy-client.key
* About to connect() to sso-f5.eupraxialabs.com port 443 (#0)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: client certificate from file
*       subject: E=jeremy.estrada@eupraxialabs.com,CN=estrada.jeremy.george.1503395688,OU=Cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
*       start date: Nov 23 21:50:36 2019 GMT
*       expire date: Nov 22 21:50:36 2020 GMT
*       common name: estrada.jeremy.george.1503395688
*       issuer: E=admin@eupraxialabs.com,CN=Eupraxia Labs Certificate Authority,OU=cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com
*       start date: Nov 22 14:25:13 2019 GMT
*       expire date: Feb 20 14:25:13 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> POST /auth/realms/logistics-dod/protocol/openid-connect/token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
> Content-Length: 135
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 135 out of 135 bytes
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Mon, 25 Nov 2019 21:43:26 GMT
< Content-Type: application/json
< Content-Length: 3852
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: INGRESSCOOKIE=1574718205.047.39074.645208; Expires=Wed, 27-Nov-19 21:43:24 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
< Cache-Control: no-store
< X-Powered-By: Undertow/1
< Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/logistics-dod/; HttpOnly
< Pragma: no-cache
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiI0Njk3ZDQ1Mi1iYTYxLTRjN2ItYWJlYS00ODI4M2Q2OGJhZGQiLCJleHAiOjE1NzQ3MTg1MDUsIm5iZiI6MCwiaWF0IjoxNTc0NzE4MjA0LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjYxOTdlODhhLTU5OWQtNDc5ZC04NjQ1LTkwODY3NjQ0ODU2MSIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiUk9MRV9VU0VSIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsInVzZXJfbmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwibmFtZSI6IkplcmVteSBFc3RyYWRhIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiZXN0cmFkYS5qZXJlbXkuZ2VvcmdlLjE1MDMzOTU2ODgiLCJnaXZlbl9uYW1lIjoiSmVyZW15IiwiZmFtaWx5X25hbWUiOiJFc3RyYWRhIiwiZW1haWwiOiJqZXJlbXkuZXN0cmFkYUBldXByYXhpYWxhYnMuY29tIn0.IhE2SucN6eZt82xuFh4L8y_0bLFjbsPQAPmZgRsnV4uXO3NFFd-g94I1pWV1wxv5f2EvgSj6mSfWUDsT70_z2X9e6_JJ55Dcj28CBOJX7OikcZbuI9VNY8HcMyY_n1AabYGYOcBszDPiDTb3iQEhmpsh_bjR9IkmbKjURRxcuGpRYXhXLhz2z_EY1TPA_fHSbVUefiz929owCxxh6N_fztixmqx2l__5G2HLYzyFmZXrwWS0XidPAu5b9Z-vx3L0TgFFuVc9SYjCv4Cv3CxHBlBNq3_YAw0DQIPzs2l-Vm4IL0rEi6FgsAOYGIh89wIhvpr8kevxbOYYJGhD_Wms5A","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiI3ODNkNTYwMy0wMjBkLTQ4YzAtYWM2Yy1jZDM5YTU1NGU1ZjgiLCJleHAiOjE1NzQ3MjAwMDUsIm5iZiI6MCwiaWF0IjoxNTc0NzE4MjA1LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjYxOTdlODhhLTU5OWQtNDc5ZC04NjQ1LTkwODY3NjQ0ODU2MSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1haWwifQ.ExGQx4WpMbccqYSDRKXlie7QiAwGeqTyHeZRqk58Fio","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiJkODNiODQ0MS1hMTIwLTRmNWQtYjg1ZC02ZmU3OTY1OGQ5YTIiLCJleHAiOjE1NzQ3MTg1MDUsIm5iZiI6MCwiaWF0IjoxNTc0NzE4MjA1LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiNjE5N2U4OGEtNTk5ZC00NzlkLTg2NDUtOTA4Njc2NDQ4NTYxIiwiYWNyIjoiMS* Connection #0 to host sso-f5.eupraxialabs.com left intact
IsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJ1c2VyX25hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZlcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmNvbSJ9.CzfAjv4A82nlCdIRTVytcqpZwd_aVQgepku8GOFYvy2jNvi2sfoWelZU--BMzD_l2GWZUx-QyLMYurSky549xY19tZinTVO204JYZa98xkbD4dNPuR8DwtH4ZCyjakHaD8YsYgcLX0ZjfO23PEVb1XDHpmaL6Rbx6e7mOGQQG6xpl-hydqlmM3CTwCl18lfqdSRusSjknCubG7Yjd6V_MS0ez87h_LY9qTlot93Cu9lC0dDkRBXBd0YimyjLr02hlBSSWTrMBd_IbkAzC-4KM_yEYNTsUL1yfRS_RMJhUwVcrKMpAgylARWw6vv53qRj2C_RDB406Lxczh0uJNy6EA","not-before-policy":1574632256,"session_state":"6197e88a-599d-479d-8645-908676448561","scope":"openid profile email"}[centos@vm-controller eupraxialabs-ca]$

Advanced Troubleshooting

This will be useful if you are looking at our source code in our XtremeCloud Access Manager (XAM) community, which is the upstream opensource fork of XtremeCloud Single Sign-On (SSO).

From the logs of one of our client applications:

2019-11-25 21:27:31.912 DEBUG 1 --- [nio-8880-exec-5] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?state=7b70ff61-d5c3-448c-b2ed-029dda80121c&session_state=e10a6aa7-0274-4e18-bc34-1b7c7d75cfa8&code=9ae7330b-fc7d-4b9f-95ed-62cf5ffa777d.e10a6aa7-0274-4e18-bc34-1b7c7d75cfa8.d038938f-0a7f-4e3f-989e-5082f40c2bfc", parameters={masked}
2019-11-25 21:27:31.914 DEBUG 1 --- [nio-8880-exec-5] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2019-11-25 21:27:31.922 DEBUG 1 --- [nio-8880-exec-5] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2019-11-25 21:27:31.926 DEBUG 1 --- [nio-8880-exec-5] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 403
2019-11-25 21:31:24.834 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-25 21:31:24.838 DEBUG 1 --- [nio-8880-exec-7] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-25 21:31:24.840 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-25 21:31:24.841 DEBUG 1 --- [nio-8880-exec-7] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-25 21:31:24.841 DEBUG 1 --- [nio-8880-exec-7] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-25 21:31:24.841 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-25 21:31:24.841 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-25 21:31:24.843 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-25 21:31:24.844 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-25 21:31:29.232 ERROR 1 --- [nio-8880-exec-9] o.k.adapters.OAuthRequestAuthenticator   : failed to turn code into token
2019-11-25 21:31:29.232 ERROR 1 --- [nio-8880-exec-9] o.k.adapters.OAuthRequestAuthenticator   : status from server: 302
2019-11-25 21:31:29.236 ERROR 1 --- [nio-8880-exec-9] o.k.adapters.OAuthRequestAuthenticator   :    <html>

Issue with getting token for authorization code (from our source code: adapters/oidc/adapter-core/src/main/java/org/keycloak/adapters/OAuthRequestAuthenticator.java):

    AccessTokenResponse tokenResponse = null;
    strippedOauthParametersRequestUri = stripOauthParametersFromRedirect();

    try {
        // For COOKIE store we don't have httpSessionId and single sign-out won't be available
        String httpSessionId = deployment.getTokenStore() == TokenStore.SESSION ? reqAuthenticator.changeHttpSessionId(true) : null;
        tokenResponse = ServerRequest.invokeAccessCodeToToken(deployment, code, rewrittenRedirectUri(strippedOauthParametersRequestUri), httpSessionId);
    } catch (ServerRequest.HttpFailure failure) {
        log.error("failed to turn code into token");
        log.error("status from server: " + failure.getStatus());
        if (failure.getError() != null) {
            log.error("   " + failure.getError());
        }
        return challenge(403, OIDCAuthenticationError.Reason.CODE_TO_TOKEN_FAILURE, null);

    } catch (IOException e) {
        log.error("failed to turn code into token", e);
        return challenge(403, OIDCAuthenticationError.Reason.CODE_TO_TOKEN_FAILURE, null);
    }

Getting an error in a Warehouse (Inventory) application on the code to token exchange:

Note: This is an excerpt from the log that highlights the issue.

2019-11-26 15:46:34.282 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : checking state cookie for after code
2019-11-26 15:46:34.282 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : ** reseting application state cookie
2019-11-26 15:46:34.582 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : failed to turn code into token
2019-11-26 15:46:34.582 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : status from server: 302
2019-11-26 15:46:34.585 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   :    <html>

Here is the longer version:

2019-11-26 14:06:34.916 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?state=654e5403-edf9-46bb-b6b3-b6fd204bb074&session_state=9523326b-97b0-42a8-ba99-31bab3b540cf&code=8a62694e-e4f6-4c1f-9692-d95f7da94236.9523326b-97b0-42a8-ba99-31bab3b540cf.d038938f-0a7f-4e3f-989e-5082f40c2bfc", parameters={masked}
2019-11-26 14:06:34.928 DEBUG 1 --- [nio-8880-exec-7] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2019-11-26 14:06:34.942 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2019-11-26 14:06:34.953 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 403
2019-11-26 14:11:15.370 DEBUG 1 --- [nio-8880-exec-9] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/
2019-11-26 14:11:15.375 DEBUG 1 --- [nio-8880-exec-9] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /
2019-11-26 14:11:15.379 DEBUG 1 --- [nio-8880-exec-9] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/
2019-11-26 14:11:15.380 DEBUG 1 --- [nio-8880-exec-9] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-26 14:11:15.381 DEBUG 1 --- [nio-8880-exec-9] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-26 14:11:15.383 DEBUG 1 --- [nio-8880-exec-9] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-26 14:11:15.388 DEBUG 1 --- [nio-8880-exec-9] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, image/webp, image/apng, application/signed-exchange;v=b3, application/xml;q=0.9, */*;q=0.8]
2019-11-26 14:11:15.389 DEBUG 1 --- [nio-8880-exec-9] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-26 14:11:15.389 DEBUG 1 --- [nio-8880-exec-9] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-26 14:11:15.390 DEBUG 1 --- [nio-8880-exec-9] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-26 14:11:15.390 DEBUG 1 --- [nio-8880-exec-9] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-26 14:11:15.404 DEBUG 1 --- [nio-8880-exec-9] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-26 14:11:15.404 DEBUG 1 --- [nio-8880-exec-9] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-26 15:46:30.061 DEBUG 1 --- [nio-8880-exec-1] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/
2019-11-26 15:46:30.063 DEBUG 1 --- [nio-8880-exec-1] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /
2019-11-26 15:46:30.064 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/
2019-11-26 15:46:30.064 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-26 15:46:30.064 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-26 15:46:30.065 DEBUG 1 --- [nio-8880-exec-1] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-26 15:46:30.066 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-26 15:46:30.066 DEBUG 1 --- [nio-8880-exec-1] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-26 15:46:30.066 DEBUG 1 --- [nio-8880-exec-1] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-26 15:46:30.066 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-26 15:46:30.066 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-26 15:46:30.102 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-26 15:46:30.105 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-26 15:46:31.509 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-26 15:46:31.513 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-26 15:46:31.513 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-26 15:46:31.514 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-26 15:46:31.514 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-26 15:46:31.514 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : there was no code
2019-11-26 15:46:31.514 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : redirecting to auth server
2019-11-26 15:46:31.514 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : callback uri: https://inventory.eupraxialabs.com/items
2019-11-26 15:46:31.515 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : Sending redirect to login page: https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/auth?response_type=code&client_id=inventory-app&redirect_uri=https%3A%2F%2Finventory.eupraxialabs.com%2Fitems&state=41f6a5fe-0013-4d65-8ac9-4c2e17941770&login=true&scope=openid
2019-11-26 15:46:34.271 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items?state=41f6a5fe-0013-4d65-8ac9-4c2e17941770&session_state=113a1236-4bbb-4268-8078-8ecd16a2e9cd&code=78732248-3760-452d-b833-3437aafff53b.113a1236-4bbb-4268-8078-8ecd16a2e9cd.d038938f-0a7f-4e3f-989e-5082f40c2bfc
2019-11-26 15:46:34.275 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-26 15:46:34.282 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-26 15:46:34.282 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-26 15:46:34.282 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-26 15:46:34.282 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : there was a code, resolving
2019-11-26 15:46:34.282 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : checking state cookie for after code
2019-11-26 15:46:34.282 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : ** reseting application state cookie
2019-11-26 15:46:34.582 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : failed to turn code into token
2019-11-26 15:46:34.582 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : status from server: 302
2019-11-26 15:46:34.585 ERROR 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   :    <html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

2019-11-26 15:46:34.585 DEBUG 1 --- [nio-8880-exec-3] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?state=41f6a5fe-0013-4d65-8ac9-4c2e17941770&session_state=113a1236-4bbb-4268-8078-8ecd16a2e9cd&code=78732248-3760-452d-b833-3437aafff53b.113a1236-4bbb-4268-8078-8ecd16a2e9cd.d038938f-0a7f-4e3f-989e-5082f40c2bfc", parameters={masked}
2019-11-26 15:46:34.586 DEBUG 1 --- [nio-8880-exec-3] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2019-11-26 15:46:34.607 DEBUG 1 --- [nio-8880-exec-3] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2019-11-26 15:46:34.610 DEBUG 1 --- [nio-8880-exec-3] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 403

Proper XtremeCloud SSO Wildfly Adapter Authentication Handshake

    ^[h2016-06-08 10:31:10,836 DEBUG [org.keycloak.adapters.PreAuthActionsHandler] (default task-15) adminRequest http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230
    2016-06-08 10:31:10,837 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) --> authenticate()
    2016-06-08 10:31:10,837 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) try bearer
    2016-06-08 10:31:10,837 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) try oauth
    2016-06-08 10:31:10,837 DEBUG [org.keycloak.adapters.undertow.ServletSessionTokenStore] (default task-15) Account was not in session, returning null
    2016-06-08 10:31:10,837 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) there was no code
    2016-06-08 10:31:10,837 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) redirecting to auth server
    2016-06-08 10:31:10,837 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) callback uri: http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230
    2016-06-08 10:31:10,838 DEBUG [org.keycloak.adapters.AuthenticatedActionsHandler] (default task-15) AuthenticatedActionsValve.invoke http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230
    2016-06-08 10:31:19,833 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) --> authenticate()
    2016-06-08 10:31:19,833 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) try bearer
    2016-06-08 10:31:19,833 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-15) try oauth
    2016-06-08 10:31:19,833 DEBUG [org.keycloak.adapters.undertow.ServletSessionTokenStore] (default task-15) Account was not in session, returning null
    2016-06-08 10:31:19,833 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) there was no code
    2016-06-08 10:31:19,833 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) redirecting to auth server
    2016-06-08 10:31:19,833 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) callback uri: http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230
    2016-06-08 10:31:19,834 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-15) Sending redirect to login page: http://localhost:8480/auth/realms/demo/protocol/openid-connect/auth?response_type=code&client_id=kitchensink&redirect_uri=http%3A%2F%2Flocalhost%3A8280%2Fjboss-kitchensink%2Findex.jsf?dswid%3D-6230&state=11%2Fbf0baf70-9f3b-4ad5-a1a2-0570cdba194f&login=true
    2016-06-08 10:31:19,834 INFO  [org.jboss.as.quickstarts.kitchensink.controller.MemberController] (default task-15) Is Auth? false
    2016-06-08 10:32:38,033 DEBUG [org.keycloak.adapters.PreAuthActionsHandler] (default task-16) adminRequest http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230&state=11%2Fbf0baf70-9f3b-4ad5-a1a2-0570cdba194f&code=bz7HlCkudVSWQfg7-dkGI6Ubk5Cbh9s-DBlqa8QvqIg.838d6c87-bed4-47b5-8b1d-50ba81ffbfe1
    2016-06-08 10:32:38,034 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-16) --> authenticate()
    2016-06-08 10:32:38,034 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-16) try bearer
    2016-06-08 10:32:38,034 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-16) try oauth
    2016-06-08 10:32:38,034 DEBUG [org.keycloak.adapters.undertow.ServletSessionTokenStore] (default task-16) Account was not in session, returning null
    2016-06-08 10:32:38,034 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-16) there was a code, resolving
    2016-06-08 10:32:38,034 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-16) checking state cookie for after code
    2016-06-08 10:32:38,034 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-16) ** reseting application state cookie
    **2016-06-08 10:32:38,068 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-16) Token Verification succeeded! **
    2016-06-08 10:32:38,069 DEBUG [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-16) successful authenticated
    2016-06-08 10:32:38,069 TRACE [org.keycloak.adapters.RefreshableKeycloakSecurityContext] (default task-16) checking whether to refresh.
    2016-06-08 10:32:38,069 TRACE [org.keycloak.adapters.AdapterUtils] (default task-16) use realm role mappings
    2016-06-08 10:32:38,069 TRACE [org.keycloak.adapters.AdapterUtils] (default task-16) Setting roles: 
    2016-06-08 10:32:38,069 TRACE [org.keycloak.adapters.AdapterUtils] (default task-16)    role: default-role
    2016-06-08 10:32:38,069 DEBUG [org.keycloak.adapters.wildfly.WildflyRequestAuthenticator] (default task-16) propagate security context to wildfly
    2016-06-08 10:32:38,070 DEBUG [org.keycloak.adapters.RequestAuthenticator] (default task-16) User '0c14952a-1b87-46e0-a56a-87d163ce1b4a' invoking 'http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230&state=11%2Fbf0baf70-9f3b-4ad5-a1a2-0570cdba194f&code=bz7HlCkudVSWQfg7-dkGI6Ubk5Cbh9s-DBlqa8QvqIg.838d6c87-bed4-47b5-8b1d-50ba81ffbfe1' on client 'kitchensink'
    2016-06-08 10:32:38,071 DEBUG [org.keycloak.adapters.RequestAuthenticator] (default task-16) AUTHENTICATED
    2016-06-08 10:32:38,074 DEBUG [org.keycloak.adapters.PreAuthActionsHandler] (default task-17) adminRequest http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230
    2016-06-08 10:32:38,074 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-17) --> authenticate()
    2016-06-08 10:32:38,074 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-17) try bearer
    2016-06-08 10:32:38,074 TRACE [org.keycloak.adapters.RequestAuthenticator] (default task-17) try oauth
    2016-06-08 10:32:38,074 DEBUG [org.keycloak.adapters.undertow.KeycloakUndertowAccount] (default task-17) session is active
    2016-06-08 10:32:38,075 DEBUG [org.keycloak.adapters.undertow.ServletSessionTokenStore] (default task-17) Cached account found
    2016-06-08 10:32:38,075 DEBUG [org.keycloak.adapters.wildfly.WildflyRequestAuthenticator] (default task-17) propagate security context to wildfly
    2016-06-08 10:32:38,075 DEBUG [org.keycloak.adapters.RequestAuthenticator] (default task-17) AUTHENTICATED: was cached
    2016-06-08 10:32:38,075 DEBUG [org.keycloak.adapters.AuthenticatedActionsHandler] (default task-17) AuthenticatedActionsValve.invoke http://localhost:8280/jboss-kitchensink/index.jsf?dswid=-6230

Let’s look at the OpenID endpoints of XtremeCloud SSO:

[centos@vm-controller eupraxialabs-ca]$ curl https://sso.eupraxialabs.com/auth/realms/logistics-dod/.well-known/openid-configuration|jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2425    0  2425    0     0   4077      0 --:--:-- --:--:-- --:--:--  4075
{
  "issuer": "https://sso.eupraxialabs.com/auth/realms/logistics-dod",
  "authorization_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/auth",
  "token_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token",
  "token_introspection_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token/introspect",
  "userinfo_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/userinfo",
  "end_session_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/logout",
  "jwks_uri": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/certs",
  "check_session_iframe": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/login-status-iframe.html",
  "grant_types_supported": [
    "authorization_code",
    "implicit",
    "refresh_token",
    "password",
    "client_credentials"
  ],
  "response_types_supported": [
    "code",
    "none",
    "id_token",
    "token",
    "id_token token",
    "code id_token",
    "code token",
    "code id_token token"
  ],
  "subject_types_supported": [
    "public",
    "pairwise"
  ],
  "id_token_signing_alg_values_supported": [
    "ES384",
    "RS384",
    "HS256",
    "HS512",
    "ES256",
    "RS256",
    "HS384",
    "ES512",
    "RS512"
  ],
  "userinfo_signing_alg_values_supported": [
    "ES384",
    "RS384",
    "HS256",
    "HS512",
    "ES256",
    "RS256",
    "HS384",
    "ES512",
    "RS512",
    "none"
  ],
  "request_object_signing_alg_values_supported": [
    "ES384",
    "RS384",
    "ES256",
    "RS256",
    "ES512",
    "RS512",
    "none"
  ],
  "response_modes_supported": [
    "query",
    "fragment",
    "form_post"
  ],
  "registration_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/clients-registrations/openid-connect",
  "token_endpoint_auth_methods_supported": [
    "private_key_jwt",
    "client_secret_basic",
    "client_secret_post",
    "client_secret_jwt"
  ],
  "token_endpoint_auth_signing_alg_values_supported": [
    "RS256"
  ],
  "claims_supported": [
    "sub",
    "iss",
    "auth_time",
    "name",
    "given_name",
    "family_name",
    "preferred_username",
    "email"
  ],
  "claim_types_supported": [
    "normal"
  ],
  "claims_parameter_supported": false,
  "scopes_supported": [
    "openid",
    "address",
    "email",
    "offline_access",
    "phone",
    "profile",
    "roles",
    "web-origins"
  ],
  "request_parameter_supported": true,
  "request_uri_parameter_supported": true,
  "code_challenge_methods_supported": [
    "plain",
    "S256"
  ],
  "tls_client_certificate_bound_access_tokens": true,
  "introspection_endpoint": "https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token/introspect"
}

Cookies used by XtremeCloud SSO:

  • KEYCLOAK_IDENTITY contains a token (JWT) with the user ids. You can view its content using jwt.io (for example). This cookie lives with your browser session and can also be refreshed with SSO. (for example, if you change some of your personal data in the “Manage my account”)

  • KEYCLOAK_SESSION your session id associated to the concerned realm.

  • Oauth_token_request_state is part of the Oauth spec in order to avoid hacking of the redirect link after login

If service accounts are not enabled for the client, this is the error that you will see:

status: 401
statusText: Unauthorized
readyState: 4
responseText: {"error":"unauthorized_client","error_description":"Client not enabled to retrieve service account"}
OAuth2 Response Error Details:
error: unauthorized_client
error_description: Client not enabled to retrieve service account
WWW-Authenticate: Bearer realm="logistics-dod", error="invalid_token", error_description="Invalid token issuer. Expected 'https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod', but was 'https://sso.eupraxialabs.com/auth/realms/logistics-dod'"

Set the hostname field on the XtremeCloud SSO “logistics-dod” realm to be sso-f5.eupraxialabs.com.


[centos@vm-controller eupraxialabs-ca]$ curl -vvvvvL https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/token --data client_secret=a24491a1-8666-49b4-826a-74598a62dd91 --data "grant_type=password&scope=openid profile&username=&password=&client_id=inventory-app" -E ./jeremy-client.crt --key ./jeremy-client.key                   * About to connect() to sso-f5.eupraxialabs.com port 443 (#0)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: client certificate from file
*       subject: E=jeremy.estrada@eupraxialabs.com,CN=estrada.jeremy.george.1503395688,OU=Cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
*       start date: Nov 23 21:50:36 2019 GMT
*       expire date: Nov 22 21:50:36 2020 GMT
*       common name: estrada.jeremy.george.1503395688
*       issuer: E=admin@eupraxialabs.com,CN=Eupraxia Labs Certificate Authority,OU=cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com
*       start date: Nov 22 14:25:13 2019 GMT
*       expire date: Feb 20 14:25:13 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> POST /auth/realms/logistics-dod/protocol/openid-connect/token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
> Content-Length: 135
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 135 out of 135 bytes
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Thu, 28 Nov 2019 17:38:24 GMT
< Content-Type: application/json
< Content-Length: 3826
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: INGRESSCOOKIE=1574962705.692.2748.182550; Expires=Sat, 30-Nov-19 17:38:24 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
< Cache-Control: no-store
< X-Powered-By: Undertow/1
< Set-Cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/logistics-dod/; HttpOnly
< Pragma: no-cache
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiI5ZWMwNzM4Yi02OTVhLTQyNzMtYjY4NC0xZmY2ZTEyNDNkYzkiLCJleHAiOjE1NzQ5NjYzMDQsIm5iZiI6MCwiaWF0IjoxNTc0OTYyNzA0LCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6Ijk0YWUzMTNjLWRjYTEtNGU3OS1iNjI5LTY5MjFkMzgyNTUxZCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9pbnZlbnRvcnkuZXVwcmF4aWFsYWJzLmNvbSIsIioiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIlJPTEVfVVNFUiIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.J8GtzXbH0JmPCturu51mQDIx4N7xf9sdZsfgHE5ZNF1vZe2lBbO8mIRXjR3AIn39fY-K4YASV_WiTKf5reBYu1xRr19YENpPnU3rJNCHmcsr1mmV95zrer6SSp189JORxEsaFmLaOzmiHmZ6dH8la-2BUX_1dos82pyyob7il7NNRgBAQWLcd-UvYBLLTjrHPpjAPwULKz3qO7_wytpCWd969yWQrC34EuMkwElCafKMBGEkL0fA-8NeSrCItD29Mm-tcqzSdlGUKkRIsTYWWcsmMVLKQ04FINSvqrkQaAKvXfcdCTRtDgDZIJPsTCjyFRDuNGDN8p1MyhVOg2kJYQ","expires_in":3600,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiIwMzA5ZDQ0Yi03OWY3LTRlMDgtOWQxNS05NTQwZWIyZjExMzkiLCJleHAiOjE1NzQ5NjQ1MDQsIm5iZiI6MCwiaWF0IjoxNTc0OTYyNzA0LCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6Ijk0YWUzMTNjLWRjYTEtNGU3OS1iNjI5LTY5MjFkMzgyNTUxZCIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1haWwifQ.ZVX7HYpDsQSOP1GeJ_05P0flvSebyS1zG06FQpYbLl8","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiJhZjIwOTM4OS1hMTExLTQ5YjAtYjkzNi1iNjlmMjZmMzA0OTUiLCJleHAiOjE1NzQ5NjYzMDQsIm5iZiI6MCwiaWF0IjoxNTc0OTYyNzA0LCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiOTRhZTMxM2MtZGNhMS00ZTc5LW* Connection #0 to host sso-f5.eupraxialabs.com left intact
I2MjktNjkyMWQzODI1NTFkIiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.aEKk606jSVChzzGTayg_311Mu8gRVpyzPM5AWY9AcmlMf9_q0lrSpDhvP3M2249T77h3W6G4abEbr3EGSpR2lE1amfMBjo0lLUeMsWArSMLJidrfBBDxITX7DaI5hHMhfzv772jIyQdhLaErEz44D8-FEhbYV-z8_e3z3OeXqhOK4qeTUQ7LjKs2t3cTUD1iOV05UwHSxqD5b2k3XSdG46FYbyRsuihGcbm_nbjuSO2G-TxURgudkMedHzIf2Pu0F63htv0VDjcZ1CZSCAcTemdxhWoYFIDetEyQ41hcht1BPRzUEG3IybiBtyQJbAW3mN2M8wZ-lebT-Ev9HIo2zA","not-before-policy":1574953210,"session_state":"94ae313c-dca1-4e79-b629-6921d382551d","scope":"openid profile email"}[centos@vm-controller eupraxialabs-ca]$
[centos@vm-controller eupraxialabs-ca]$ curl -i -X GET -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiI5ZWMwNzM4Yi02OTVhLTQyNzMtYjY4NC0xZmY2ZTEyNDNkYzkiLCJleHAiOjE1NzQ5NjYzMDQsIm5iZiI6MCwiaWF0IjoxNTc0OTYyNzA0LCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6Ijk0YWUzMTNjLWRjYTEtNGU3OS1iNjI5LTY5MjFkMzgyNTUxZCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9pbnZlbnRvcnkuZXVwcmF4aWFsYWJzLmNvbSIsIioiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIlJPTEVfVVNFUiIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.J8GtzXbH0JmPCturu51mQDIx4N7xf9sdZsfgHE5ZNF1vZe2lBbO8mIRXjR3AIn39fY-K4YASV_WiTKf5reBYu1xRr19YENpPnU3rJNCHmcsr1mmV95zrer6SSp189JORxEsaFmLaOzmiHmZ6dH8la-2BUX_1dos82pyyob7il7NNRgBAQWLcd-UvYBLLTjrHPpjAPwULKz3qO7_wytpCWd969yWQrC34EuMkwElCafKMBGEkL0fA-8NeSrCItD29Mm-tcqzSdlGUKkRIsTYWWcsmMVLKQ04FINSvqrkQaAKvXfcdCTRtDgDZIJPsTCjyFRDuNGDN8p1MyhVOg2kJYQ" https://inventory.eupraxialabs.com/items
HTTP/1.1 200
Server: openresty/1.15.8.1
Date: Thu, 28 Nov 2019 17:39:49 GMT
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: nginx-route=1574962789.427.2748.811408; Expires=Sat, 30-Nov-19 17:39:48 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Language: en-US
Strict-Transport-Security: max-age=15724800; includeSubDomains


<html>
<h1>Confidential Warehouse Items</h1>
<ul>
        <li>AIM-9 Sidewinder Missile</li>
        <li>AGM-123 Skipper Missile</li>
        <li>Towed Dummy Target</li>
        <li>Air-Towed Dummy Target</li>
</ul>
<p> <a href="/logout">Logout</a> </p>

As we can see, the results (protected resource) are returned as expected.

The log trace of the protected application, a Kubernetes pod, is even more definitive:

[centos@vm-controller certificate-management]$ kubectl logs -f inventory-deployment-6ddff68f5f-2p5rf


2019-11-28 18:04:30.820 DEBUG 1 --- [nio-8880-exec-1] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-28 18:04:30.820 TRACE 1 --- [nio-8880-exec-1] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-28 18:04:30.820 TRACE 1 --- [nio-8880-exec-1] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-28 18:04:30.820 DEBUG 1 --- [nio-8880-exec-1] o.k.a.BearerTokenRequestAuthenticator    : Found [1] values in authorization header, selecting the first value for Bearer.
2019-11-28 18:04:30.820 DEBUG 1 --- [nio-8880-exec-1] o.k.a.BearerTokenRequestAuthenticator    : Verifying access_token
2019-11-28 18:04:30.826 TRACE 1 --- [nio-8880-exec-1] o.k.a.BearerTokenRequestAuthenticator    :        access_token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJuZG9XUUJwSDZkZE94dHh0djhPLTYybV9vaUg3c1pwWTdPTlZUTVF0V2pzIn0.eyJqdGkiOiI5ZWMwNzM4Yi02OTVhLTQyNzMtYjY4NC0xZmY2ZTEyNDNkYzkiLCJleHAiOjE1NzQ5NjYzMDQsIm5iZiI6MCwiaWF0IjoxNTc0OTYyNzA0LCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6Ijk0YWUzMTNjLWRjYTEtNGU3OS1iNjI5LTY5MjFkMzgyNTUxZCIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9pbnZlbnRvcnkuZXVwcmF4aWFsYWJzLmNvbSIsIioiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIlJPTEVfVVNFUiIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJvcGVuaWQgcHJvZmlsZSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.signature
2019-11-28 18:04:30.827 DEBUG 1 --- [nio-8880-exec-1] o.k.a.BearerTokenRequestAuthenticator    : successful authorized
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] o.k.a.RefreshableKeycloakSecurityContext : checking whether to refresh.
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       : use realm role mappings
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       : Setting roles:
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       :    role: ROLE_USER
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       :    role: offline_access
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       :    role: uma_authorization
2019-11-28 18:04:30.827 TRACE 1 --- [nio-8880-exec-1] org.keycloak.adapters.AdapterUtils       :    role: user
2019-11-28 18:04:30.827 DEBUG 1 --- [nio-8880-exec-1] o.k.adapters.RequestAuthenticator        : User 'b09a6104-9f1a-46f0-8ce6-c2aefcc2958a' invoking 'https://inventory.eupraxialabs.com/items' on client 'inventory-app'
2019-11-28 18:04:30.827 DEBUG 1 --- [nio-8880-exec-1] o.k.adapters.RequestAuthenticator        : Bearer AUTHENTICATED
2019-11-28 18:04:30.827 DEBUG 1 --- [nio-8880-exec-1] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /items
2019-11-28 18:04:30.828 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/items
2019-11-28 18:04:30.828 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-28 18:04:30.828 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : GET "/items", parameters={}
2019-11-28 18:04:30.828 DEBUG 1 --- [nio-8880-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.eupraxialabs.inventoryapp.InventoryController#getItems(Model)
2019-11-28 18:04:30.845 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.v.ContentNegotiatingViewResolver : Selected '*/*' given [*/*]
2019-11-28 18:04:30.848 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.view.freemarker.FreeMarkerView   : View name 'items', model {items=[AIM-9 Sidewinder Missile, AGM-123 Skipper Missile, Towed Dummy Target, Air-Towed Dummy Target]}
2019-11-28 18:04:30.848 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.view.freemarker.FreeMarkerView   : Rendering [items.ftl]
2019-11-28 18:04:30.857 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK

Confirming that the certificates are being properly served by NGINX:

Note: Ensure that the -servername (SNI) directive is included.

[centos@vm-controller ssl]$ openssl s_client -showcerts -servername sso-f5.eupraxialabs.com -connect sso-f5.eupraxialabs.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = sso-f5.eupraxialabs.com
verify return:1
---
Certificate chain
 0 s:/CN=sso-f5.eupraxialabs.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFZzCCBE+gAwIBAgISA4OP5PnmqSRd3S+BXHFmJA+VMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTExMjIxNDI1MTNaFw0y
MDAyMjAxNDI1MTNaMCIxIDAeBgNVBAMTF3Nzby1mNS5ldXByYXhpYWxhYnMuY29t
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0RTXqFRb33FADKXbMYni
M3tV1gZYBg4jaXGMNa3H+aaWnEssSIppDacaz5ff1HlIwC03MFUKuRrjF2YH+JKv
CN+LGRGNOILFXMH6ecdpnukoS5cV7u3Jbp76Xz4eMsPPJFp++1GxIXQubdzAhiXe
ic1MXLCwdTms/UmTrkYgA1U3qudVnDO6G5GAziI6mbFs3wPNtF++YuveiZ2pqdib
r34Jr3e9Mu50mxhOKo5Dxc5DVl8ZIegBxyBs7CrJVc2wrArdsvoroFoxx3nmREz0
sTBeVsE6uAtXaKkHjcrws795kFeoPwAFcOz8viRVBYUZrTMXvdm6KEK+kGxV2Qv3
hwIDAQABo4ICbTCCAmkwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF
BwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTLZr7Cgjv++ZIK
5oBTd90USEW2iDAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggr
BgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRz
ZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRz
ZW5jcnlwdC5vcmcvMCIGA1UdEQQbMBmCF3Nzby1mNS5ldXByYXhpYWxhYnMuY29t
MEwGA1UdIARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUH
AgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB
9gSB8wDxAHYA8JWkWfIA0YJAEC0vk4iOrUv+HUfjmeHQNKawqKqOsnMAAAFuk7ed
PQAABAMARzBFAiEA3I5F1fGBlGsIlxRF3MPVXsIfXOVE2SpBHGUBPQv7EkkCIHun
WD3eeawf2bmlnw48uhlD/rE0mouW25DZV6ADsP6aAHcAsh4FzIuizYogTodm+Su5
iiUgZ2va+nDnsklTLe+LkF4AAAFuk7edLgAABAMASDBGAiEAuBMA71QdkZ6ognOI
yU7UWU9tvzdMQ8wiWWRMuwuZKmQCIQDNU0EjU69cQScQ7Lmh7rZtaGV2zS5qSxh7
1T9xc1gEEjANBgkqhkiG9w0BAQsFAAOCAQEABOlR0rlL8Rr+CgcbcTJpJ7gFPq5j
RbYxbXEtY+6Yya+jhgWkkjs2TwFAHn8hGXR7vxmnL7U2EUJz4pKbt0zFy3NrRDgJ
gdLxF4jVrhY31A3R7yW0XrjfjzQw7bTUSE6RpZXn9im12gIfmFi9c8F2RNaOblvF
NK92kcQZSUZmHKtjg0zR88HHKFbBZWY8MEW2ZHAECfoHgsg1FVAFAOatIooealqm
usuw1QdwJpFRy5+jYu508Hi11K4mw69+9Ss2jwjK1FtC+yug2sx0E/Yww8CmNNjk
oXY+/E7fFjXERWKPyuBsNjnUf+Tlol78hVQLEAU925Ui9/I0oCH/NYyQPg==
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=sso-f5.eupraxialabs.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
Acceptable client certificate CA names
/C=US/ST=Texas/L=Austin/O=Eupraxia Labs/OU=cloud/CN=Eupraxia Labs Certificate Authority/emailAddress=admin@eupraxialabs.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x07+0x08:0x08+0x08:0x09+0x08:0x0A+0x08:0x0B+0x08:0x04+0x08:0x05+0x08:0x06+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3503 bytes and written 459 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0B910BB898E177C34C5D09B6A2F215AF1E2AFAE460A1E7D460674FF43B71FBFC
    Session-ID-ctx:
    Master-Key: C3A4DEB5126A9B2751613B0DDF534C33C3E001D3DE7EDC2C9810AC69BA2EDBCF08278B86458D8B4BB3C42B78F5D8178F
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 3d 12 bb 20 dc 00 d6 78-01 5e b4 fd 61 39 d2 fd   =.. ...x.^..a9..
    0010 - 3e 68 fe a8 ce 8b f6 28-d5 43 16 4e 7f a2 8b 02   >h.....(.C.N....
    0020 - 2c 2b 88 77 51 55 3a 38-2a 08 85 93 3e 6d 1a b0   ,+.wQU:8*...>m..
    0030 - 8a 82 e3 1f 57 b3 7f 3a-b2 a0 a3 8d ea 7a 92 2c   ....W..:.....z.,
    0040 - 2e fb d0 e2 39 16 6b 7a-47 c7 1a 49 00 6d a6 4e   ....9.kzG..I.m.N
    0050 - 26 31 a7 1e 9c 2f 34 b2-95 1a 26 99 fa c5 85 08   &1.../4...&.....
    0060 - 0c 17 b9 c7 f8 75 04 93-2c ed d8 57 9f 7d be c7   .....u..,..W.}..
    0070 - f8 ac f1 e6 1e 08 a9 85-97 66 e0 65 4f a4 99 46   .........f.eO..F
    0080 - 0a 9b c8 b0 7a 85 7c db-df e8 17 81 d1 12 89 0f   ....z.|.........
    0090 - fc 7f e8 14 84 17 51 00-21 06 c9 24 3a 5d 49 de   ......Q.!..$:]I.
    00a0 - 7c b6 be 0b 22 0c b6 24-76 16 5b ce 14 78 8d ce   |..."..$v.[..x..
    00b0 - 09 74 c3 a1 0c 2f 54 1d-b5 fe 64 5e 96 68 c4 25   .t.../T...d^.h.%
    00c0 - 72 2d 3c b5 ba 9b 83 3c-03 5d 11 59 49 32 d9 06   r-<....<.].YI2..

    Start Time: 1575042806
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
^C
[centos@vm-controller ssl]$ openssl s_client -showcerts -servername inventory.eupraxialabs.com -connect inventory.eupraxialabs.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = inventory.eupraxialabs.com
verify return:1
---
Certificate chain
 0 s:/CN=inventory.eupraxialabs.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFazCCBFOgAwIBAgISBGptMdhDVBCPAPNg07TRmCweMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTExMjIxNDM0MzJaFw0y
MDAyMjAxNDM0MzJaMCUxIzAhBgNVBAMTGmludmVudG9yeS5ldXByYXhpYWxhYnMu
Y29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnR52F/43Rhq4cN8n
gTkGswMlPvkcJ1fVzYHabQntQbO2oGfGfYbPuDT4PPj5bAsWvmIhndTHeAd955Zj
jcBoHp9IE/tIoVCqlIUCuA2lKvyADopfcIcsWd9Axse94cRUY5MZnz0wdiOhq9mB
C5cr6DcNYwH5pscrL8NdE2OBtj8cxOYtw9oQ1wNjxPLcSYYqKjxZZEZyXP9dcd9n
JCgpHRHglu0ZZOjKdFcsB/s4FpGXWdtqdCiDuoQZ3RnfnsftNOC468FXU3xllzY6
YuJ6oYKslYVWIkUkX5gg7w51BoAClCnZXNhfjPeKhers+9nkEAolRCQvJanaqvnK
Nx/IOwIDAQABo4ICbjCCAmowDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRSAs59Wnc6
c1QA/hznBgEfulYiiTAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBv
BggrBgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5s
ZXRzZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5s
ZXRzZW5jcnlwdC5vcmcvMCUGA1UdEQQeMByCGmludmVudG9yeS5ldXByYXhpYWxh
YnMuY29tMEwGA1UdIARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYI
KwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHW
eQIEAgSB9ASB8QDvAHUAsh4FzIuizYogTodm+Su5iiUgZ2va+nDnsklTLe+LkF4A
AAFuk8AgrgAABAMARjBEAiA7Jj4YKv67PX+KHGoiA+cEOeWYdHUx201M2t+m7vP+
PgIgJ9w8TqAIeUcR7j12fsb4Vh8Zvohv5SXk0VC2GRJ2djcAdgBvU3asMfAxGdiZ
AKRRFf93FRwR2QLBACkGjbIImjfZEwAAAW6TwCDiAAAEAwBHMEUCIQDGLk9Ny7bh
PRaNlgnsN4SoVt8sfYe601IRaGr1J/dyAAIgXMcvZ1n2zJfoKFfR4c1pHbqEI/EY
jhkXDAO6QYo8pkswDQYJKoZIhvcNAQELBQADggEBAAzNvcbcJ5V7F7hZazXVD9tV
bCMhNmruhHgSi/IXqqjePkYgR3rwaejFtOo/ozkbWcByHsxD4DkDl8Z9oZEGl18t
Y7ZXTqy0QL9/edfbTsUSIzvjK2XsxwHZpvzyNWJ6A4r3xdqvO0ydDW4tEHO4y5nu
MAdtge2xtEVYqswCBDGoGMybV2VH68zH4shO/nknquubIZa0bxb4SSOoFW0Y7XBy
qLf/dJd3PYvJLX6sgmOtSkUfqPwLgUGj3cHWg8FXBCf53hvD34Bc8LpnhWwhVndL
FaUVVhGjjFqT0rmEZW/zBMwdcc5wW9SrSfvIQDujGJYI/VLsCAXKTQTAe4goClM=
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=inventory.eupraxialabs.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3274 bytes and written 450 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0302FCE794CAC8863FA99858E4183733045C8B210F07C31D102DBF5BD0A61611
    Session-ID-ctx:
    Master-Key: 7BDB477F3134CEA860535101B872B5FAE332A3E756BFCAEBDAEF20F77BF106DBFE057C86AA1A67E6A10A796D74F4E2CE
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 3d 12 bb 20 dc 00 d6 78-01 5e b4 fd 61 39 d2 fd   =.. ...x.^..a9..
    0010 - e9 7a f7 f2 be 61 a4 4a-e9 01 7e 14 3d 33 33 3a   .z...a.J..~.=33:
    0020 - 1c 8f b8 d3 5f de c2 b1-3a f0 c9 3b 75 b5 05 0e   ...._...:..;u...
    0030 - ad 2d 1c 77 16 22 39 52-0c 2d 89 34 97 b6 96 d7   .-.w."9R.-.4....
    0040 - 82 42 d6 54 8e bf a0 b8-42 4a 1f 5e 7c af b9 ad   .B.T....BJ.^|...
    0050 - 16 b8 75 30 a9 27 01 e9-8c f6 bc 88 be 2b fe 0e   ..u0.'.......+..
    0060 - 59 11 da f6 72 78 e6 e8-a1 06 da f3 89 31 cf 5b   Y...rx.......1.[
    0070 - 49 3d 52 aa e4 17 a9 6a-c5 88 80 a1 2c ae a3 be   I=R....j....,...
    0080 - 22 34 86 b5 7e 23 1e 08-48 f4 dd a7 48 5f c2 e7   "4..~#..H...H_..
    0090 - ce 54 a5 10 b9 1c 70 04-6b 40 2c 52 33 0b b2 51   .T....p.k@,R3..Q
    00a0 - f7 95 ea 33 ed dc 96 53-a7 50 d9 95 2a 61 f2 cb   ...3...S.P..*a..
    00b0 - 14 40 9a 3b 90 08 c0 eb-0d 1e 97 0f 09 74 44 88   .@.;.........tD.
    00c0 - 10 7b aa 61 ab f0 76 5d-a6 8a 21 29 67 42 12 e2   .{.a..v]..!)gB..

    Start Time: 1575042850
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed
[centos@vm-controller ssl]$

They are confirmed to be installed and valid. These are production Let’s Encrypt certificates.

With auth-tls-verify-client set to optional in the annotation nginx.ingress.kubernetes.io/auth-tls-verify-client: “optional”, the end user will get prompted for his/her digital certificate. The call from the oAuth2 application (or Relying Party) for an access token to the OpenID Provider (OP) (XtremeCloud SSO) will succeed, since this there are no certificates available to present to XtremeCloud SSO.

Note: If the user does not present a client certificate for authentication, a login form will be presented. Depending on an organization’s security policy, they may or may not create a password for their end users and allow them to access a non-secure version of the application using a password. Typically, only users with valid digital certificates will be able to authenticate.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/load-balance: ip_hash  # ip_hash provides stickiness to the same XtremeCloud SSO pod
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
    # Enable client certificate authentication
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional"
    # Create the secret containing the trusted ca certificates
    nginx.ingress.kubernetes.io/auth-tls-secret: "dev/ca-secret"
    # Specify the verification depth in the client certificates chain
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "2"
    # Specify an error page to be redirected to verification errors
    nginx.ingress.kubernetes.io/auth-tls-error-page: "https://docs.eupraxia.io/docs/error-cert/certificate-error"
    # Specify if certificates are passed to upstream server
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  name: 
spec:
  rules:
    - host: 
      http:
        paths:
          - backend:
              serviceName: 
              servicePort: 
            path: /
  tls:
    - hosts:
        - 
      secretName: sso-f5-eupraxialabs-com

2019-11-29 18:34:13.630 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/
2019-11-29 18:34:13.633 DEBUG 1 --- [nio-8880-exec-2] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /
2019-11-29 18:34:13.633 DEBUG 1 --- [nio-8880-exec-2] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/
2019-11-29 18:34:13.634 DEBUG 1 --- [nio-8880-exec-2] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-29 18:34:13.634 DEBUG 1 --- [nio-8880-exec-2] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-29 18:34:13.649 DEBUG 1 --- [nio-8880-exec-2] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-29 18:34:13.651 DEBUG 1 --- [nio-8880-exec-2] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-29 18:34:13.651 DEBUG 1 --- [nio-8880-exec-2] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-29 18:34:13.651 DEBUG 1 --- [nio-8880-exec-2] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-29 18:34:13.652 DEBUG 1 --- [nio-8880-exec-2] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-29 18:34:13.652 DEBUG 1 --- [nio-8880-exec-2] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-29 18:34:13.690 DEBUG 1 --- [nio-8880-exec-2] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-29 18:34:13.693 DEBUG 1 --- [nio-8880-exec-2] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-29 18:34:15.316 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-29 18:34:15.322 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-29 18:34:15.322 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-29 18:34:15.322 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-29 18:34:15.322 TRACE 1 --- [nio-8880-exec-3] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-29 18:34:15.322 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : there was no code
2019-11-29 18:34:15.322 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : redirecting to auth server
2019-11-29 18:34:15.323 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : callback uri: https://inventory.eupraxialabs.com/items
2019-11-29 18:34:15.323 DEBUG 1 --- [nio-8880-exec-3] o.k.adapters.OAuthRequestAuthenticator   : Sending redirect to login page: https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/auth?response_type=code&client_id=inventory-app&redirect_uri=https%3A%2F%2Finventory.eupraxialabs.com%2Fitems&state=fe28c617-c87f-47e2-8941-ed801999efd0&login=true&scope=openid
2019-11-29 18:34:18.151 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items?state=fe28c617-c87f-47e2-8941-ed801999efd0&session_state=9077ce30-8df6-405a-9b81-284f79f8538e&code=e4554d93-c232-48ec-bebc-d99c0dac999c.9077ce30-8df6-405a-9b81-284f79f8538e.d038938f-0a7f-4e3f-989e-5082f40c2bfc
2019-11-29 18:34:18.157 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-29 18:34:18.158 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-29 18:34:18.160 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-29 18:34:18.160 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-29 18:34:18.160 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   : there was a code, resolving
2019-11-29 18:34:18.160 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   : checking state cookie for after code
2019-11-29 18:34:18.160 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   : ** reseting application state cookie
2019-11-29 18:34:18.482 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   : Verifying tokens
2019-11-29 18:34:18.486 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   :                access_token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJSN2EwV3VsTHFnX1RDckpORGdqSTNPZ0JRYVFGY2duZUxrcklIWFR2X0VrIn0.eyJqdGkiOiJmY2E1NTg1NS1kNTA3LTQzYTMtODJmMi05NDlhOTEyN2ZmMmIiLCJleHAiOjE1NzUwNTYwNTgsIm5iZiI6MCwiaWF0IjoxNTc1MDUyNDU4LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MTU3NTA1MjQ1Nywic2Vzc2lvbl9zdGF0ZSI6IjkwNzdjZTMwLThkZjYtNDA1YS05YjgxLTI4NGY3OWY4NTM4ZSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiUk9MRV9VU0VSIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZlcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmNvbSJ9.signature
2019-11-29 18:34:18.486 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   :                id_token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJSN2EwV3VsTHFnX1RDckpORGdqSTNPZ0JRYVFGY2duZUxrcklIWFR2X0VrIn0.eyJqdGkiOiJjMWQ5ODUxNi1mZjBjLTRhNzEtYWUyZi02NTY4YzU0NDk3YWEiLCJleHAiOjE1NzUwNTYwNTgsIm5iZiI6MCwiaWF0IjoxNTc1MDUyNDU4LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0aF90aW1lIjoxNTc1MDUyNDU3LCJzZXNzaW9uX3N0YXRlIjoiOTA3N2NlMzAtOGRmNi00MDVhLTliODEtMjg0Zjc5Zjg1MzhlIiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSmVyZW15IEVzdHJhZGEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsImdpdmVuX25hbWUiOiJKZXJlbXkiLCJmYW1pbHlfbmFtZSI6IkVzdHJhZGEiLCJlbWFpbCI6ImplcmVteS5lc3RyYWRhQGV1cHJheGlhbGFicy5jb20ifQ.signature
2019-11-29 18:34:18.492 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   :                refresh_token: eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiI1N2MwMmY0Mi0zZWU0LTQ1YjItOWFiNy0yZDE4ZTRkMjQ5MmEiLCJleHAiOjE1NzUwNTQyNTgsIm5iZiI6MCwiaWF0IjoxNTc1MDUyNDU4LCJpc3MiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJodHRwczovL3Nzby5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjkwNzdjZTMwLThkZjYtNDA1YS05YjgxLTI4NGY3OWY4NTM4ZSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1haWwifQ.signature
2019-11-29 18:34:18.500 ERROR 1 --- [nio-8880-exec-4] o.k.adapters.OAuthRequestAuthenticator   : failed verification of token: Invalid token signature
2019-11-29 18:34:18.505 DEBUG 1 --- [nio-8880-exec-4] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?state=fe28c617-c87f-47e2-8941-ed801999efd0&session_state=9077ce30-8df6-405a-9b81-284f79f8538e&code=e4554d93-c232-48ec-bebc-d99c0dac999c.9077ce30-8df6-405a-9b81-284f79f8538e.d038938f-0a7f-4e3f-989e-5082f40c2bfc", parameters={masked}
2019-11-29 18:34:18.510 DEBUG 1 --- [nio-8880-exec-4] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2019-11-29 18:34:18.530 DEBUG 1 --- [nio-8880-exec-4] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2019-11-29 18:34:18.530 DEBUG 1 --- [nio-8880-exec-4] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 403
2019-11-29 19:10:18.000 DEBUG 1 --- [nio-8880-exec-6] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/
2019-11-29 19:10:18.002 DEBUG 1 --- [nio-8880-exec-6] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /
2019-11-29 19:10:18.002 DEBUG 1 --- [nio-8880-exec-6] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/
2019-11-29 19:10:18.002 DEBUG 1 --- [nio-8880-exec-6] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-29 19:10:18.002 DEBUG 1 --- [nio-8880-exec-6] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-29 19:10:18.003 DEBUG 1 --- [nio-8880-exec-6] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-29 19:10:18.003 DEBUG 1 --- [nio-8880-exec-6] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-29 19:10:18.003 DEBUG 1 --- [nio-8880-exec-6] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-29 19:10:18.003 DEBUG 1 --- [nio-8880-exec-6] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-29 19:10:18.003 DEBUG 1 --- [nio-8880-exec-6] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-29 19:10:18.004 DEBUG 1 --- [nio-8880-exec-6] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-29 19:10:18.016 DEBUG 1 --- [nio-8880-exec-6] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-29 19:10:18.018 DEBUG 1 --- [nio-8880-exec-6] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-29 19:10:18.474 DEBUG 1 --- [nio-8880-exec-7] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/favicon.ico
2019-11-29 19:10:18.478 DEBUG 1 --- [nio-8880-exec-7] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /favicon.ico
2019-11-29 19:10:18.478 DEBUG 1 --- [nio-8880-exec-7] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/favicon.ico
2019-11-29 19:10:18.478 DEBUG 1 --- [nio-8880-exec-7] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-29 19:10:18.478 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : GET "/favicon.ico", parameters={}
2019-11-29 19:10:18.478 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-29 19:10:18.480 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.r.ResourceHttpRequestHandler     : Resource not found
2019-11-29 19:10:18.480 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : Completed 404 NOT_FOUND
2019-11-29 19:10:18.480 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error", parameters={}
2019-11-29 19:10:18.481 DEBUG 1 --- [nio-8880-exec-7] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
2019-11-29 19:10:18.485 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [*/*] and supported [application/json, application/*+json, application/json, application/*+json]
2019-11-29 19:10:18.485 DEBUG 1 --- [nio-8880-exec-7] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{timestamp=Fri Nov 29 19:10:18 GMT 2019, status=404, error=Not Found, message=No message available,  (truncated)...]
2019-11-29 19:10:18.487 DEBUG 1 --- [nio-8880-exec-7] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 404
2019-11-29 19:10:24.348 DEBUG 1 --- [nio-8880-exec-8] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-29 19:10:24.348 TRACE 1 --- [nio-8880-exec-8] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-29 19:10:24.348 TRACE 1 --- [nio-8880-exec-8] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-29 19:10:24.348 TRACE 1 --- [nio-8880-exec-8] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-29 19:10:24.350 TRACE 1 --- [nio-8880-exec-8] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-29 19:10:24.350 DEBUG 1 --- [nio-8880-exec-8] o.k.adapters.OAuthRequestAuthenticator   : there was no code
2019-11-29 19:10:24.350 DEBUG 1 --- [nio-8880-exec-8] o.k.adapters.OAuthRequestAuthenticator   : redirecting to auth server
2019-11-29 19:10:24.350 DEBUG 1 --- [nio-8880-exec-8] o.k.adapters.OAuthRequestAuthenticator   : callback uri: https://inventory.eupraxialabs.com/items
2019-11-29 19:10:24.356 DEBUG 1 --- [nio-8880-exec-8] o.k.adapters.OAuthRequestAuthenticator   : Sending redirect to login page: https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/auth?response_type=code&client_id=inventory-app&redirect_uri=https%3A%2F%2Finventory.eupraxialabs.com%2Fitems&state=a90ad2f4-cd45-447b-a556-026b2ed3acbc&login=true&scope=openid

Let’s get the public key for the realm logistics-dod:

[centos@vm-controller certificate-management]$ curl -vvvvL https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod
* About to connect() to sso-f5.eupraxialabs.com port 443 (#0)
*   Trying 40.87.48.212...
* Connected to sso-f5.eupraxialabs.com (40.87.48.212) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: using client certificate: estrada.jeremy.george.1503395688 - Eupraxia Labs
*       subject: E=jeremy.estrada@eupraxialabs.com,CN=estrada.jeremy.george.1503395688,OU=Cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
*       start date: Nov 23 21:50:36 2019 GMT
*       expire date: Nov 22 21:50:36 2020 GMT
*       common name: estrada.jeremy.george.1503395688
*       issuer: E=admin@eupraxialabs.com,CN=Eupraxia Labs Certificate Authority,OU=cloud,O=Eupraxia Labs,L=Austin,ST=Texas,C=US
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=sso-f5.eupraxialabs.com
*       start date: Nov 22 14:25:13 2019 GMT
*       expire date: Feb 20 14:25:13 2020 GMT
*       common name: sso-f5.eupraxialabs.com
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> GET /auth/realms/logistics-dod HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sso-f5.eupraxialabs.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.1
< Date: Sat, 30 Nov 2019 15:12:22 GMT
< Content-Type: application/json
< Content-Length: 644
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: INGRESSCOOKIE=1575126743.947.79356.737265; Expires=Mon, 02-Dec-19 15:12:22 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
< Cache-Control: no-cache
< X-Powered-By: Undertow/1
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
* Connection #0 to host sso-f5.eupraxialabs.com left intact
{"realm":"logistics-dod","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo5oGTcqbxNhGurufKhC2ARdpUaSvXKKGy1DRSRK35toTydMcnONsQs6RxYd1kzs+6i8U+HQJdR85CgdpY+pNUU+IWx0jORta3ii1ElBATq6GO8KukiXKPVvpCYKk6XaezVd8uAjT8TpbYei8djumklibFJ0A8UzLYEF2Sd+AV+ia0kFLPFQKnygaJoZ7qfL+vkNUOmtfmscP0ZXvC1B+9uPiYY8AxZBXC0wjDdxTyAr/iHjgcPxevfHuKrM3M7IK6JHUUQYpAjdS0/3qU+4wED5Z3Qluo/Z0OpFpNofb5l9PFcdRNXy3Jkjvy4rOF27RA7PQOgSI3anyhwPToXCiHQIDAQAB","token-service":"https://sso.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect","account-service":"https://sso.eupraxialabs.com/auth/realms/logistics-dod/account","tokens-not-before":1574610403}[centos@vm-controller certificate-management]$````

Take this key and add it to the Spring Boot Relying Party’s property in *application.properties:

Note: Do not copy it from the console in XtremeCloud SSO web-based user interface.

keycloak.realm-key=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo5oGTcqbxNhGurufKhC2ARdpUaSvXKKGy1DRSRK35toTydMcnONsQs6RxYd1kzs+6i8U+HQJdR85CgdpY+pNUU+IWx0jORta3ii1ElBATq6GO8KukiXKPVvpCYKk6XaezVd8uAjT8TpbYei8djumklibFJ0A8UzLYEF2Sd+AV+ia0kFLPFQKnygaJoZ7qfL+vkNUOmtfmscP0ZXvC1B+9uPiYY8AxZBXC0wjDdxTyAr/iHjgcPxevfHuKrM3M7IK6JHUUQYpAjdS0/3qU+4wED5Z3Qluo/Z0OpFpNofb5l9PFcdRNXy3Jkjvy4rOF27RA7PQOgSI3anyhwPToXCiHQIDAQAB

Here is a working trace from the Spring Boot Relying Party’s log:

2019-11-30 15:39:36.442 DEBUG 1 --- [nio-8880-exec-1] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/
2019-11-30 15:39:36.449 DEBUG 1 --- [nio-8880-exec-1] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /
2019-11-30 15:39:36.449 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/
2019-11-30 15:39:36.450 DEBUG 1 --- [nio-8880-exec-1] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-30 15:39:36.450 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : GET "/", parameters={}
2019-11-30 15:39:36.451 DEBUG 1 --- [nio-8880-exec-1] o.s.b.a.w.s.WelcomePageHandlerMapping    : Mapped to ParameterizableViewController [view="forward:index.html"]
2019-11-30 15:39:36.451 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-30 15:39:36.451 DEBUG 1 --- [nio-8880-exec-1] o.s.w.servlet.view.InternalResourceView  : View name 'forward:', model {}
2019-11-30 15:39:36.451 DEBUG 1 --- [nio-8880-exec-1] o.s.w.servlet.view.InternalResourceView  : Forwarding to [index.html]
2019-11-30 15:39:36.452 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : "FORWARD" dispatch for GET "/index.html", parameters={}
2019-11-30 15:39:36.453 DEBUG 1 --- [nio-8880-exec-1] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/", "classpath:/resources/", "classpath:/static/", "classpath:/public/", "/"]
2019-11-30 15:39:36.477 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting from "FORWARD" dispatch, status 200
2019-11-30 15:39:36.477 DEBUG 1 --- [nio-8880-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2019-11-30 15:39:37.860 DEBUG 1 --- [io-8880-exec-10] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-30 15:39:37.863 TRACE 1 --- [io-8880-exec-10] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-30 15:39:37.863 TRACE 1 --- [io-8880-exec-10] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-30 15:39:37.863 TRACE 1 --- [io-8880-exec-10] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-30 15:39:37.863 TRACE 1 --- [io-8880-exec-10] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-30 15:39:37.863 DEBUG 1 --- [io-8880-exec-10] o.k.adapters.OAuthRequestAuthenticator   : there was no code
2019-11-30 15:39:37.864 DEBUG 1 --- [io-8880-exec-10] o.k.adapters.OAuthRequestAuthenticator   : redirecting to auth server
2019-11-30 15:39:37.864 DEBUG 1 --- [io-8880-exec-10] o.k.adapters.OAuthRequestAuthenticator   : callback uri: https://inventory.eupraxialabs.com/items
2019-11-30 15:39:37.867 DEBUG 1 --- [io-8880-exec-10] o.k.adapters.OAuthRequestAuthenticator   : Sending redirect to login page: https://sso-f5.eupraxialabs.com/auth/realms/logistics-dod/protocol/openid-connect/auth?response_type=code&client_id=inventory-app&redirect_uri=https%3A%2F%2Finventory.eupraxialabs.com%2Fitems&state=de305055-561e-4277-bec5-782542e86efd&login=true&scope=openid
2019-11-30 15:39:40.505 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items?state=de305055-561e-4277-bec5-782542e86efd&session_state=02e58450-1839-4f79-971d-ce913978c929&code=f8c8154c-52fe-4e85-ae86-a44593c5f6bb.02e58450-1839-4f79-971d-ce913978c929.d038938f-0a7f-4e3f-989e-5082f40c2bfc
2019-11-30 15:39:40.511 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-30 15:39:40.513 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-30 15:39:40.513 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-30 15:39:40.513 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-30 15:39:40.513 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : there was a code, resolving
2019-11-30 15:39:40.513 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : checking state cookie for after code
2019-11-30 15:39:40.513 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : ** reseting application state cookie
2019-11-30 15:39:40.781 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : Verifying tokens
2019-11-30 15:39:40.784 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   :                access_token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJSN2EwV3VsTHFnX1RDckpORGdqSTNPZ0JRYVFGY2duZUxrcklIWFR2X0VrIn0.eyJqdGkiOiIzODhiZDQwMS1mZDMyLTRmNjktOThkMS02MjdlYjc3MGYyOTgiLCJleHAiOjE1NzUxMzE5ODAsIm5iZiI6MCwiaWF0IjoxNTc1MTI4MzgwLCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MTU3NTEyODM4MCwic2Vzc2lvbl9zdGF0ZSI6IjAyZTU4NDUwLTE4MzktNGY3OS05NzFkLWNlOTEzOTc4YzkyOSIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiUk9MRV9VU0VSIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsInVzZXIiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBwcm9maWxlIGVtYWlsIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsInVzZXJfbmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwibmFtZSI6IkplcmVteSBFc3RyYWRhIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiZXN0cmFkYS5qZXJlbXkuZ2VvcmdlLjE1MDMzOTU2ODgiLCJnaXZlbl9uYW1lIjoiSmVyZW15IiwiZmFtaWx5X25hbWUiOiJFc3RyYWRhIiwiZW1haWwiOiJqZXJlbXkuZXN0cmFkYUBldXByYXhpYWxhYnMuY29tIn0.signature
2019-11-30 15:39:40.784 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   :                id_token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJSN2EwV3VsTHFnX1RDckpORGdqSTNPZ0JRYVFGY2duZUxrcklIWFR2X0VrIn0.eyJqdGkiOiI1YmI5YzQyNS0xYTA2LTQ2YTMtYmJmZC0yZjA1MDA1ZWI0ODIiLCJleHAiOjE1NzUxMzE5ODAsIm5iZiI6MCwiaWF0IjoxNTc1MTI4MzgwLCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJpbnZlbnRvcnktYXBwIiwic3ViIjoiYjA5YTYxMDQtOWYxYS00NmYwLThjZTYtYzJhZWZjYzI5NThhIiwidHlwIjoiSUQiLCJhenAiOiJpbnZlbnRvcnktYXBwIiwiYXV0aF90aW1lIjoxNTc1MTI4MzgwLCJzZXNzaW9uX3N0YXRlIjoiMDJlNTg0NTAtMTgzOS00Zjc5LTk3MWQtY2U5MTM5NzhjOTI5IiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJ1c2VyX25hbWUiOiJlc3RyYWRhLmplcmVteS5nZW9yZ2UuMTUwMzM5NTY4OCIsIm5hbWUiOiJKZXJlbXkgRXN0cmFkYSIsInByZWZlcnJlZF91c2VybmFtZSI6ImVzdHJhZGEuamVyZW15Lmdlb3JnZS4xNTAzMzk1Njg4IiwiZ2l2ZW5fbmFtZSI6IkplcmVteSIsImZhbWlseV9uYW1lIjoiRXN0cmFkYSIsImVtYWlsIjoiamVyZW15LmVzdHJhZGFAZXVwcmF4aWFsYWJzLmNvbSJ9.signature
2019-11-30 15:39:40.785 TRACE 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   :                refresh_token: eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyYmQ5MzY3ZC02OTlkLTRlZTMtOGQxZi1hNjI2MTNkZjExYzEifQ.eyJqdGkiOiI0MGIxMWVhNi03NGMxLTQyMjEtOGI2Mi0yZDhhNDcxYWRjNjQiLCJleHAiOjE1NzUxMzAxODAsIm5iZiI6MCwiaWF0IjoxNTc1MTI4MzgwLCJpc3MiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJhdWQiOiJodHRwczovL3Nzby1mNS5ldXByYXhpYWxhYnMuY29tL2F1dGgvcmVhbG1zL2xvZ2lzdGljcy1kb2QiLCJzdWIiOiJiMDlhNjEwNC05ZjFhLTQ2ZjAtOGNlNi1jMmFlZmNjMjk1OGEiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoiaW52ZW50b3J5LWFwcCIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjAyZTU4NDUwLTE4MzktNGY3OS05NzFkLWNlOTEzOTc4YzkyOSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJST0xFX1VTRVIiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwidXNlciJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIHByb2ZpbGUgZW1haWwifQ.signature
2019-11-30 15:39:40.793 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : Token Verification succeeded!
2019-11-30 15:39:40.793 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.OAuthRequestAuthenticator   : successful authenticated
2019-11-30 15:39:40.793 TRACE 1 --- [nio-8880-exec-2] o.k.a.RefreshableKeycloakSecurityContext : checking whether to refresh.
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       : use realm role mappings
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       : Setting roles:
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       :    role: ROLE_USER
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       :    role: offline_access
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       :    role: uma_authorization
2019-11-30 15:39:40.807 TRACE 1 --- [nio-8880-exec-2] org.keycloak.adapters.AdapterUtils       :    role: user
2019-11-30 15:39:40.808 TRACE 1 --- [nio-8880-exec-2] o.k.a.RefreshableKeycloakSecurityContext : checking whether to refresh.
2019-11-30 15:39:40.808 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : User 'b09a6104-9f1a-46f0-8ce6-c2aefcc2958a' invoking 'https://inventory.eupraxialabs.com/items?state=de305055-561e-4277-bec5-782542e86efd&session_state=02e58450-1839-4f79-971d-ce913978c929&code=f8c8154c-52fe-4e85-ae86-a44593c5f6bb.02e58450-1839-4f79-971d-ce913978c929.d038938f-0a7f-4e3f-989e-5082f40c2bfc' on client 'inventory-app'
2019-11-30 15:39:40.808 DEBUG 1 --- [nio-8880-exec-2] o.k.adapters.RequestAuthenticator        : AUTHENTICATED
2019-11-30 15:39:40.893 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.PreAuthActionsHandler       : adminRequest https://inventory.eupraxialabs.com/items
2019-11-30 15:39:40.897 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : --> authenticate()
2019-11-30 15:39:40.897 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try bearer
2019-11-30 15:39:40.897 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try query paramter auth
2019-11-30 15:39:40.897 TRACE 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : try oauth
2019-11-30 15:39:40.897 DEBUG 1 --- [nio-8880-exec-4] o.k.adapters.RequestAuthenticator        : AUTHENTICATED: was cached
2019-11-30 15:39:40.897 DEBUG 1 --- [nio-8880-exec-4] .k.a.t.AbstractAuthenticatedActionsValve : AuthenticatedActionsValve.invoke /items
2019-11-30 15:39:40.897 DEBUG 1 --- [nio-8880-exec-4] o.k.a.AuthenticatedActionsHandler        : AuthenticatedActionsValve.invoke https://inventory.eupraxialabs.com/items
2019-11-30 15:39:40.897 DEBUG 1 --- [nio-8880-exec-4] o.k.a.AuthenticatedActionsHandler        : Policy enforcement is disabled.
2019-11-30 15:39:40.898 DEBUG 1 --- [nio-8880-exec-4] o.s.web.servlet.DispatcherServlet        : GET "/items", parameters={}
2019-11-30 15:39:40.898 DEBUG 1 --- [nio-8880-exec-4] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.eupraxialabs.inventoryapp.InventoryController#getItems(Model)
2019-11-30 15:39:40.899 DEBUG 1 --- [nio-8880-exec-4] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8]
2019-11-30 15:39:40.900 DEBUG 1 --- [nio-8880-exec-4] o.s.w.s.view.freemarker.FreeMarkerView   : View name 'items', model {items=[AIM-9 Sidewinder Missile, AGM-123 Skipper Missile, Towed Dummy Target, Air-Towed Dummy Target]}
2019-11-30 15:39:40.900 DEBUG 1 --- [nio-8880-exec-4] o.s.w.s.view.freemarker.FreeMarkerView   : Rendering [items.ftl]
2019-11-30 15:39:40.909 DEBUG 1 --- [nio-8880-exec-4] o.s.web.servlet.DispatcherServlet        : Completed 200 OK

For reference purposes, the Token Verification succeeded! output comes from line 135 of *RefreshableKeycloakSecurityContext.java in our source code.

### OCSP Stapling

Some basic OCSP stapling support.

New directives:

ssl_trusted_certificate /path/to/file;

  Specifies a file with CA certificates in the PEM format used for
  certificate verification.  In contrast to ssl_client_certificate, DNs
  of these certificates aren't sent to a client in CertificateRequest.

ssl_stapling on|off;

  Activates OCSP stapling.

ssl_stapling_file /path/to/file;

  Use predefined OCSP response for stapling, do not query responder.
  Assumes OCSP response in DER format as produced by "openssl ocsp".

ssl_stapling_responder URL;

  Use specified OCSP responder instead of one found in AIA certificate
  extension.

ssl_stapling_verify on|off;

  This directive allows to switch off OCSP response verification, resulting
  in less configuration of trusted certificates needed in some cases.

  Note that for stapling OCSP response verification isn't something required
  as it will be done by a client anyway.  But doing verification on a server
  allows to mitigate some attack vectors, most notably stop an attacker from
  presenting some specially crafted data to all site clients.

Example configuration:

  server {
      listen 443 ssl;

      ssl_certificate /path/to/cert.pem;
      ssl_certificate_key /path/to/key.pem;

      ssl_stapling on;
      ssl_trusted_certificate /path/to/ca.pem;

      resolver 8.8.8.8;
  }

Known limitations:

- Unless externally set OCSP response is used (via the "ssl_stapling_file"
  directive), stapled response won't be sent in a first connection.  This
  is due to the fact that OCSP responders are currently queried by nginx
  once it receives connection with certificate_status extension in ClientHello,
  and due to limitations in OpenSSL API (certificate status callback is
  blocking).

- Cached OCSP responses are currently stored in local process memory (thus
  each worker process will query OCSP responders independently).  This
  shouldn't be a problem as typical number of worker processes is low, usually
  set match number of CPUs.

- Various timeouts are hardcoded (connect/read/write timeouts are 60s,
  response is considered to be valid for 1h after loading).  Adding
  configuration directives to control these would be trivial, but it may
  be a better idea to actually omit them for simplicity.

- Only "http://" OCSP responders are recognized.

Patch can be found here:

http://nginx.org/patches/ocsp-stapling/

Testing and review appreciated.  Thanks to Comodo, DigiCert and GlobalSign
for sponsoring this work.

-- 
NGINX, Inc., http://nginx.com

Configuring OCSP for XtremeCloud Single Sign-On

The Online Certificate Status Protocol (OCSP) enables applications to determine the (revocation) state of an identified certificate. OCSP may be used to satisfy some of the operational requirements of providing more timely revocation information than is possible with CRLs and may also be used to obtain additional status information. An OCSP client issues a status request to an OCSP responder and suspends acceptance of the certificate in question until the responder provides a response.

[centos@vm-controller certificate-management]$ openssl ocsp -issuer letsencrypt-root-cert.crt -cert server-sso-f5.crt -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
          Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
          Serial Number: 030EC0E06AA0E1321A28C3A07201F2DDE3E8
    Request Extensions:
        OCSP Nonce:
            041000E53C1E5D61AAC71BC325EF637B9A11
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Produced At: Dec 12 20:51:00 2019 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
      Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
      Serial Number: 030EC0E06AA0E1321A28C3A07201F2DDE3E8
    Cert Status: good
    This Update: Dec 12 20:00:00 2019 GMT
    Next Update: Dec 19 20:00:00 2019 GMT

    Signature Algorithm: sha256WithRSAEncryption
         39:34:bf:15:c2:83:45:c7:07:48:00:86:3a:12:66:26:d1:66:
         65:e0:2f:cb:4b:1b:58:71:c8:31:c7:5a:8d:b5:30:53:a5:3f:
         35:0a:f1:bc:dd:2b:ef:f9:1e:0a:44:a7:fb:db:55:65:29:3c:
         8e:56:e9:cb:cf:8a:61:da:c1:1e:77:55:b4:98:6f:b0:1a:9b:
         fa:7b:99:60:cb:df:96:59:3d:4b:93:4d:9e:ab:05:1c:28:91:
         66:11:b0:79:5a:cc:92:b9:b3:de:4f:63:2e:a9:7f:cd:48:4c:
         be:1f:fa:45:86:9a:b0:0f:0e:17:e9:77:68:27:a4:33:ac:e4:
         cd:f6:80:2e:82:52:00:3c:51:13:26:09:49:71:05:80:d6:fd:
         d2:da:52:7b:66:1c:c5:f9:56:30:40:de:e1:be:4e:aa:cc:ee:
         89:c9:90:34:cd:4f:24:aa:4b:f8:67:5f:54:b2:0f:df:c3:78:
         c4:7c:ff:6c:f8:e3:22:8a:c2:81:ca:c7:00:cb:5e:97:91:15:
         8c:c2:31:3c:be:de:94:15:cb:d4:2c:d0:f8:cf:02:c6:5c:cc:
         95:05:0d:18:f6:77:ba:98:a2:c0:6e:6a:28:77:a3:b0:77:22:
         e8:91:cd:ff:a4:bc:59:ed:2d:f0:b0:4f:dc:c4:2e:0d:05:95:
         f9:92:50:3b
WARNING: no nonce in response
Response Verify Failure
139858255161232:error:27069076:OCSP routines:OCSP_basic_verify:signer certificate not found:ocsp_vfy.c:92:
server-sso-f5.crt: good
        This Update: Dec 12 20:00:00 2019 GMT
        Next Update: Dec 19 20:00:00 2019 GMT

As we can see, the Let’s Encrypt OCSP responder has responded that the certificate is good.

With OCSP there are three possible statuses for a certificate: GOOD, REVOKED, or UNKNOWN.

  • GOOD means that the certificate is not revoked according to this OCSP responder. It does not mean the certificate is valid otherwise (I.E. could still be expired).
  • REVOKED means that the certificate is revoked.
  • UNKNOWN means that the OCSP responder does not know about this certificate. You must be careful with unknown responses, as many applications consider UNKNOWN responses to be the same as GOOD.

Referring to a CA’s CRL PEM file, let’s take a look at a few revocations:

[centos@vm-controller crls]$ openssl crl -inform PEM -text -noout -in DODEMAILCA_50.crl.pem|more
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-50
        Last Update: Dec  3 08:00:00 2019 GMT
        Next Update: Dec 10 17:00:00 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:65:0A:7B:5D:26:1B:30:CA:2D:DC:FC:D6:73:C7:1E:C1:7F:04:8C:C8

            X509v3 CRL Number:
                2762
Revoked Certificates:
    Serial Number: 0BFFFD
        Revocation Date: Nov 19 20:28:50 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
    Serial Number: 05FFFE
        Revocation Date: Jul 26 17:31:40 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
    Serial Number: 0BFFFC
        Revocation Date: Aug 21 15:01:24 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
.... many more...

Referring to a CA’s DER file, let’s look at the same CRL:

[centos@vm-controller crls]$ openssl crl -inform DER -text -noout -in DODEMAILCA_50.crl|more
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /C=US/O=U.S. Government/OU=DoD/OU=PKI/CN=DOD EMAIL CA-50
        Last Update: Dec  3 08:00:00 2019 GMT
        Next Update: Dec 10 17:00:00 2019 GMT
        CRL extensions:
            X509v3 Authority Key Identifier:
                keyid:65:0A:7B:5D:26:1B:30:CA:2D:DC:FC:D6:73:C7:1E:C1:7F:04:8C:C8

            X509v3 CRL Number:
                2762
Revoked Certificates:
    Serial Number: 0BFFFD
        Revocation Date: Nov 19 20:28:50 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
    Serial Number: 05FFFE
        Revocation Date: Jul 26 17:31:40 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
    Serial Number: 0BFFFC
        Revocation Date: Aug 21 15:01:24 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Superseded
    Serial Number: 05FFF5
        Revocation Date: May 16 17:38:12 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Key Compromise
            Invalidity Date:
                May 16 17:29:45 2019 GMT
    Serial Number: 05FFF3
        Revocation Date: Jan 24 15:27:00 2019 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:
                Key Compromise
            Invalidity Date:
                Jan 24 15:19:44 2019 GMT