Trust custom Certificate Authority

If your AWX has to trust custom Certificate Authority, you can pass the CA certificates to AWX. This is helpful in cases:

Use private Git repository via SSL, without ignoring SSL verification.
Use LDAPS to authenticate users.

Refer the official documentation for more information.

Overview

Trusting custom Certificate Authority can be achieved by following steps:

Creating new Secret which includes your certificates
Passing it to your AWX by specifying the name of the Secret as your AWX's specification

There are two kinds of certificate, one is used to trust LDAP server, and the other is used as the CA bundle.

Fields in the specification for AWX	Keys in Secret	Containers in AWX pod that the certificate will be mounted	Paths that the certificate will be mounted as
`ldap_cacert_secret`	`ldap-ca.crt`	`awx-web`	`/etc/openldap/certs/ldap-ca.crt`
`bundle_cacert_secret`	`bundle-ca.crt`	`awx-web`, `awx-task`, and `awx-ee`	`/etc/pki/ca-trust/source/anchors/bundle-ca.crt`

Note that the awx-ee container is used to run management jobs only, not EE which runs your playbooks. If the EE running your playbook needs a certificate, you will need to customize the pod specification.

If you want to mount the certificate to the additional containers in AWX pod or the additional path other than above, you shoud add extra volumes and extra mounts using extra_volumes and _extra_volume_mounts field, but this is not covered in this guide. Refer to the official documentation.

Prepare required CA certificatess

Place your certificates under base directory.

$ ls -l base
total 32
-rw-rw-r--. 1 kuro kuro  801 Feb 27 00:23 awx.yaml
-rw-rw-r--. 1 kuro kuro 1339 Feb 27 00:44 cacert.pem     👈👈👈
-rw-rw-r--. 1 kuro kuro  610 Feb 27 00:23 kustomization.yaml
...

Note that your certificates have to have PEM format. You can check the format of the certificates depending on which of the following commands succeeds.

# Works for PEM format
openssl x509 -in cacert.crt -text

# Works for DER format
openssl x509 -in cacert.crt -inform DER -text

# Works for PKCS #7 format
openssl pkcs7 -in cacert.crt -text

# Works for PKCS #12 format
openssl pkcs12 -in cacert.crt -info

If your certificate doesn't have PEM format, you can convert it by followings:

# Convert DER to PEM
openssl x509 -in cacert.crt -inform DER -out cacert.pem -outform PEM

# Convert PKCS #7 to PEM
openssl pkcs7 -print_certs -in cacert.crt -out cacert.pem -outform PEM

# Convert PKCS #12 to PEM
openssl pkcs12 -in cacert.crt -out cacert.pem -nokeys -nodes

Modify `base/kustomization.yaml`

Add following lines under secretGenerator in base/kustomization.yaml.

Note that this example provides both ldap-ca.crt and bundle-ca.crt, but you can remove unnecessary line if you don't need both of them. ldap-ca.crt will be used as the CA certificate for LDAP server, and bundle-ca.crt will be used as the CA bundle.

...
secretGenerator:
  ...
  - name: awx-custom-certs     👈👈👈
    type: Opaque     👈👈👈
    files:     👈👈👈
      - ldap-ca.crt=<Name Of Your Certificate File>     👈👈👈
      - bundle-ca.crt=<Name Of Your Certificate File>     👈👈👈
  ...

Modify `base/awx.yaml`

Add following lines under secretGenerator in base/kustomization.yaml.

Note that this example provides both ldap_cacert_secret (should have ldap-ca.crt) and bundle_cacert_secret (should have bundle-ca.crt), but you can remove unnecessary line if you don't need both of them.

...
spec:
  ...
  ldap_cacert_secret: awx-custom-certs     👈👈👈
  bundle_cacert_secret: awx-custom-certs     👈👈👈
  ...

Apply configuration

Invoke apply command. This will start re-deployment of your AWX.

kubectl apply -k base

You can monitor the progress of the re-deployment by following command:

kubectl -n awx logs -f deployments/awx-operator-controller-manager

Troubleshooting

If you have problem with SSL connection such as LDAPS, you can verify your certificates inside the pod.

# Open Bash shell of the "awx-web" container
$ kubectl -n awx exec -it deployment/awx -c awx-web -- bash
bash-5.1$

First of all, you should ensure your CA certificate is mounted and has PEM format. The certificate should be be dumped as readable plain text by following command, without any error.

# The secret ldap_cacert_secret is mounted as /etc/openldap/certs/ldap-ca.crt
bash-5.1$ openssl x509 -in /etc/openldap/certs/ldap-ca.crt -text

# The secret bundle_cacert_secret is mounted as /etc/pki/ca-trust/source/anchors/bundle-ca.crt
bash-5.1$ openssl x509 -in /etc/pki/ca-trust/source/anchors/bundle-ca.crt

Note that your certificate file should contain both intermediate CA and root CA, if your server certificate is signed by intermediate CA.

# Example output of concatenated CA cert; one for intermediate CA, one for root CA
bash-5.1$ cat /etc/openldap/certs/ldap-ca.crt
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlswDQYJKoZIhvcNAQEL
...
3Eyhv0l7mJw/86twDMFFax+cKOCRFV6NoPOpzK1mzAXmxth6vk8DeRm0ipVpQVQ=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlwwDQYJKoZIhvcNAQEL
...
lVsDxZfbZVpRGkDr8odNurNmz0Xcttr+ZVRkoTy5KUxqIZhQuS6ySJj7yoLawWY=
-----END CERTIFICATE-----

Now you can test SSL connection.

# This is an example to test connection to LDAP server over SSL using /etc/openldap/certs/ldap-ca.crt
bash-5.1$ openssl s_client -connect ldap.example.com:636 -no-CAfile -CAfile /etc/openldap/certs/ldap-ca.crt
CONNECTED(00000003)
depth=2 C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
verify return:1
depth=1 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
verify return:1
depth=0 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
verify return:1
---
Certificate chain     👈👈👈 Ensure that the full certificate chain is recognized
 0 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
   ...
 1 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   ...
 2 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   ...
---
...
---
SSL handshake has read 3210 bytes and written 413 bytes
Verification: OK     👈👈👈 Ensure there is no verification error
---
...
SSL-Session:
    ...
    Verify return code: 0 (ok)     👈👈👈 Ensure there is no verification error
    ...