In looking to setup load balancing of FreeIPA, there is not too much documentation on how to do this to where Kerberos authentication is successful. After lots of research, I found the key to making this work.

Example haproxy config

global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  log  /dev/log local0 info
  maxconn  200000
  pidfile  /var/run/haproxy.pid
  ssl-default-bind-ciphers  ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
  ssl-default-bind-options  no-sslv3 no-tls-tickets
  stats  socket /var/lib/haproxy/stats mode 600 level admin
  tune.ssl.default-dh-param  4096
  user  haproxy

defaults
  maxconn  200000
  option  redispatch
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

frontend http_connections
  bind :80
  mode http
  default_backend http_backends
  option tcplog

backend http_backends
  mode http
  balance roundrobin
  option forwardfor
  server server1 ipa1.example.com:80 check
  server server2 ipa2.example.com:80 check

frontend https_connections
  bind :443 ssl crt /etc/ssl/certs/http.pem
  mode http
  default_backend https_backends
  option tcplog

backend https_backends
  mode http
  balance roundrobin
  option forwardfor
  server server1 ipa1.example.com:443 check ssl verify none
  server server2 ipa2.example.com:443 check ssl verify none

frontend ldap_connections
  bind :389
  mode tcp
  default_backend ldap_backends
  option tcplog

backend ldap_backends
  mode tcp
  balance roundrobin
  option ldap-check
  server server1 ipa1.example.com:389 check
  server server2 ipa2.example.com:389 check

frontend ldaps_connections
  bind :636 ssl crt /etc/ssl/certs/ldap.pem
  mode tcp
  default_backend ldaps_backends
  option tcplog

backend ldaps_backends
  mode tcp
  balance roundrobin
  option tcp-check
  server server1 ipa1.example.com:636 check ssl verify none
  server server2 ipa2.example.com:636 check ssl verify none

frontend krb_connections
  bind :88
  mode tcp
  default_backend krb_backends
  option tcplog

backend krb_backends
  mode tcp
  balance roundrobin
  option tcp-check
  server server1 ipa1.example.com:88 check
  server server2 ipa2.example.com:88 check

frontend krb_passwd_connections
  bind :464
  mode tcp
  default_backend krb_passwd_backends
  option tcplog

backend krb_passwd_backends
  mode tcp
  balance roundrobin
  option tcp-check
  server server1 ipa1.example.com:464 check
  server server2 ipa2.example.com:464 check

Load balancer FreeIPA config

The load balanacer needs to have a host entry in FreeIPA, you can either enroll the load balancer to FreeIPA as a client or you can use the following to add a host entry for it without enrolling the server.

ipa host-add lb.example.com --random --force

Once you have the host in the system, you need to setup the ldap and http service kerberos key so that systems can authenticate against the load balancer's domain.

ipa service-add ldap/lb.example.com
ipa service-add HTTP/lb.example.com

Allow your administrator user to pull the kerberos key for the ldap and http service created.

ipa service-allow-retrieve-keytab ldap/lb.example.com --users=admin
ipa service-allow-retrieve-keytab HTTP/lb.example.com --users=admin

For a later step where we create SAN certificates for LDAP on each master and replica of FreeIPA, we need to allow each master and replica access to the ldap service.

ipa service-add-host ldap/lb.example.com --host=ipa1.example.com
ipa service-add-host ldap/lb.example.com --host=ipa2.example.com

Allowing clients to authenticate with through the load balancer

Now that you have the service prepared in FreeIPA, we need to add the key for the load balancer domain to the directory server's key tab. Examples online tell you to use the --retrieve argument, however with the fact we just added the service above, the service has not been provisioned with a key yet. So we at least need to pull the key once without the retrieve argument so that the key would be created by the first call. Additional calls without the --retrieve argument will re-create the key, so we only should call this once.

By default, the keytab should be /etc/dirsrv/ds.keytab however this may be over ridden with the KRB5_KTNAME= configuration in one of the configurations in /etc/sysconfig/ prefixed with dirsrv. You can check the keytab used with grep KRB5_KTNAME= /etc/sysconfig/dirsrv* if you do not know what it is.

ipa-getkeytab -s ipa1.example.com -p ldap/lb.example.com -k /etc/dirsrv/ds.keytab --retrieve

Also get the key for the HTTP service as otherwise you may get the error HTTP response code is 401, not 200 when you use ipa-join. The keytab may be either /etc/httpd/conf/krb5.keytab or /var/lib/ipa/gssproxy/http.keytab, you can check with the following:

klist -tke /etc/httpd/conf/krb5.keytab
klist -tke /var/lib/ipa/gssproxy/http.keytab

Get key with the following, same thing applies with the --retrieve option mentioned above:

ipa-getkeytab -s ipa1.example.com -p HTTP/lb.example.com -k /var/lib/ipa/gssproxy/http.keytab --retrieve

If you plan on using the load balancer to access the FreeIPA web gui, you should make sure that the /etc/httpd/alias/ipasession.key file is copied to all replica nodes. This ensures that a connection switch from one replica to another keeps sessions active. You will need to make sure that the FreeIPA service has been restarted for this to take affect.

After the key tab has the key for the load balancer's ldap service, we need to configure kerberos to accept incoming authentications using any key in the key tab we added the load balancer's key to. To do this, edit /etc/krb5.conf and add to the configuration group [libdefaults] the following:

ignore_acceptor_hostname = true

This is the key to allowing the cluster to accept authentication attempts via the load balancer, you can learn more about this option here. Without this option, you will receive the error Invalid credentials when testing. After changing this configuration, you need to restart the directory server on each IPA server.

ipactl restart

Generate SAN certificate for LDAP

For things like STARTTLS to work over the standard LDAP protocol, we need a SAN certificate with the load balancer domain added so that it is trusted. To do this, we must resubmitt the LDAP certificate with the additional domains. As we added each master and replica node to the ldap service of the load balancer, we can do this as follows.

# ipa-getcert request -d /etc/dirsrv/slapd-DIRSRV -n Server-Cert -D ipa1.example.com -D lb.example.com
New signing request "20220121215539" added

Check the status:

# ipa-getcert list -i 20220121215539
Number of certificates and requests being tracked: 8.
Request ID '20220121215539':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DIRSRV',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DIRSRV/pwdfile.txt'
    certificate: type=NSSDB,location='/etc/dirsrv/slapd-DIRSRV',nickname='Server-Cert',token='NSS Certificate DB'
    CA: IPA
    issuer: CN=Certificate Authority,O=REALM
    subject: CN=ipa1.example.com,O=REALM
    expires: 2024-10-19 20:48:27 UTC
    dns: ipa1.example.com,lb.example.com
    principal name: ldap/ipa1.example.com@REALM
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-kp-clientAuth
    pre-save command:
    post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DIRSRV
    track: yes
    auto-renew: yes

If it fails for a temporary error or a reason you have to fix it, you can try submitting it again:

# ipa-getcert resubmit -i 20220121215539
Resubmitting "20220121215539" to "IPA".

You can read the certificate on the system with the following:

sudo certutil -L -d /etc/dirsrv/slapd-DIRSRV -n Server-Cert

Update the http rewrites to accept the load balancer

By default, the /etc/httpd/conf.d/ipa-rewrite.conf configuration will force any connection to be redirected back to itself if the hostname is not right. We need to both fix this and fix referals. Example below:

# VERSION 6 - DO NOT REMOVE THIS LINE

RewriteEngine on

# By default forward all requests to /ipa. If you don't want IPA
# to be the default on your web server comment this line out.
RewriteRule ^/$ https://ipa1.example.com/ipa/ui [L,NC,R=301]

# Redirect to the fully-qualified hostname. Not redirecting to secure
# port so configuration files can be retrieved without requiring SSL.
RewriteCond %{HTTP_HOST}    !^ipa1.example.com$ [NC]
RewriteCond %{HTTP_HOST}    !^lb.example.com$ [NC]
RewriteRule ^/ipa/(.*)      http://ipa1.example.com/ipa/$1 [L,R=301]

# Redirect to the secure port if not displaying an error or retrieving
# configuration.
RewriteCond %{SERVER_PORT}  !^443$
RewriteCond %{REQUEST_URI}  !^/ipa/(errors|config|crl)
RewriteCond %{REQUEST_URI}  !^/ipa/[^\?]+(\.js|\.css|\.png|\.gif|\.ico|\.woff|\.svg|\.ttf|\.eot)$
RewriteRule ^/ipa/(.*)      https://ipa1.example.com/ipa/$1 [L,R=301,NC]

# Rewrite for plugin index, make it like it's a static file
RewriteRule ^/ipa/ui/js/freeipa/plugins.js$    /ipa/wsgi/plugins.py [PT]

# Rewrite rule for load balancer
RequestHeader edit Referer ^https://lb.example.com/ https://ipa1.example.com/

After fixing, you can reload httpd configurations to make them live:

systemctl reload httpd

Generating ldap and http certificates

If the load balancer is a client member of FreeIPA, this is simple. Otherwise, you may have to generate and pull from a server that is a member of FreeIPA. To generate the certificates, do the following:

systemctl enable --now certmonger
ipa-getcert request -f /etc/pki/tls/certs/ldap.crt -k /etc/pki/tls/private/ldap.key -K ldap/lb.example.com -D lb.example.com
ipa-getcert request -f /etc/pki/tls/certs/http.crt -k /etc/pki/tls/private/http.key -K HTTP/lb.example.com -D lb.example.com

You can check the status of the certificate generation with the following:

ipa-getcert list

Once we get the certificates on disk, we then need to generate a PEM file for haproxy. You will likely want to set this up on a cron to auto update.

cat /etc/pki/tls/certs/http.crt /etc/pki/tls/private/http.key > /etc/ssl/certs/http.pem
chmod 600 /etc/ssl/certs/http.pem
cat /etc/pki/tls/certs/ldap.crt /etc/pki/tls/private/ldap.key > /etc/ssl/certs/ldap.pem
chmod 600 /etc/ssl/certs/ldap.pem

With this, you should be able to start haproxy and perform tests.

Testing

You can use klist to list which keys are in a keytab with the following:

klist -tke /etc/dirsrv/ds.keytab
klist -tke /var/lib/ipa/gssproxy/http.keytab

You can test the load balancer on a client system with python and the ldap module as follows:

# kinit -kt /etc/krb5.keytab host/$(hostname -f)
# python
Python 2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ldap, ldap.sasl
>>> l = ldap.initialize('ldap://lb.example.com')
>>> auth_tokens = ldap.sasl.gssapi()
>>> l.sasl_interactive_bind_s('', auth_tokens)
0
>>> l.whoami_s()
'dn: fqdn=host1.example.com,cn=computers,cn=accounts,dc=ipa,dc=example,dc=com'

Example python scripts

1. None SSL

   import ldap, ldap.sasl
   l = ldap.initialize('ldap://lb.example.com')
   auth_tokens = ldap.sasl.gssapi()
   l.sasl_interactive_bind_s('', auth_tokens)
   l.whoami_s()

2. SSL with CA trusted certificate

   import ldap, ldap.sasl
   l = ldap.initialize('ldaps://lb.example.com:636')
   auth_tokens = ldap.sasl.gssapi()
   l.sasl_interactive_bind_s('', auth_tokens)
   l.whoami_s()

3. SSL without trusted certificate

   import ldap, ldap.sasl
   ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
   l = ldap.initialize('ldaps://lb.example.com:636')
   auth_tokens = ldap.sasl.gssapi()
   l.sasl_interactive_bind_s('', auth_tokens)
   l.whoami_s()