Skip to content

Ingress NGINX retirement and migration

The Ingress NGINX controller, which has been the default Ingress controller in our Kubernetes clusters, is being retired by the Kubernetes project.

As a result, we are migrating all clusters to the HAProxy ingress controller.

Why did we choose HAProxy?

How will we migrate?

Ingress annotations are specific to each Ingress controller. This means there is no drop-in replacement for Ingress NGINX, and every Ingress resource will need to be updated to use HAProxy annotations. You will need to update your Ingress resources accordingly.

During this migration period both Ingress controllers will run on your Kubernetes cluster.
We will install HAProxy ingress controller in migration mode.
In this migration mode all traffic is still handled by Ingress NGINX. Once you create a new Ingress resource with ingressClassName: haproxy the traffic for this domain is handled by the HAProxy ingress controller. This way you can migrate to the HAProxy ingress controller on a domain by domain basis.

Phase 1: Installing HAProxy ingress in migration mode

In the first phase, we will deploy the HAProxy ingress controller in migration mode alongside the existing Ingress NGINX controller.

During this period:

  • Nothing should fundamentally change, all traffic is still handled by Ingress NGINX.
  • Evaluate that everything still works as expected and the migration mode introduces no unexpected side effects.

Duration: 1 week

The main purpose of this phase is making sure the HAProxy ingress migration mode does not cause any unexpected issues for your specific application.
We have already tested this migration mode internally and have not noticed any issues.

Phase 2: Switch to HAProxy on a domain by domain basis

In the second phase, it's time to add HAProxy ingress resources to replace your Ingress NGINX resources.

During this period:

  • Add new Ingress resources using ingressClassName: haproxy.
  • Most Ingress resources have NGINX specific annotations, have a look at migrating ingress annotations for an overview on how to migrate these annotations. Not all annotations supported by Ingress NGINX are supported in HAProxy ingress.
  • Once an Ingress resource using ingressClassName: haproxy exists, this traffic for this domain is automatically handled by the HAProxy ingress controller. It's recommended to leave your existing Ingress NGINX resources in place. If unexpected side effects appear you can delete the Ingress with ingressClassName: haproxy and the traffic will be handled by Ingress NGINX once again.
  • Have a look at the known limitations.

Phase 3: Remove NGINX

In the final phase, all traffic will be handled by the HAProxy ingress controller. Now we can disable the migration mode and delete the Ingress NGINX controller.
After the Ingress NGINX controller has been removed, you can safely remove the Ingress resources with ingressClassName: nginx from your cluster.

Known limitations

The technical implementation of the migration mode allows you to migrate on a domain by domain basis, not on an Ingress by Ingress basis.
This has two major side effects.

Multiple Ingress resources with the same domain

When you have multiple Ingress resources with the same domain, they should be migrated at the same time.

Example: Here we have two Ingress resources for the mydomain.com domain, when migrating these resources both should be migrated at the same time, since they use the same domain. 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress-api
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              name: http
        path: /api
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress-admin
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              name: http
        path: /admin
        pathType: Prefix

In this case the Ingress resources don't use any specific annotations, creating an identical Ingress resource with ingressClassName: haproxy will suffice.

Wildcard certificates

When you have a wildcard certificate, web browsers try to reuse this existing connection for subdomains.
This can cause issues when the root domain is migrated to HAProxy ingress but the subdomain isn't migrated yet.

Example: When mydomain.com shows images from images.mydomain.com, your web browser will try to reuse the existing TCP session to fetch the images. Since the HAProxy ingress controller doesn't know this subdomain yet, this causes issues. 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
spec:
  ingressClassName: haproxy
  rules:
  - host: mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - mydomain.com
    secretName: wildcard-mydomain.com
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress-images
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: images.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-service
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - images.mydomain.com
    secretName: wildcard-mydomain.com

When using wildcard certificates, migrate all subdomains and the root domain to HAProxy ingress at the same time.