Migrate your services to service mesh with permissive mTLS

Applications typically consist of many interdependent services spread across independent teams. Traditionally, adopting Consul service mesh required teams to coordinate and migrate all the services to the mesh at once. Otherwise, with transparent proxy enabled, services outside of the mesh will not have the mTLS certificates required to securely communicate with services inside the mesh.

Permissive mTLS mode makes it easier for teams to onboard and migrate their services to Consul. Instead of deploying and configuring ingress gateways and API gateways, operators can temporarily let sidecar proxies allow both mTLS and non-mTLS traffic. This essentially allows any non-mesh services to communicate with services inside the mesh.

In this tutorial, you will:

• Deploy a Kubernetes cluster with Terraform
• Deploy Consul and a demo application on the cluster
• Connect non-mesh services to Consul services using permissive mTLS
• Migrate services to Consul and configuring intentions
• Re-secure the mesh by restricting permissive mTLS

Scenario overview

HashiCups is a coffee shop demo application. It has a microservices architecture and uses Consul service mesh to securely connect the services. At the beginning of this tutorial, the HashiCups backend (products-api and postgres) will be on Consul service mesh. First, you will use permissive mTLS to enable non-mesh traffic (public-api) to seamlessly access mesh services (products-api). Then, you will migrate the non-mesh services to Consul then disable permissive mTLS to harden your security posture.

HashiCups uses the following microservices:

• The nginx service is an NGINX instance that routes requests to the frontend service and serves as a reverse proxy to the public-api service.
• The frontend service provides a React-based UI.
• The public-api service is a GraphQL public API that communicates with the products-api and the payments services.
• The product-api service stores the core HashiCups application logic, including authentication, coffee (product) information, and orders.
• The postgres service is a Postgres database instance that stores user, product, and order information.
• The payments service is a gRPC-based Java application service that handles customer payments.

Prerequisites

If you are not familiar with Consul's core functionality, refer to the Consul Getting Started tutorials collection (VMs, Kubernetes) first.

For this tutorial, you will need:

• An AWS account configured for use with Terraform
• aws-cli v2.0 or later
• kubectl v1.21 or later
• git v2.0 or later
• terraform v1.2 or later
• consul-k8s v1.2.0 or later

Confirm you have the consul-k8s v1.2.0 or later.

$ consul-k8s version  consul-k8s v1.2.0-rc1 (7f6b0ee)

Clone example repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-permissive-mtls

Change into the directory with the newly cloned repository.

$ cd learn-consul-permissive-mtls

This repository has the following:

• The Terraform configuration files in the root directory deploys an EKS cluster in us-east-2. You will deploy Consul and HashiCups (the demo application) onto this Kubernetes cluster.
• The k8s-yamls directory contains YAML configuration files that support this tutorial.
• The hashicups directory contains YAML configuration files for deploying HashiCups.

Deploy Consul and demo application

Initialize your Terraform configuration to download the necessary providers and modules.

$ terraform initInitializing the backend...## ...Initializing provider plugins...## ...Terraform has been successfully initialized!## ...

Then, create the infrastructure. Confirm the run by entering yes. This will take about 15 minutes to deploy your infrastructure.

$ terraform apply## ...Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes## ...Apply complete! Resources: 59 added, 0 changed, 0 destroyed.Outputs:cluster_endpoint = "https://17AAEDA2B5B51F37CB79E4AD63A10BC5.gr7.us-east-2.eks.amazonaws.com"cluster_id = "education-eks-z8SyM53Q"cluster_name = "education-eks-z8SyM53Q"cluster_security_group_id = "sg-034b0e35b4079283e"region = "us-east-2"

Configure kubectl

Now that you have deployed the Kubernetes cluster, configure kubectl to interact with it.

$ aws eks update-kubeconfig \  --region $(terraform output -raw region) \  --name $(terraform output -raw cluster_name)

Deploy Consul

You will now deploy Consul on your Kubernetes cluster with consul-k8s. By default, Consul deploys into its own dedicated namespace (consul). The Consul installation will use the Consul Helm chart file in the k8s-yaml directory. Permissive mTLS requires Consul v1.16.0+. Deploying Consul on each Kubernetes cluster should only take a few minutes.

global:  image: "hashicorp/consul:1.16.0-rc1"  imageK8S: "hashicorp/consul-k8s-control-plane:1.2.0-rc1"

Deploy Consul and confirm the installation with a y.

$ consul-k8s install -config-file k8s-yamls/values.yaml## ...==> Checking if Consul can be installed## ...    Proceed with installation? (Y/n) Y## ...==> Installing Consul ✓ Downloaded charts.## ... ✓ Consul installed in namespace "consul".

Verify that you have installed Consul by inspecting the Kubernetes pods in the consul namespace.

$ kubectl get pods --namespace consulNAME                                           READY   STATUS    RESTARTS   AGEconsul-connect-injector-59b5b4fccd-j9hj8       1/1     Running   0          8m36sconsul-mesh-gateway-7b86b77d99-hhgpr           1/1     Running   0          8m36sconsul-server-0                                1/1     Running   0          8m36sconsul-webhook-cert-manager-57c5bb695c-6b4c5   1/1     Running   0          8m36s

Confirm that the Helm chart version is 1.2.0 or later.

$ helm list --namespace consulNAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSIONconsul  consul          1               2023-06-12 16:43:41.409139 -0700 PDT    deployed        consul-1.2.0-rc1        1.16.0-rc1

Deploy HashiCups

Deploy the HashiCups services.

$ for service in {frontend,nginx,public-api,payments,products-api,postgres,intentions-api-db}; do kubectl apply -f hashicups/$service.yaml; doneservice/frontend createdserviceaccount/frontend createdservicedefaults.consul.hashicorp.com/frontend createddeployment.apps/frontend createdservice/nginx createdserviceaccount/nginx createdservicedefaults.consul.hashicorp.com/nginx createdconfigmap/nginx-configmap createddeployment.apps/nginx createdservice/public-api createdserviceaccount/public-api createdservicedefaults.consul.hashicorp.com/public-api createddeployment.apps/public-api createdservice/payments createdserviceaccount/payments createdservicedefaults.consul.hashicorp.com/payments createddeployment.apps/payments createdservice/products-api createdserviceaccount/products-api createdservicedefaults.consul.hashicorp.com/products-api createdconfigmap/db-configmap createddeployment.apps/products-api createdservice/postgres createdserviceaccount/postgres createdservicedefaults.consul.hashicorp.com/postgres createddeployment.apps/postgres createdserviceintentions.consul.hashicorp.com/postgres createdserviceintentions.consul.hashicorp.com/deny-all created

Verify that you have deployed the services on Kubernetes.

$ kubectl get servicesNAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGEfrontend       ClusterIP   172.20.170.161   <none>        3000/TCP   42skubernetes     ClusterIP   172.20.0.1       <none>        443/TCP    15mnginx          ClusterIP   172.20.255.205   <none>        80/TCP     41spayments       ClusterIP   172.20.222.231   <none>        1800/TCP   37spostgres       ClusterIP   172.20.213.162   <none>        5432/TCP   34sproducts-api   ClusterIP   172.20.19.151    <none>        9090/TCP   36spublic-api     ClusterIP   172.20.62.239    <none>        8080/TCP   39s

List the Consul services. Notice that only the backend services products-api and postgres are registered to Consul since they are currently the only HashiCups services within the service mesh.

$ kubectl exec --namespace consul -it consul-server-0 -- consul catalog servicesDefaulted container "consul" out of: consul, locality-init (init)consulmesh-gatewaypostgrespostgres-sidecar-proxyproducts-apiproducts-api-sidecar-proxy

Open HashiCups in your browser. In a new terminal, port-forward the nginx service to port 8080.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. Notice that it displays no products, since the public-api cannot connect to the products-api.

Connect non-mesh services to Consul services

By default, Consul's sidecar proxies reject all non-mTLS traffic. This prevents services outside Consul from connecting to services inside the mesh unless you set up ingress or API gateways. However, setting up and configuring ingress and API gateways increase service mesh onboarding complexity. Permissive mTLS configures sidecar proxies to temporarily accept both incoming mTLS and incoming non-mTLS connections to simplify service mesh onboarding for your services.

In order to enable your sidecar proxies to accept both mTLS and non-mTLS traffic, you need to:

• Allow permissive mTLS at the mesh level: Configure the service mesh to allow the use of permissive mTLS with the mesh configuration entry.
• Enable permissive mTLS at the service level: Once you have configured your mesh to allow the use of permissive mTLS, set mTLS mode in the service-defaults configuration entry. This overrides the proxy-defaults setting.

Allow permissive mTLS at mesh level

The following file configures the Consul datacenter (partition) to allow permissive mTLS:

apiVersion: consul.hashicorp.com/v1alpha1kind: Meshmetadata:  name: meshspec:  allowEnablingPermissiveMutualTLS: true

Enable permissive mTLS by applying the mesh configuration entry.

$ kubectl apply -f k8s-yamls/mesh-config-entry.yaml --namespace consulmesh.consul.hashicorp.com/mesh created

Set permissive mTLS at service level

The following file configures the products-api service to accept both mTLS and non-mTLS traffic:

apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: products-apispec:  protocol: http  mutualTLSMode: "permissive"

Enable products-api to allow non-mTLS traffic.

$ kubectl apply -f k8s-yamls/service-defaults-products-api.yaml --namespace consul

You may need to re-forward the port of the nginx service.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. It should show a list of coffees.

Notice that even though there is a deny-all intention on the Consul datacenter, public-api was able to connect to products-api. Intentions only take effect for mTLS connections.

Migrate non-mesh services into Consul service mesh

Now, you will migrate the remaining HashiCups services to Consul service mesh.

First, the each service deployment definition, update the consul.hashicorp.com/connect-inject annotation from false to true. You will need to do this for four files:

1. hashicups/nginx.yaml
2. hashicups/frontend.yaml
3. hashicups/public-api.yaml
4. hashicups/payments.yaml

apiVersion: apps/v1kind: Deployment## ...      annotations:        consul.hashicorp.com/connect-inject: "true"

Once you have done this to all four files, apply the changes. In addition, you will apply a file that creates intentions between these services.

$ for service in {frontend,nginx,public-api,payments,intentions-new-services}; do kubectl apply -f hashicups/$service.yaml; doneservice/frontend unchangedserviceaccount/frontend unchangedservicedefaults.consul.hashicorp.com/frontend unchangeddeployment.apps/frontend configuredservice/nginx unchangedserviceaccount/nginx unchangedservicedefaults.consul.hashicorp.com/nginx unchangedconfigmap/nginx-configmap unchangeddeployment.apps/nginx configuredservice/public-api unchangedserviceaccount/public-api unchangedservicedefaults.consul.hashicorp.com/public-api unchangeddeployment.apps/public-api configuredservice/payments unchangedserviceaccount/payments unchangedservicedefaults.consul.hashicorp.com/payments unchangeddeployment.apps/payments configuredserviceintentions.consul.hashicorp.com/public-api createdserviceintentions.consul.hashicorp.com/payments createdserviceintentions.consul.hashicorp.com/frontend created

Verify the services appear in Consul.

$ kubectl exec --namespace consul -it consul-server-0 -- consul catalog servicesDefaulted container "consul" out of: consul, locality-init (init)consulfrontendfrontend-sidecar-proxymesh-gatewaynginxnginx-sidecar-proxypaymentspayments-sidecar-proxypostgrespostgres-sidecar-proxyproducts-apiproducts-api-sidecar-proxypublic-apipublic-api-sidecar-proxy

Set up intentions

Create intentions between the public-api and products-api to allow traffic communication between the services.

$ kubectl apply -f hashicups/intentions-public-products-api.yaml --namespace consulserviceintentions.consul.hashicorp.com/public-api created

Restrict permissive mTLS at service level

Restrict products-api to only accept mTLS traffic.

In k8s-yamls/service-defaults-products-api.yaml, update mutualTLSMode to strict.

apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: products-apispec:  protocol: http  mutualTLSMode: "strict"

Then, apply the changes.

$ kubectl apply -f k8s-yamls/service-defaults-products-api.yaml --namespace consulservicedefaults.consul.hashicorp.com/products-api configured

Restrict permissive mTLS at mesh level

In k8s-yamls/mesh-config-entry.yaml, update allowEnablingPermissiveMutualTLS to false.

apiVersion: consul.hashicorp.com/v1alpha1kind: Meshmetadata:  name: meshspec:  allowEnablingPermissiveMutualTLS: false

Disallowing permissive mTLS at the mesh level does not impact permissive mTLS settings at the service level (proxy-defaults and service-defaults, MutalTLSMode=permissive). This lets organizations disallow permissive mTLS moving forward, while services currently in permissive mode can eventually migrate to the mesh at their own pace.

Then, apply the changes.

$ kubectl apply -f k8s-yamls/mesh-config-entry.yaml --namespace consulmesh.consul.hashicorp.com/mesh configured

Confirm that the HashiCups services can still connect to each other. You may need to re-forward the port of the nginx service.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. It should show a list of coffees.

Clean up environment

To destroy the environment, first uninstall Consul from your Kubernetes cluster. Confirm with a y.

$ consul-k8s uninstall##...    Proceed with uninstall? (y/N) y##...   Only approve if all data from this installation can be deleted. (y/N) y##...

Then, destroy the supporting infrastructure Enter yes to confirm the destroy operation.

$ terraform destroy##...Plan: 0 to add, 0 to change, 59 to destroy.##...Do you really want to destroy all resources?  Terraform will destroy all your managed infrastructure, as shown above.  There is no undo. Only 'yes' will be accepted to confirm.  Enter a value: yes##...Destroy complete! Resources: 59 destroyed.

Next steps

In this tutorial, you used the Consul permissive mTLS to let Consul services accept both non-mTLS and mTLS traffic. In the process, you learned how to seamlessly connect non-service mesh traffic to Consul service mesh and migrate services to Consul.

Feel free to explore these tutorials and collections to learn more about Consul service mesh, microservices, and Kubernetes security.

• Onboard services with permissive mTLS mode enabled documentation
• Consul Kubernetes Deployment Guide
• Migrate to Microservices collection
• Consul Kubernetes Security