Kubernetes on AWS Resources

Kubernetes on AWS Resources

Table of Contents

  1. Introduction
  2. Why is Managing Access to AWS Services a Problem? 
  3. Diving into Implementation with Kube2iam
    1. Overall Architecture
    2. Implementation 
      1. Creating and Attaching IAM Roles 
      2. Deploying Kube2iam
      3. Testing Access from a Test Pod
  4. Kiam
    1. Overall Architecture
    2. Implementation
      1. Creating and Attaching IAM Roles
      2. Deploying Cert Manager and Generating Certificates
      3. Generate CA Private Key and Self-signed Certificate for Kiam Agent-server TLS
      4. Annotating Resources
      5. Deploying Kiam Agent and Server
        1. Kiam Server
  5. IAM Roles for Service Accounts (IRSA)
  6. Conclusion

Introduction

Kubernetes is an open-source container orchestration system that allows you to get the most out of your machine. Using Kubernetes, however, raises the problem of managing access for pods to various Amazon Web Services (AWS). This article covers how to overcome these problems by using specific tools. Here’s how we’ve organized the information:

  • Why managing access can be a problem 
  •  Managing access through Kube2iam
  •  Managing access through KIAM
  •  IAM Roles for Service Accounts (IRSA)

‍ 

Why is Managing Access to AWS Services a Problem? 

Imagine this: A Kubernetes node is hosting an application pod that needs access to AWS DynamoDB tables. Meanwhile, another pod on the same node needs access to an AWS S3 bucket. For both applications to work properly, the Kubernetes worker node must access both the DynamoDB tables and the S3 bucket at the same time.

Now think about this happening to hundreds of pods, all requiring access to various AWS resources. The pods are constantly being scheduled on a Kubernetes cluster that needs to access several different AWS services simultaneously… It’s a lot!

One way to solve this would be to give the Kubernetes node—and therefore the pods—access to all AWS resources. However, this leaves your system an easy target for any potential attacker: if a single pod or node is compromised, an attacker will gain access to your entire AWS infrastructure. To avoid this, you can use tools like Kube2iam, Kiam, and IAM IRSA to improve access from the Kubernetes pods into the AWS resources. The best part? All the access API calls and authentication metrics can be pulled by Prometheus and visualized in Grafana. If you want to try the Prometheus/Grafana part for yourself, get onto our MetricFire free trial and start sending your data.

‍ 

Diving into Implementation with Kube2iam

 

Overall Architecture

Kube2iam is deployed as a DaemonSet in your cluster. Therefore, a pod of Kube2iam will be scheduled to run on every worker node of your Kubernetes cluster. Whenever a different pod makes an AWS API call to access resources, that call will be intercepted by the Kube2iam pod running on that node. Kube2iam then ensures that the pod is assigned appropriate credentials to access the resource. 

You also need to specify an Identity and Access Management (IAM) role in the pod spec. Under the hood, the Kube2iam pod will retrieve temporary credentials for the IAM role of the caller and return them to said caller. Basically, all the Amazon Elastic Compute Cloud (EC2) metadata API calls are made into a proxy. (A Kube2iam pod should run with host networking enabled so that it can make the EC2 metadata API calls.) 

 

Implementation 

Creating and Attaching IAM Roles 

  1. Create an IAM role named my-role which has access to the required AWS resources (for example, an AWS S3 bucket).
  2. Follow these steps to enable trust relationship between the role and the role attached to the Kubernetes worker nodes. (Make sure that the role attached to the Kubernetes API worker has very limited permissions—all the API calls or access requests are made by containers running on the node and will receive credentials using Kube2iam, so the worker node IAM roles do not need access to a large number of AWS Resources.)

            a. Go to the newly created role in AWS console and select the ‘Trust relationships’ tab

            b. Click ‘Edit trust relationship’

            c. Add the following content to the policy:

  

{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "AWS": "<ARN_KUBERNETES_NODES_IAM_ROLE>"
  },
  "Action": "sts:AssumeRole"
}

  

          d. Enable ‘Assume role’ for Node Pool IAM roles. Add the following content to Nodes IAM policy:

  

{
        "Sid": "",
    "Effect": "Allow",
    "Action": [
    	"sts:AssumeRole"
    ],
    "Resource": [
        "arn:aws:iam::810085094893:instance-profile/*"
    ]
}

  

       3. Add the IAM role's name to deployment as an annotation. 

  

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mydeployment
  namespace: default
spec:
...
  minReadySeconds: 5
  template:
      annotations:
        iam.amazonaws.com/role: my-role
    spec:
      containers:
...

   

Deploying Kube2iam

  1. Create the service account, ClusterRole and ClusterRoleBinding to be used by Kube2iam pods. The ClusterRole should have 'get', 'watch' and 'list' access to namespaces and pods under all API groups. You can use the manifest below to create them:

 ‍ 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube2iam
  namespace: kube-system
---
apiVersion: v1
kind: List
items:
  - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: kube2iam
    rules:
      - apiGroups: [""]
        resources: ["namespaces","pods"]
        verbs: ["get","watch","list"]
  - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: kube2iam
    subjects:
    - kind: ServiceAccount
      name: kube2iam
      namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: kube2iam
      apiGroup: rbac.authorization.k8s.io
---

‍  

        2. Deploy the Kube2iam DaemonSet by using the manifest below: 

  

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube2iam
  labels:
    app: kube2iam
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: kube2iam
    spec:
      hostNetwork: true
      serviceAccount: kube2iam
      containers:
        - image: jtblin/kube2iam:latest
          name: kube2iam
          args:
            - "--auto-discover-base-arn"
            - "--iptables=true"
            - "--host-ip=$(HOST_IP)"
            - "--host-interface=cali+"
            - "--verbose"
            - "--debug"
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 8181
              hostPort: 8181
              name: http
          securityContext:
            privileged: true
---

 ‍ 

Note: The Kube2iam container is being run with the arguments --iptables=true and --host-ip=$(HOST_IP), and in privileged mode as true. 

 ‍ 

...
    securityContext:
        privileged: true
...

 ‍ 

The following settings prevent containers running in other pods from directly accessing the EC2 metadata API and gaining unwanted access to AWS resources. The traffic to 169.254.169.254 must be made into a proxy for Docker containers. This can be alternatively applied by running the following command on each Kubernetes worker node:  

‍  

iptables \
  --append PREROUTING \
  --protocol tcp \
  --destination 169.254.169.254 \
  --dport 80 \
  --in-interface docker0 \
  --jump DNAT \
  --table nat \
--to-destination `curl 169.254.169.254/latest/meta-data/local-ipv4`:8181

  

Testing Access from a Test Pod

To check whether your Kube2iam deployment and IAM settings work, you can deploy a test pod with an IAM role specified as an annotation. If everything works, you should be able to check which IAM node gets attached to your pod. This can be easily verified by querying the EC2 Metadata API. Let’s deploy a test pod using the manifest below: 

  

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: access-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: access-test
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: access-test
      annotations:
        iam.amazonaws.com/role: my-role
    spec:
      containers:
      - name: access-test
        image: "iotapi322/worker:v4"

 ‍ 

Run the following command in the test pod created:  

‍  

curl 169.254.169.254/latest/meta-data/iam/security-credentials/

 ‍ 

You should get myrole as the response to this API. 

I highly recommend tailing the logs of the Kube2iam pod running on that node in order to gain a deeper understanding of how and when the API calls are being intercepted. Once the setup works as expected, you should turn off verbosity in the Kube2iam deployment in order to avoid bombarding your logging backend.

‍  

  

Kiam

While very helpful, Kube2iam has two major issues that Kiam aims to resolve:

  • Data races under load condition: When you have very high spikes in application load and there are several pods in the cluster, sometimes Kube2iam returns incorrect credentials to those pods. The GitHub issue can be referenced here
  • Pre-fetch credentials: Access credentials are assigned to the IAM role specified in the pod spec before the container processes boots in the pod. By assigning the credentials before, Kiam reduces start latency and improves reliability.

 

Additional features of Kiam include:

  • Use of structured logging to improve the integration into your Elacsticsearch, Logstash, Kibana (ELK) setup with pod names, roles, access key IDs, etc.
  • Use of metrics to track response times, cache hit rates, etc. These metrics can be readily scraped by Prometheus and rendered over Grafana. 

  

Overall Architecture

Kiam is based on agent-server architecture. 

  • Kiam Agent: This is the process that would typically be deployed as a DaemonSet to ensure that pods have no access to the AWS Metadata API. Instead, the Kiam agent runs an HTTP proxy which intercepts credentials requests and passes on everything else. 
  • Kiam Server: This process is responsible for connecting the Kubernetes API servers to watch pods, and for communicating with AWS Security Token Service (STS) to request credentials. It also maintains a cache of credentials for roles currently in use by running pods, ensuring that credentials are refreshed every few minutes and stored before the pods need them.

  

Implementation

Similar to Kube2iam, in order for a pod to get credentials for any IAM role, that role should be specified as an annotation in the deployment manifest. Additionally, you need to specify which IAM roles can be allocated inside a particular namespace using appropriate annotations. This enhances security and lets you fine tune control of IAM roles. 

  

Creating and Attaching IAM Roles

1. Create an IAM role named kiam-server with appropriate access to AWS resources.

2. Enable trust relationship between the kiam-server role and the role attached to Kubernetes master nodes by following these steps. (Make sure that the role attached to the Kubernetes API worker has very limited permissions—all the API calls or access requests are made by containers running on the node and will receive credentials using Kiam. The worker node IAM roles do not need access to many AWS resources.)

     a. Go to the newly created role in AWS console and select the ‘Trust relationships’ tab. 

     b. Click on ‘Edit trust relationship’.

     c. Add the following content to the policy:

 ‍ 

{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "AWS": "<ARN_KUBERNETES_MASTER_IAM_ROLE>"
  },
  "Action": "sts:AssumeRole"
}

  

3.    Add in-line policy to the kiam-server role.

‍  

{
  "Version": "2012-10-17",
  "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "sts:AssumeRole"
       ],
       "Resource": "*"
   }
 ]
}

  

4.   Create the IAM role (let's call it my-role) with appropriate access to AWS resources.

5.   Enable trust relationship between the newly created role and the Kiam server role. 

     To do so: 

     a. Go to the newly created role in AWS console and select ‘Trust relationships’ 

     b. Click ‘Edit trust relationship’

     c. Add the following content to the policy:

 ‍ 

{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "AWS": "<ARN_KIAM-SERVER_IAM_ROLE>"
  },
  "Action": "sts:AssumeRole"
}

  

6.    Enable ‘Assume Role’ for Master Pool IAM roles. Add the following content as in-line policy to master IAM roles:

‍  

{
  "Version": "2012-10-17",
  "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "sts:AssumeRole"
       ],
       "Resource": "<ARN_KIAM-SERVER_IAM_ROLE>"
   }
 ]

‍  

All the communication between Kiam agent and server is TLS encrypted. This enhances security. To do this we need to first deploy cert-manager in our Kubernetes cluster and generate certificates for our agent-server communication.

‍  

Deploying Cert Manager and Generating Certificates

1.   Install the custom resource definition resources separately.

  

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml

  

2.   Create the namespace for cert-manager.

  

kubectl create namespace cert-manager

  

3.   Label the cert-manager namespace to disable resource validation.

  

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

  

4.   Add the Jetstack Helm repository.

  

helm repo add jetstack https://charts.jetstack.io

  

5.   Update your local Helm chart repository cache.

  

helm repo update

  

6.   Install the cert-manager Helm chart.

  

helm install --name cert-manager --namespace cert-manager --version v0.8.0 jetstack/cert-manager

  

Generate CA Private Key and Self-signed Certificate for Kiam Agent-server TLS

1.   Generate the CRT file.

‍  

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=kiam" -out kiam.cert -days 3650 -reqexts v3_req -extensions v3_ca -out ca.crt

  

2.   Save the CA key pair as a secret in Kubernetes.

 ‍ 

kubectl create secret tls kiam-ca-key-pair \
  --cert=ca.crt \
  --key=ca.key \
  --namespace=cert-manager

  

3.   Deploy cluster issuer and issue the certificate.

     a.    Create the Kiam namespace. 

 ‍ 

apiVersion: v1
kind: Namespace
metadata:
  name: kiam
  annotations:
    iam.amazonaws.com/permitted: ".*"
---

  ‍

     b.   Deploy the cluster issuer and issue the certificate.

 ‍ 

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: kiam-ca-issuer
  namespace: kiam
spec:
  ca:
    secretName: kiam-ca-key-pair
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-agent
  namespace: kiam
spec:
  secretName: kiam-agent-tls
  issuerRef:
    name: kiam-ca-issuer
    kind: ClusterIssuer
  commonName: kiam
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kiam-server
  namespace: kiam
spec:
  secretName: kiam-server-tls
  issuerRef:
    name: kiam-ca-issuer
    kind: ClusterIssuer
  commonName: kiam
  dnsNames:
  - kiam-server
  - kiam-server:443
  - localhost
  - localhost:443
  - localhost:9610
---

‍  

4.   Test if certificates are issued correctly.

 ‍ 

kubectl -n kiam get secret kiam-agent-tls -o yaml
kubectl -n kiam get secret kiam-server-tls -o yaml

 ‍ 

Annotating Resources

1.    Add the IAM role’s name to deployment as an annotation. 

‍  

apiVersion: extensions/v1beta1
 kind: Deployment
 metadata:
 	name: mydeployment
 	namespace: default
 spec:
 ...
 	minReadySeconds: 5
 	template:
     	annotations:
       	iam.amazonaws.com/role: my-role
   	spec:
     	containers:
 ... 

 ‍ 

2.    Add role annotation to the namespace in which the pods will run. You don’t need to do this with Kube2iam. 

 ‍ 

apiVersion: v1
 kind: Namespace
 metadata:
 	name: default
 	annotations:
 		iam.amazonaws.com/permitted: ".*"

  

The default is not to allow any roles. You can use a regex as shown above to allow all roles or can even specify a particular role per namespace.

  

Deploying Kiam Agent and Server

Kiam Server

The manifest below deploys the following:

  1. Kiam Server DaemonSet which will run on Kubernetes master nodes (configure to use the TLS secret created above)
  2. Kiam Server service
  3. Service account, ClusterRole and ClusterRoleBinding required by Kiam server

‍  

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: kiam-server
  namespace: kiam
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kiam-read
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - watch
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kiam-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kiam-read
subjects:
- kind: ServiceAccount
  name: kiam-server
  namespace: kiam
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kiam-write
rules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kiam-write
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kiam-write
subjects:
- kind: ServiceAccount
  name: kiam-server
  namespace: kiam
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  namespace: kiam
  name: kiam-server
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kiam
        role: server
    spec:
      tolerations:
       - key: node-role.kubernetes.io/master
         effect: NoSchedule
      serviceAccountName: kiam-server
      nodeSelector:
        kubernetes.io/role: master
      volumes:
        - name: ssl-certs
          hostPath:
      nodeSelector:
      nodeSelector:
        kubernetes.io/role: master
      volumes:
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: tls
          secret:
            secretName: kiam-server-tls
      containers:
        - name: kiam
          image: quay.io/uswitch/kiam:b07549acf880e3a064e6679f7147d34738a8b789
          imagePullPolicy: Always
          command:
            - /kiam
          args:
            - server
            - --level=info
            - --bind=0.0.0.0:443
            - --cert=/etc/kiam/tls/tls.crt
            - --key=/etc/kiam/tls/tls.key
            - --ca=/etc/kiam/tls/ca.crt
            - --role-base-arn-autodetect
            - --assume-role-arn=<KIAM_SERVER_ROLE_ARN>
            - --sync=1m
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: ssl-certs
            - mountPath: /etc/kiam/tls
              name: tls
          livenessProbe:
            exec:
              command:
              - /kiam
              - health
              - --cert=/etc/kiam/tls/tls.crt
              - --key=/etc/kiam/tls/tls.key
              - --ca=/etc/kiam/tls/ca.crt
              - --server-address=localhost:443
              - --gateway-timeout-creation=1s
              - --timeout=5s
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /kiam
              - health
              - --cert=/etc/kiam/tls/tls.crt
              - --key=/etc/kiam/tls/tls.key
              - --ca=/etc/kiam/tls/ca.crt
              - --server-address=localhost:443
              - --gateway-timeout-creation=1s
              - --timeout=5s
            initialDelaySeconds: 3
            periodSeconds: 10
            timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: kiam-server
  namespace: kiam
spec:
  clusterIP: None
  selector:
    app: kiam
    role: server
  ports:
  - name: grpclb
    port: 443
    targetPort: 443
    protocol: TCP

 ‍ 

Note: 

  1. The scheduler toleration and node selector that we have in place here make sure that the Kiam pods get scheduled on Kiam master nodes only. This is why we enable the trust relationship between the Kiam-server IAM role and the IAM role attached to the Kubernetes master nodes (above). 

  

...
       tolerations:
       - key: node-role.kubernetes.io/master
         effect: NoSchedule 
...

  

...
      nodeSelector:
        kubernetes.io/role: master
....

  

  1. The kiam-server role ARN is provided as an argument to the Kiam server container. Make sure you update the <KIAM_SERVER_ROLE_ARN> field in the manifest above to the ARN of the role you created. 
  2. The ClusterRole and ClusterRoleBinding created for a Kiam server grant it the minimal permissions required to operate effectively. Please consider thoroughly before changing them. 
  3. Make sure the path to SSL Certs is set correctly according to the secret you created using cert-manager certificates. This is important to establish secure communication between the Kiam server and Kiam agent pods. 

  

Kiam Agent

The manifest provided below will deploy the following Kiam Agent DaemonSet which will run on Kubernetes worker nodes only:

‍  

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  namespace: kiam
  name: kiam-agent
spec:
  template:
    metadata:
      labels:
        app: kiam
        role: agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: tls
          secret:
            secretName: kiam-agent-tls
        - name: xtables
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
      containers:
        - name: kiam
          securityContext:
            capabilities:
              add: ["NET_ADMIN"]
          image: quay.io/uswitch/kiam:b07549acf880e3a064e6679f7147d34738a8b789
          imagePullPolicy: Always
          command:
            - /kiam
          args:
            - agent
            - --iptables
            - --host-interface=cali+
            - --json-log
            - --port=8181
            - --cert=/etc/kiam/tls/tls.crt
            - --key=/etc/kiam/tls/tls.key
            - --ca=/etc/kiam/tls/ca.crt
            - --server-address=kiam-server:443
            - --gateway-timeout-creation=30s
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: ssl-certs
            - mountPath: /etc/kiam/tls
              name: tls
            - mountPath: /var/run/xtables.lock
              name: xtables
          livenessProbe:
            httpGet:
              path: /ping
              port: 8181
            initialDelaySeconds: 3
            periodSeconds: 3

 ‍ 

It should be noted that Kiam agent also runs with host networking set to true, similar to Kube2iam. Also, one of the arguments to the Kiam agent’s container is the name of the Kiam service to access Kiam server, in this case  kiam-server:443 Therefore, we should deploy the Kiam server before deploying the Kiam agent. 

Also, the container argument --gateway-timeout-creation defines the waiting period for the Kiam server pod to be up before the agent tries to connect. It can be tweaked depending on how long the pod takes to come up in your Kubernetes cluster. Ideally, a thirty second waiting period is enough.

‍  

Testing

The processes for testing the Kiam and Kube2iam setups are the same. You can use a test pod and curl the metadata to check the assigned role. Please ensure that both deployment and namespace are properly annotated.

  

 ‍ 

IAM Roles for Service Accounts (IRSA)

Recently, AWS released its own service to allow pods to access AWS resources: IAM roles for Service Accounts (IRSA). Since a role is authenticated with a service account, it can be shared by all pods to which that service account gets attached. This service is available in both AWS EKS and a KOPS based installation. You can read more about it here.

  

Conclusion

The tools covered in this blog help manage access from Kubernetes pods to AWS resources, and they all have their pros and cons.

While Kube2IAM is the easiest to implement, the ease of setup compromises efficiency: Kube2iam might not perform reliably under high load conditions. It is more suited for non-production environments or scenarios that don’t experience major traffic surges.

IAM IRSA requires more work than Kube2iam, but given Amazon’s detailed documentation, it may be easier to implement. Since it is so recent, there is not enough implementation of IRSA in the industry at the time this article was written.

KIAM’s implementation needs cert-manager running and, unlike with Kube2iam, you need to annotate the namespace along with the deployment. Regardless, we highly recommend using Kiam because it can be used in all cases provided you have the resources for running cert manager, and that your master nodes are equipped to handle a DaemonSet running on them. By using the manifests provided in this post, your set-up will be seamless and production ready.

If you want to try visualizing your metrics on Grafana dashboards that are powered by Prometheus, get on to the MetricFire free trial today. Also, feel free to sign up for a demo and talk to us directly about what monitoring solutions work for you.

This article was written by our guest blogger Vaibhav Thakur. If you liked this article, check out his LinkedIn for more.

Hungry for more knowledge?

Related posts