Launch Templates and Custom Images

By default, Karpenter generates launch templates that use EKS Optimized AMI for nodes. Often, users need to customize the node image to integrate with existing infrastructure or meet compliance requirements. Karpenter supports custom node images through Launch Templates. If you need to customize the node, then you need a custom launch template.

Note: By customizing the image, you are taking responsibility for maintaining the image, including security updates. In the default configuration, Karpenter will use the latest version of the EKS optimized AMI, which is maintained by AWS.

Introduction

Karpenter follows existing AWS patterns for customizing the base image of instances. More specifically, Karpenter uses EC2 launch templates. Launch templates may specify many values. The pivotal value is the base image (AMI). Launch templates further specify many different parameters related to networking, authorization, instance type, and more.

Launch Templates and AMIs are unique to AWS regions, similar to EKS clusters. IAM resources are global.

Karpenter only implementes a subset of launch template fields, and some fields should not be set.

This guide describes requirements for using launch templates with Karpenter, and later an example procedure.

Launch Template Requirements

The Launch Template resource includes a large number of fields. AWS accepts launch templates with any subset of these fields defined.

Certain fields are obviously critical, such as AMI and User Data. Some fields are useful for particular workloads, such as storage and IAM Instance Profile.

Finally, the majority of Launch Template fields should not be set (or will have no effect), such as network interfaces and instance type.

Important Fields

When creating a custom launch template, the AMI and User Data are the defining characteristics. Instance Profile (IAM Role) and Security Group (firewall rules) are also important for Karpenter.

AMI

AMI (Amazon Machine Image), is the base image/VM for a launch template.

Review the instructions for importing a VM to AWS. Note the AMI id generated by this process, such as, ami-074cce78125f09d61.

User Data - Autoconfigure

Importantly, the AMI must support automatically connecting to a cluster based on “user data”, or a base64 encoded string passed to the instance at startup. The syntax and purpose of the user data varies between images. The Karpenter default OS, Amazon Linux 2 (AL2), accepts shell scripts (bash commands).

AWS calls data passed to an instance at launch time “user data”.

In the default configuration, Karpenter uses an EKS optimized version of AL2 and passes the hostname of the Kubernetes API server, and a certificate. The EKS Optimized AMI includes a bootstrap.sh script which connects the instance to the cluster, based on the passed data.

Alternatively, you may reference AWS’s bootstrap.sh file when building a custom base image.

#!/bin/bash
/etc/eks/bootstrap.sh <my-cluster-name> \
--kubelet-extra-args <'--max-pods=40'> \
--b64-cluster-ca <certificateAuthority> \
--apiserver-endpoint <endpoint> \
--dns-cluster-ip <serviceIpv4Cidr> \
--use-max-pods false

Note, you must populate this command with live values. Karpenter will not change the user data in the launch template.

Encode using yaml function !Base64 yaml function or cat userdata.sh | base64 > userdata-encoded.txt shell command.

Bootstrap Script Parameters

The sample bootstrap script requires information to join the cluster.

These values may be found using:

aws eks describe-cluster --name MyKarpenterCluster

Kubelet Arguments

Specifying max-pods can break Karpenter’s binpacking logic (it has no way to know what this setting is). If Karpenter attempts to pack more than this number of pods, the instance may be oversized, and additional pods will reschedule.

Situational Fields

Configure these values in response to a particular use case, such as nodes interacting with another AWS service, or using EBS storage on the node.

Instance Profile - IAM

The launch template must include an “instance profile” – an IAM role.

The instance profile must include at least the permissions of the default Karpenter node instance profile. See the default role, KarpenterNodeRole, in the full example below for more information.

See also, the managed policy “AmazonEKSWorkerNodePolicy” which includes permission to describe clusters and subnets.

Storage

Karpenter expects nothing of node storage. Configure as needed for your base image.

Security Groups - Firewall

The launch template may include a security group (i.e., instance firewall rules) and the security group must be associated with the virtual private cloud (VPC) of the EKS cluster. If none is specified, the default security group of the cluster VPC is used.

The security group must permit communication with EKS control plane. Outbound access should be permitted for at least: HTTPS on port 443, DNS (UDP and TCP) on port 53, and your subnet’s network access control list (network ACL).

Fields with Undefined Behavior

Resources referenced by these fields are controlled by EKS/Karpenter, and not the launch template.

Instance Type

The instance type should not be specified in the launch template. Karpenter will determine the launch template at run time.

Network Interfaces

The AWS CNI will configure the network interfaces. Do not configure network instances in the launch template.

Creating the Launch Template

Launch Templates may be created via the web console, the AWS CLI, or CloudFormation.

CloudFormation

The procedure, in summary, is to:

  1. Create an AMI as described in the EC2 documentation.
  2. Write a EC2 Launch Template specification including the AMI.
  3. Push the specification to AWS with CloudFormation.
  4. Update the Provisioner CRD to specify the new Launch Template.

An example yaml cloudformation definition of a launch template for Karpenter is provided below.

CloudFormation yaml is suited for the moderately high configuration density of launch templates, and creating the unusual InstanceProfile resource.

You must manually replace these values in the template:

  • SecurityGroupID
    • list all security groups with aws ec2 describe-security-groups
  • Parameters in UserData
  • AMI
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  # create InstanceProfile wrapper on NodeRole
  KarpenterNodeInstanceProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      InstanceProfileName: "KarpenterNodeInstanceProfile"
      Path: "/"
      Roles:
        - Ref: "KarpenterNodeRole"
  # create role with basic permissions for EKS node
  KarpenterNodeRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "KarpenterNodeRole"
      Path: /
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                !Sub "ec2.${AWS::URLSuffix}"
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy"
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
  MyLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties: 
      LaunchTemplateData: 
        IamInstanceProfile:
          # Get ARN of InstanceProfile defined above
          Arn: !GetAtt
            - KarpenterNodeInstanceProfile
            - Arn
        ImageId: ami-074cce78125f09d61
        # UserData is Base64 Encoded
        UserData: !Base64 > 
            #!/bin/bash
            /etc/eks/bootstrap.sh 'MyClusterName' \
            --kubelet-extra-args '--node-labels=node.k8s.aws/capacity-type=spot' \
            --b64-cluster-ca 'LS0t....0tCg==' \
            --apiserver-endpoint 'https://B0385BE29EA792E811CB5866D23C856E.gr7.us-east-2.eks.amazonaws.com' 
        BlockDeviceMappings: 
          - Ebs:
              VolumeSize: 80
              VolumeType: gp3
            DeviceName: /dev/xvda
        # The SecurityGroup must be associated with the cluster VPC
        SecurityGroupIds:
          - sg-a69adfdb
      LaunchTemplateName: KarpenterCustomLaunchTemplate

Create the Launch Template by uploading the CloudFormation yaml file. The sample yaml creates an IAM Object (InstanceProfile), so --capabilities CAPABILITY_NAMED_IAM must be indicated.

aws cloudformation create-stack \
  --stack-name KarpenterLaunchTemplateStack \
  --template-body file://$(pwd)/lt-cfn-demo.yaml \
  --capabilities CAPABILITY_NAMED_IAM

Define LaunchTemplate for Provisioner

The LaunchTemplate is ready to be used. Specify it by name in the Provisioner CRD. Karpenter will use this template when creating new instances.

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
spec:
  provider:
    launchTemplate: CustomKarpenterLaunchTemplateDemo
Last modified November 18, 2021 : change versions (f18b796)