Finding the balance between ensuring the security of user identities while providing a self-service user experience takes continuous effort. With the constantly evolving cloud landscape and the security around it, IAM user administration can become a burdensome task. In this post, I'll provide the steps to an approach that attempts to ensure the security of AWS IAM users while streamlining administrative overhead and maintaining a self-service user experience by using a combination of Terraform code and Python Boto3 scripts.

Code Functionality

The solution below is comprised of both Terraform code and Python boto3 scripts creating an efficient method for managing IAM users and their associated login profiles, MFA devices and access keys.

Terraform Code

• Manages IAM user identities, email address tags and optional custom tags (i.e. Access Key descriptions)

• aws_iam_user resource block is used with a for_each meta-argument which loops through the map of users assigned to the user_info variable

• local-exec provisioner is used to call external executable scripts which are uniquely defined based on the creation and destruction of an IAM user

• create-time provisioner creates the login profile and generates a temporary login password during user creation

• destroy-time provisioner destroys the login profile, MFA devices and access keys during user destruction

Python Boto3 Scripts

• Automatically invoked by the local-exec provisioner upon terraform apply and are unique based on the definition set in the when meta-argument

• These scripts allow users to manage their MFA devices and access keys rather than having these resources checked into the terraform state and managed by an IAM admin

• Allows IAM admins to run a single command to create or destroy a user rather than running additional commands to invoke scripts to create or destroy user resources such as the login profile, MFA devices and access keys

Code Resources

Link to the Terraform code: https://github.com/jksprattler/aws-security/blob/main/terraform/iam/users.tf

Link to the local-exec provisioner boto3 script that creates the login profile and generates the temp password for the new user: https://github.com/jksprattler/aws-security/blob/main/scripts/aws_iam_user_password_reset.py

Link to the local-exec provisioner boto3 script that destroys the login profile, MFA devices and access keys for a removed user: https://github.com/jksprattler/aws-security/blob/main/scripts/aws_iam_user_cleanup.py

Link to my blog post on IAM User Password Expiry Notice which is why I have the email tags required in this post on IAM user management: https://blog.jennasrunbooks.com/aws-lambda-function-iam-user-password-expiry-notice-ses-boto3-terraform

Usage

Create IAM User

Creating a new AWS IAM user with the provided terraform configuration requires updating the user_info variable definition by adding a new map object assignment containing the new username and the required corresponding email address key-value tag. The optional user_tags can be disregarded for now - these are only needed if the user decides to apply a description to their access key ID. The need to update these optional tags would be indicated by a future terraform plan output where the key-value tags are missing from the terraform state.

Below is some sample terraform output from creating a new IAM user called "userB":

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_user.this["userB"] will be created
  + resource "aws_iam_user" "this" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "userB"
      + path          = "/"
      + tags          = {
          + "email"            = "userB@jennasrunbooks.com"
          + "environment-type" = "lab"
          + "provisioner"      = "terraform"
          + "repo"             = "aws-security"
          + "resource-owner"   = "aws-landing-zone@jennasrunbooks.com"
        }
      + tags_all      = {
          + "email"            = "userB@jennasrunbooks.com"
          + "environment-type" = "lab"
          + "provisioner"      = "terraform"
          + "repo"             = "aws-security"
          + "resource-owner"   = "aws-landing-zone@jennasrunbooks.com"
        }
      + unique_id     = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_iam_user.this["userB"]: Creating...
aws_iam_user.this["userB"]: Provisioning with 'local-exec'...
aws_iam_user.this["userB"] (local-exec): Executing: ["/bin/sh" "-c" "python ../../scripts/aws/aws_iam_user_password_reset.py profile -u userB"]
aws_iam_user.this["userB"] (local-exec): New login profile has been created for: userB
aws_iam_user.this["userB"] (local-exec): Login with temp password:
aws_iam_user.this["userB"] (local-exec): ,)y3}"fXafTHj=Acei';
aws_iam_user.this["userB"] (local-exec): Password reset will be enforced upon initial login
aws_iam_user.this["userB"]: Creation complete after 1s [id=userB]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

As seen in the above output, the local-exec provisioner provides the temporary password to the login profile for the new user based on this line:

aws_iam_user.this["userB"] (local-exec): ,)y3}"fXafTHj=Acei';

This can now be directly provided to the new user so they can log into the AWS Console UI where they'll be forced to reset their password and then proceed to set up their MFA devices and Access keys on their own.

Destroy IAM User

Destroying an AWS IAM user is as simple as deleting their map object assignment values from the user_info variable. Additionally, you'd want to remove the user from any IAM groups however, this post is focused on IAM user management using the aws_iam_user terraform resource block.

Below is some sample terraform output from destroying the "userB" IAM user created in the previous section:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_iam_user.this["userB"] will be destroyed
  # (because key ["userB"] is not in for_each map)
  - resource "aws_iam_user" "this" {
      - arn           = "arn:aws:iam::<accountID>:user/userB" -> null
      - force_destroy = false -> null
      - id            = "userB" -> null
      - name          = "userB" -> null
      - path          = "/" -> null
      - tags          = {
          - "AKIASRJ6UGTM2MP47QO6" = "testkey2"
          - "AKIASRJ6UGTMV3JU6CU2" = "testkey1"
          - "email"                = "userB@jennasrunbooks.com"
          - "environment-type"     = "lab"
          - "provisioner"          = "terraform"
          - "repo"                 = "aws-security"
          - "resource-owner"       = "aws-landing-zone@jennasrunbooks.com"
        } -> null
      - tags_all      = {
          - "AKIASRJ6UGTM2MP47QO6" = "testkey2"
          - "AKIASRJ6UGTMV3JU6CU2" = "testkey1"
          - "email"                = "userB@jennasrunbooks.com"
          - "environment-type"     = "lab"
          - "provisioner"          = "terraform"
          - "repo"                 = "aws-security"
          - "resource-owner"       = "aws-landing-zone@jennasrunbooks.com"
        } -> null
      - unique_id     = "AIDASRJ6UGTMQUHFQ77G4" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_iam_user.this["userB"]: Destroying... [id=userB]
aws_iam_user.this["userB"]: Provisioning with 'local-exec'...
aws_iam_user.this["userB"] (local-exec): Executing: ["/bin/sh" "-c" "python ../../scripts/aws/aws_iam_user_cleanup.py userB"]
aws_iam_user.this["userB"] (local-exec): Deleting login profile for userB
aws_iam_user.this["userB"] (local-exec): Deleting MFA device for userB: arn:aws:iam::<accountID>:u2f/user/testuser/testfido-I5GXZJQHZL3FXDIXX4
aws_iam_user.this["userB"] (local-exec): Deleted access key: AKIASRJ6UGTMV3JU6CU2 for user: userB
aws_iam_user.this["userB"] (local-exec): Deleted access key: AKIASRJ6UGTM2MP47QO6 for user: userB
aws_iam_user.this["userB"]: Destruction complete after 2s
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

╷
│ Error: deleting IAM User (blahblah): DeleteConflict: Cannot delete entity, must delete login profile first.
│       status code: 409, request id: ca2c4e6e-620a-4082-9077-04af49b29bcb
│

Conclusion

By integrating Terraform and Python Boto3 scripts, the above approach attempts to provide an efficient solution for managing AWS IAM users while maintaining a secure yet self-service environment for the end user. Administrative overhead should be reduced since MFA device and access key management remain in the hands of the user and outside of the terraform state. As long as you have good documentation and your users are savvy enough managing MFA devices and access keys should not be an issue for them. The Terraform code also follows the DRY principle to reduce repetitive lines of code with a single aws_iam_user resource block used.

Managing a global fleet of virtual machines (VMs) across various cloud service providers (CSPs) can be a challenging task. Depending on the naming convention applied to your hostnames, quickly identifying the CSP and region for a VM could be a challenge. To simplify this process, I've developed a bash script that takes advantage of each supported CSPs' available Instance Metadata Service (IMDS) to effectively identify the CSP and region, making troubleshooting more efficient.

Shout out to my colleague, Aaron Fujimoto who gave me the idea and an initial framework for this as discussed in his blog post here!

Purpose

At Kentik, we manage hundreds of global agents hosted across more than a dozen different CSPs for our Synthetics product. The hostnames assigned to these VMs follow a naming convention that identifies the CSP using an assigned numerical value and region using an IATA code. While the intent for this was to provide standardization across all environments, it's difficult to speedily identify what CSP and/or region you might be dealing with upon receiving an alert for a specific host. There are methods for identifying this information such as searching Kibana logs or matching up the CSP numerical identifier and IATA code for the region which has been my use case. To streamline this identification process, I've developed this simple tool supporting 9 different CSPs, including Azure, GCP, AWS, Alibaba Cloud, Vultr/Choopa, DigitalOcean, Tencent Cloud, Exoscale, and IBM Cloud.

Script Functionality

This script utilizes a simple yet effective combination of Linux tools including curl, jq, and grep. By extracting the CSP and region details, the script enables faster navigation within the CSPs' portal and any other internal applications or tools you might use allowing you to pinpoint the affected VM within the appropriate region. For example, when troubleshooting an issue on an AWS host you need to know the region where it's hosted so you can navigate to the appropriate region's page within the AWS portal.

Link to the script: https://github.com/jksprattler/multi-cloud-tools/blob/main/scripts/get-cloud-provider-region.sh

Usage

Using the script is straightforward and is intended to be run by the root user after being loaded into the /usr/local/sbin directory on all cloud hosts across an infrastructure. Once executed, it cycles through a list of if/else statements until it matches the conditional for the CSPs' metadata the VM is running on. Next, it identifies the region based on the matching CSP statement.

Below are some sample test results from running the script across the 9 different supported CSPs in my environment. You can see how these example hostnames make it difficult to identify the CSP/region:

root@dev-2-mxp:~# get-cloud-provider-region.sh
CSP: aws
Region: eu-south-1
root@dev-12-cdg:~# get-cloud-provider-region.sh
CSP: azure
Region: FranceCentral
root@dev-5-ams:~# get-cloud-provider-region.sh
CSP: digitalocean
Region: ams3
root@dev-9-hkg:~# get-cloud-provider-region.sh
CSP: alibabacloud
Region: cn-hongkong
root@dev-1-ams:~# get-cloud-provider-region.sh
CSP: gcp
Region: europe-west4-b
root@dev-4-atl:~# get-cloud-provider-region.sh
CSP: vultr/choopa
Region: ATL US
root@dev-10-bkk:~# get-cloud-provider-region.sh
CSP: tencentcloud
Region: ap-bangkok
root@dev-11-fra:~# get-cloud-provider-region.sh
CSP: exoscale
Region: de-fra-1
root@dev-7-dfw:~# get-cloud-provider-region.sh
CSP: ibmcloud
Region: us-south-1

Conclusion

Managing VMs across various CSPs and regions is now much more efficient using this script. By combining curl, jq, and grep Linux tools, this solution leads to faster troubleshooting. The ability to quickly identify the CSP and region of a VM, right from within the host, showcases the potential of leveraging simple tools like this for enhanced operational efficiency.

🔎 Quickly find AWS S3 Objects inside buckets hosting huge volumes of files using my latest Boto3 script! You can easily locate specific objects in your AWS profile by providing a few command line arguments. By specifying the AWS profile, bucket name, object name, and optional prefix you can efficiently search for objects and obtain a list of matching keys, allowing you to pinpoint the location of a particular file.

This 📜 performs the following tasks:

✔️ Handle pagination when dealing with large buckets containing thousands of objects

✔️ Perform a recursive search for an object in an S3 bucket, considering the object's name and optional prefix

✔️ Display the keys of matching objects found in the bucket

✔️ List all buckets in a specified AWS profile

✔️ List top-level prefixes in a specified bucket

Purpose

This AWS S3 Object Finder script is particularly helpful when troubleshooting upload error logs especially when dealing with buckets that host thousands of objects and extensive prefixes. In these scenarios, manually searching for objects can be very time-consuming. By utilizing the script's pagination and object prefix features, you can significantly speed up the search process and narrow down the results to the desired subset of objects. This saves both time and effort allowing you to efficiently troubleshoot specific object data without having to search for objects manually.

Script Functionality

Link to the script: https://github.com/jksprattler/aws-tools/blob/main/scripts/aws_s3_object_finder.py

Usage

Here's a look at the options available from the -h/--help output:

❯ python aws_s3_object_finder.py --help
usage: aws_s3_object_finder.py [-h] -p PROFILE [-b BUCKET] [-o OBJECT] [-x PREFIX] {list-buckets,list-prefixes} ...

Search for an object in an S3 bucket.

positional arguments:
  {list-buckets,list-prefixes}
                        Additional commands
    list-buckets        List buckets in profile, use after -p
    list-prefixes       List top-level prefixes in bucket, use after -p and -b

optional arguments:
  -h, --help            show this help message and exit
  -p PROFILE, --profile PROFILE
                        AWS profile name
  -b BUCKET, --bucket BUCKET
                        S3 bucket name
  -o OBJECT, --object OBJECT
                        Object name to search for
  -x PREFIX, --prefix PREFIX
                        Object prefix for high volume bucket search

Here's an example of listing all buckets in a specified AWS profile:

❯ python aws_s3_object_finder.py -p cloud-sandbox list
sandbox-cloudtrail
sandbox-trail-bucket
lstest2022

Here's an example of listing all top-level prefixes in a specified bucket:

❯ python aws_s3_object_finder.py -p default -b jennasprattler.com list-prefixes
assets/
images/

Here's an example of listing all S3 objects in a specified bucket by calling a keyword in the prefix as the object:

❯ python aws_s3_object_finder.py -p cloud-sandbox -b sandbox-cloudtrail -o us-west-2
'us-west-2' found in bucket 'sandbox-cloudtrail'
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_HzrxfJgKaRrQr2d3.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_Uyg59kXzzIB0y2YS.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_rzKX1qISHRXL6aru.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2105Z_GlZuUn6pUrCDzbhr.json.gz
### Redacted for brevity

Here's an example of listing all objects in a bucket under a specific prefix where there are 1,000s of objects requiring pagination. If you don't know the specific file/object name then you can define the closest matching prefix anywhere in the key as the object argument followed by the initial prefix of the key as the prefix argument:

❯ python aws_s3_object_finder.py -p cloud-sandbox -b sandbox-cloudtrail -o us-west-2 -x AWSLogs/
'us-west-2' found in bucket 'sandbox-cloudtrail'
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_HzrxfJgKaRrQr2d3.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_Uyg59kXzzIB0y2YS.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_rzKX1qISHRXL6aru.json.gz
### Redacted for brevity

In some cases, you will need to be more granular with how you specify objects and prefixes based on your prefix structure. In other words, you could have duplicate or overlapping prefix names causing a search to fail. The solution for this is to specify additional values in your delimiter-separated values for '/' for the prefix argument:

❯ python aws_s3_object_finder.py -p cloud-sandbox -b sandbox-cloudtrail -o us-west-2 -x AWSLogs/accountId/CloudTrail/
'us-west-2' found in bucket 'sandbox-cloudtrail'
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_HzrxfJgKaRrQr2d3.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_Uyg59kXzzIB0y2YS.json.gz
Key(s) matching 'us-west-2': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_rzKX1qISHRXL6aru.json.gz
### Redacted for brevity

Here's an example of finding a specific object/filename in a bucket hosting thousands of objects by specifying a prefix with the object argument:

❯ python aws_s3_object_finder.py -p cloud-sandbox -b sandbox-cloudtrail -o accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz -x AWSLogs/
'accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz' found in bucket 'sandbox-cloudtrail'
Key(s) matching 'accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz': sandbox-cloudtrail/AWSLogs/accountId/CloudTrail/us-west-2/2022/01/20/accountId_CloudTrail_us-west-2_20220120T2100Z_1bxLMBY0uRL1X67x.json.gz

Code Explanation

Inside the main function, the args are set to the parse_args() function. If no arguments are provided the script outputs the usage from the parser and exits. The next conditional will run the list_buckets function if the list-buckets argument is provided with a profile. Next, it will run the list_prefixes function if the list-prefixes argument is provided with a profile and bucket. Finally, the search_s3_object function is run when at least the profile, bucket and object are provided - the prefix is optional.

def main():
    """ Parse args and call either S3 bucket list or object search function """
    args = parse_args()

    if not args:
        return

    if args.command == 'list-buckets':
        list_buckets(args.profile)
    elif args.command == 'list-prefixes':
        list_prefixes(args.profile, args.bucket)
    else:
        search_s3_object(args.profile, args.bucket, args.object, args.prefix)

if __name__ == "__main__":
    main()

Below is the parse_arguments() function. This is self-explanatory by reviewing the help comments for each of the arguments. Also, see usage in the output example above.

def parse_args():
    """ Define cli args to be parsed into main() """
    parser = argparse.ArgumentParser(description='Search for an object in an S3 bucket.')
    parser.add_argument('-p', '--profile', required=True, help='AWS profile name')
    parser.add_argument('-b', '--bucket', help='S3 bucket name')
    parser.add_argument('-o', '--object', help='Object name to search for')
    parser.add_argument('-x', '--prefix', help='Object prefix for high volume bucket search')
    subparser = parser.add_subparsers(dest='command', help='Additional commands')
    subparser.add_parser('list-buckets',
                         help='List buckets in profile, use after -p')
    subparser.add_parser('list-prefixes', help='List top-level prefixes in bucket,\
                                                use after -p and -b')
    args = parser.parse_args()

    if args.command == 'list-buckets':
        return args

    if args.command == 'list-prefixes' and args.bucket:
        return args

    if not args.bucket or (not args.bucket and not args.command == 'list-prefixes') \
        or not args.object:
        parser.print_usage()
        return None

    return args

If the user provides the list-buckets argument then the list_buckets(profile_name) function is called and provides the output of all buckets in the specified AWS profile. It creates a session using the provided profile name, establishes an S3 client using the session, and sends a request to list the buckets associated with the profile.

def list_buckets(profile_name):
    """ List buckets in specified profile """
    session = boto3.Session(profile_name=profile_name)
    s3_client = session.client('s3')
    buckets = s3_client.list_buckets()['Buckets']

    for bucket in buckets:
        print(bucket['Name'])

If the user provides the list-prefixes argument then the list_prefixes(profile_name, bucket_name) function is called and provides the output of all top-level prefixes within the specified bucket.

def list_prefixes(profile_name, bucket_name):
    """ List top-level prefixes in a specified bucket """
    session = boto3.Session(profile_name=profile_name)
    s3_client = session.client('s3')
    paginator = s3_client.get_paginator('list_objects_v2')
    list_objects_args = {'Bucket': bucket_name, 'Delimiter': '/'}
    page_iterator = paginator.paginate(**list_objects_args)

    prefixes = []
    for page in page_iterator:
        if 'CommonPrefixes' in page:
            # Iterate over the CommonPrefixes list and extract top-level prefixes
            prefixes.extend([prefix['Prefix'] for prefix in page['CommonPrefixes']])

    if prefixes:
        for prefix in prefixes:
            print(prefix)
    else:
        print(f"No top-level prefixes found in bucket '{bucket_name}'")

Excluding any of the "list*" arguments, when the user provides the --object and/or --prefix argument with the AWS profile then the search_s3_object(profile_name, bucket_name, object_name, object_prefix) function is called. It creates a session using the specified profile name, creates an S3 client using the session, and sends a request to list the objects in the specified bucket. The response is then processed, filtering the objects based on the provided object name and object prefix. If any matching objects are found, their keys are printed to the console, indicating their location within the bucket. If no matching objects are found, a message is printed to indicate that the object was not found in the bucket.

def search_s3_object(profile_name, bucket_name, object_name, object_prefix):
    """
    Create a session using the specified profile
    Create s3 client using the session
    Perform recursive search using pagination for the S3 bucket object
    Use optional prefix when dealing w/ high volume objects
    """
    session = boto3.Session(profile_name=profile_name)
    s3_client = session.client('s3')
    paginator = s3_client.get_paginator('list_objects_v2')
    list_objects_args = {'Bucket': bucket_name}

    if object_prefix:
        list_objects_ars['Prefix'] = object_prefix

    page_iterator = paginator.paginate(**list_objects_args)

    for page in page_iterator:
        if 'Contents' in page:
            # Check if the object name is in the retrieved keys
            objects = [obj['Key'] for obj in page['Contents'] if object_name in obj['Key']]

            if objects:
                print(f"'{object_name}' found in bucket '{bucket_name}'")
                for obj in objects:
                    print(f"Key(s) matching '{object_name}': {bucket_name}/{obj}")
                return

    print(f"'{object_name}' not found in bucket '{bucket_name}'")

A few items to highlight from the search_s3_object() function:

• list_objects_args = {'Bucket': bucket_name}: This line creates a dictionary called list_objects_args with a key-value pair. The key is 'Bucket', and the value is the bucket_name variable. This dictionary is used to specify the bucket name when calling the list_objects_v2 API.

• if object_prefix: list_objects_args['Prefix'] = object_prefix: This conditional statement checks if the object_prefix variable has a value. If object_prefix is not empty or None, it means a prefix was provided. In that case, it adds another key-value pair to the list_objects_args dictionary. The key is 'Prefix', and the value is the object_prefix variable. This prefix specifies a filter to narrow down the objects returned by the API call.

• page_iterator = paginator.paginate(**list_objects_args): This line creates a page_iterator object using the paginator.paginate() method. The double asterisks ** before list_objects_args unpacks the dictionary into keyword arguments. This means that the paginator.paginate() method receives the dictionary key-value pairs as separate arguments. In this case, it passes the 'Bucket' and 'Prefix' arguments to the paginator.paginate() method.

• These provide the necessary arguments for the paginator.paginate() method based on the provided bucket name and optional object prefix. It allows you to paginate through the S3 bucket objects, retrieving a subset of objects at a time, based on the provided arguments while still offering flexibility in that the --prefix argument is not required to run the script although it will still be needed if your bucket consists of thousands of objects.

Conclusion

This approach to searching for S3 bucket objects provides quick troubleshooting capability and enables you to locate data with ease. Whether you need to identify a precise object location or navigate through large buckets with complex prefixes, this AWS Object Finder script can be a helpful tool when working with AWS S3!

Boto3 Resources

• list_objects_v2 - Boto3 1.26.151 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_objects_v2.html

• list_buckets - Boto3 1.26.151 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_buckets.html

• Paginators - Boto3 1.26.151 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/paginators.html

Perform checkouts against your AWS Kubernetes cluster before and after implementing changes such as upgrades by using my latest Bash shell script! I took my AKS checkouts script from a couple of weeks ago and converted it into an EKS checkouts script 😎

This script performs the following tasks:

✅ Sets the kubeconfig context based on the provided cluster name (names must match)

✅ Display addresses of the master and services

✅ Checks the k8's version running on both the control plane and the nodes

✅ Checks supported EKS versions

✅ Lists all nodes and their status, age and version

✅ Display custom columns for pods with node, pod, namespace, status, and age

✅ Output count for total # pods running

✅ Gets all pods in the cluster, loops through each and runs a verbose describe outputting values from Node to Status

Link to the script: https://github.com/jksprattler/kubernetes/blob/main/eks/scripts/eks-checkouts.sh

Usage & Sample Output

The script output can be useful during instances where you need to submit a vendor ticket for troubleshooting. I'll also be including the output in my internal ticket tracking upgrades to validate it was completed.

Here's some sanitized sample output with redactions for brevity from running the script:

❯ ./eks-checkouts.sh --help
Usage: ./eks-checkouts.sh <cluster-name> <region> <aws-profile>

❯ ./eks-checkouts.sh my-cluster my-region my-profile
Switched to context "my-cluster".
########################################################
Kubernetes control plane is running at https://my-cluster.gr7.my-region.eks.amazonaws.com
CoreDNS is running at https://my-cluster.gr7.my-region.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
########################################################
Control Plane Version: 
Client Version: v1.23.6
Server Version: v1.23.17-eks-0a21954
########################################################
Cluster Version: v1.23.16-eks-48e63af
Cluster Status: ACTIVE
########################################################
Available EKS versions:
1.22
1.23
1.24
1.25
1.26
1.27
########################################################
NAME                                         STATUS   ROLES    AGE   VERSION
ip-my-node.my-region.compute.internal   Ready    <none>   73d   v1.23.16-eks-48e63af
ip-my-node.my-region.compute.internal   Ready    <none>   73d   v1.23.16-eks-48e63af
#### Redacted for brevity ####
########################################################
NODE                                         POD                                             NAMESPACE      STATUS    AGE
ip-my-node.my-region.compute.internal   aws-node-zs123                                  kube-system    Running   2023-03-24T18:35:15Z
#### Redacted for brevity ####
########################################################
Total number of pods running:      163
########################################################
Checking pod: cert-manager/cert-manager-12345678-45svg
Node:         ip-my-node.my-region.compute.internal/ip-my-node
Start Time:   Thu, 16 Mar 2023 06:25:19 -0500
Labels:       app=cert-manager
              app.kubernetes.io/component=controller
              app.kubernetes.io/instance=cert-manager
              app.kubernetes.io/name=cert-manager
              app.kubernetes.io/version=v1.8.2
              pod-template-hash=12345678
Annotations:  kubernetes.io/psp: eks.privileged
              prometheus.io/path: /metrics
              prometheus.io/port: 9090
              prometheus.io/scrape: true
Status:       Running
########################################################
#### Redacted for brevity ####

Since I'm performing upgrades to my EKS clusters this week, I decided to put this script together and share it! Enjoy!

Capture a list of available AWS AMIs for a specified region and OS version with my latest boto3 script. This Python script will output a color-coded table listing AMIs that have been released within the last 30 days with the owner alias set to Amazon. The table includes columns for AMI ID, Name, Architecture type and Virtualization type. Output is sorted by Architecture type (x86_64, arm64, etc), then sorted by AMI names which include release dates.

Sample output against us-west-2 region for a list of debian-11* AMI's:

Purpose

For my particular use case, I have a dev environment where AMIs for various OS versions and instance types need to be built for testing purposes. I also have a fleet of production instances that all run the same OS version and machine type using a custom AWS Terraform module. I'm still able to use this same Terraform module for my dev infrastructure however, I've created a unique variable to be able to assign a specific AMI to certain dev resources.  The user_specified_ami var is what I'm using to assign my AMIs.  I've applied the coalesce function to the "ami" resource which forces this value to return as precedent over the next sequential data.aws_ami source as long as there's a string assigned to this variable. It looks like this in the aws_instance resource block of the module:

ami = coalesce(var.user_specified_ami, data.aws_ami.prod_os.id)

Then in my configuration module, I can simply assign the user_specified_ami an ID from the script output.

Script Functionality

Link to the script: https://github.com/jksprattler/aws-tools/blob/main/scripts/aws_ec2_find_amis.py

Usage

Here's a look at the options available from the -h/--help output:

❯ python aws_ec2_find_amis.py --help
usage: aws_ec2_find_amis.py [-h] region os_version

Find AMIs based on region and OS

positional arguments:
  region      AWS region
  os_version  OS version

options:
  -h, --help  show this help message and exit

Here's an example of entering an OS version not set in the valid_os_versions list of elements:

❯ python aws_ec2_find_amis.py us-west-2 debian
Invalid OS version 'debian'. Available OS versions:
amzn2
debian-11
ubuntu-jammy
Windows_Server-2019-English

Here's an example of entering an invalid AWS region:

❯ python aws_ec2_find_amis.py us-west debian-11
Invalid region 'us-west'. Available regions:
af-south-1
ap-south-1
eu-north-1
eu-west-3
eu-south-1
eu-west-2
eu-west-1
ap-northeast-3
ap-northeast-2
me-south-1
ap-northeast-1
ca-central-1
sa-east-1
ap-east-1
ap-southeast-1
ap-southeast-2
eu-central-1
ap-southeast-3
us-east-1
us-east-2
us-west-1
us-west-2

Code Explanation

Inside the main function, it calls the parse_arguments() function to parse the user input arguments and assigns them to the region and os_version variables. See the above help output for example.

def main():
    """
    Parse user arguments, assign region and os values
    Lookup latest available AMI's and output them to color-coded table
    """
    region, os_version = parse_arguments()
    validate_inputs(region, os_version)
    find_amis(region, os_version)

Here's the parse_arguments() function :

def parse_arguments():
    """ Parse args """
    parser = argparse.ArgumentParser(description='Find AMIs based on region and OS')
    parser.add_argument('region', type=str, help='AWS region')
    parser.add_argument('os_version', type=str, help='OS version')
    args = parser.parse_args()
    return args.region, args.os_version

Next, it checks if the provided inputs for the region and os_version arguments are valid with the validate_inputs(region, os_version) function.

def validate_inputs(region, os_version):
    """ Validate region and OS version if provided"""
    if region:
        available_regions = get_available_regions()
        if region not in available_regions:
            print(f"Invalid region '{region}'. Available regions:")
            for regions in available_regions:
                print(regions)
            sys.exit()

    if os_version:
        validate_os_version(os_version)

This will perform a conditional check for the region input against the get_available_regions() function which performs an API call using the EC2 client to describe the AWS regions:

def get_available_regions():
    """
    Get a list of available AWS regions
    """
    ec2_client = boto3.client('ec2')
    response = ec2_client.describe_regions()
    regions = [region['RegionName'] for region in response['Regions']]
    return regions

Then it performs a conditional check for the os_version input against the validate_os_version(os_version) function which checks if the provided versions starts with any of the elements in the valid_os_versions list:

def validate_os_version(os_version):
    """ Check if the provided OS version is valid """
    valid_os_versions = ['amzn2', 'debian-11', 'ubuntu-jammy', 'Windows_Server-2019-English']
    for version in valid_os_versions:
        if os_version.startswith(version):
            return
    print(f"Invalid OS version '{os_version}'. Available OS versions:")
    for version in valid_os_versions:
        print(version)
    sys.exit()

Finally, the find_amis(region, os_version) function is executed. The API of the EC2 client is called to describe images hosted in the specified region and filtered based on the provided OS version and AMI alias owner, Amazon:

def find_amis(region, os_version):
    """
    Output a list of the latest Amazon owned AMI's for the specified
    region and OS version in color-coded Table format
    """

    ec2_client = boto3.client('ec2', region_name=region)

    response = ec2_client.describe_images(
        Filters=[
            {'Name': 'name', 'Values': ['*'+os_version+'*']},
            {'Name': 'owner-alias', 'Values': ['amazon']}
        ]
    )

    amis = response['Images']

    # Sort the AMIs by architecture type and name
    sorted_amis = sorted(amis, key=lambda x: (x['Architecture'], x['Name']))
    latest_amis = []
    for ami in sorted_amis:
        creation_date = datetime.datetime.strptime(ami['CreationDate'], "%Y-%m-%dT%H:%M:%S.%fZ")
        if creation_date >= DATE_THRESHOLD:
            latest_amis.append(ami)

    table = []
    table_headers = ["AMI ID", "Name", "Architecture", "VirtualizationType"]
    for ami in latest_amis:
        table.append([
            f"{Fore.CYAN}{ami['ImageId']}{Style.RESET_ALL}",
            ami['Name'],
            f"{Fore.GREEN}{ami['Architecture']}{Style.RESET_ALL}",
            ami['VirtualizationType']
        ])
    print(tabulate(table, headers=table_headers, tablefmt="grid"))

The response contains a list of AMIS matching these filters and is sorted by the Architecture type. Further sorting is done by AMI name which contains release dates. Only AMIs with release dates in the last 30 days are provided.

Next, a table is generated by creating an empty list called table to store the AMI information for tabulation. As the script iterates over the sorted AMIs, the requested data for the AMI ID, Name, Architecture and Virtualization Type are appended to the table list. The ImageID and the Architecture values have color formatting applied using the Fore class from the colorama library. I added this feature to help with the readability of the table output.

Conclusion

Now I'm able to get these AMI values much more quickly with this handy script rather than crafting an aws ec2 describe-images command and parsing through the output. Hope you found this useful! What kind of use cases do you have for capturing and assigning AMIs?

In this implementation, you'll be guided through the necessary steps to set up an AWS Lambda function to email notifications to IAM Users when their AWS Web Console passwords are expiring. The function is written in Python (boto3) and integrated with AWS SES using a verified domain. Terraform code samples are provided for all of the infrastructure configuration steps.

The purpose of this initiative is to provide proactive security measures while streamlining administrative tasks by sending an advanced notice to AWS IAM users that their passwords are expiring. By promptly alerting users before their passwords reach the expiration threshold defined in the security policy, you can facilitate regular password resets, ensuring stronger security measures are maintained. This approach reduces the administrative burden on administrators, as users can self-reset their passwords in advance of expiration. Additionally, it empowers users to take ownership of their account security, promoting a culture of proactive password management.

With this implementation, you will enhance the overall security, efficiency, and user experience within your AWS IAM environment.

Pre-requisites

• Domain ownership and access to manage DNS records

  • Domain jennasrunbooks.com will be used as an example in the below sample code output. Update this with your appropriate domain.

• Terraform installed

• Install Python, boto3 and other script modules as needed to review the code locally without errors

• Clone repo containing Lambda Python script

• AWS IAM permissions to deploy Terraform resources

• Email tags assigned to each AWS IAM User, sample Terraform code from a terraform.tfvars file:

  user_names = {
    "user1" = {
      "name" = "user1",
      "tag"  = { email = "user1@jennasrunbooks.com", role = "engineering" }
    }
  }
  

Procedure

Set up and verify the email domain

• Register and verify your email domain in the Amazon SES console to establish your email sender identity.

• If you'd like to use a custom MAIL FROM domain, check out this AWS doc and note the following:

  Important

  To successfully set up a custom MAIL FROM domain with Amazon SES, you must publish exactly one MX record to the DNS server of your MAIL FROM domain. If the MAIL FROM domain has multiple MX records, the custom MAIL FROM setup with Amazon SES will fail.

• Sample Terraform code to deploy an AWS SES domain identity using Easy DKIM settings and a custom MAIL FROM domain:

  # Update the domain variable to your domain
  # using the SES service which is integrated into other services ie Lambda
  
  variable "domain" {
    type    = string
    default = "jennasrunbooks.com"
  }
  
  resource "aws_ses_domain_identity" "ses_domain" {
    domain = var.domain
  }
  
  resource "aws_ses_domain_mail_from" "main" {
    domain           = aws_ses_domain_identity.ses_domain.domain
    mail_from_domain = "mail.${var.domain}"
  }
  
  resource "aws_ses_domain_identity_verification" "email_identity_verification" {
    domain = aws_ses_domain_identity.ses_domain.domain
  }
  

  • The initial terraform apply will likely timeout on the aws_ses_domain_identity_verification resource depending on how quickly your DNS provider can propagate the changes to verify the domain identity with AWS. You can simply rerun an apply to update the state once the domain has been verified.

  • Update your DNS provider records for your domain with the DKIM CNAME and MAIL FROM MX and SPF records provided by AWS as shown in this example after running terraform apply:

    

  • You have the option to download a csv of each record set which can be useful if another team or 3rd party manages the records for your domain.

  • DNS propagation can take anywhere from a few minutes to several hours. If entered correctly, the SES page for the domain verification will display the identity status as Verified. Once verified, you'll also receive an email from AWS for each configuration type you've configured.

Write the Lambda function Python script

• Link to the script: https://github.com/jksprattler/aws-security/blob/main/lambda/password_notification/password_notification.py

• Only IAM users with valid email tags using the verified domain in their address will be notified.

• Update the SES email body contents with an appropriate message for your audience.

• Include a direct link to the account's AWS console sign-in page. If you have multiple accounts used in your infrastructure, it can also be helpful to specify to your clients which account the password reset is for by providing both the friendly name and account# in the message body.

• Update the Source email account in the ses_client.send_email method as appropriate to your cloud admins team Distribution List.

Configure the Lambda IAM policy and role

• Create an IAM policy that includes the necessary permissions for the Lambda function to generate and get the credential report, send emails using SES, and access user tags (we specifically need the email tags for each user).

• Give the policy permissions to create the log group, and log streams and push log events to the streams. This will be used by CloudWatch when it's triggered on the scheduled event.

• Create an IAM role and attach the IAM policy to it. This role will be assumed by the Lambda function to access the required AWS services.

• Sample Terraform code to create the IAM policy, role and policy attachment needed for the lambda function:

  # Update Resource ARN references to region, accountID and aws lambda log-group name as needed for your environment
  
  resource "aws_iam_policy" "lambda_password_notification_policy" {
    name   = "lambda_password_notification_policy"
    policy = <<EOF
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "iam:GenerateCredentialReport",
              "ses:SendEmail",
              "ses:SendRawEmail",
              "iam:GetCredentialReport",
              "iam:ListUserTags"
            ],
            "Resource": "*"
          },
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogGroup"
            ],
            "Resource": "arn:aws:logs:region:accountID:*"
          }
        ]
      },
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:region:accountID:log-group:/aws/lambda/password_notification:*"
          }
        ]
      }
      EOF
    tags   = var.common_tags
  }
  
  resource "aws_iam_role" "lambda_password_notification_role" {
    name               = "lambda_password_notification_role"
    assume_role_policy = <<EOF
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
              "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }
      EOF
  }
  
  resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
    role       = aws_iam_role.lambda_password_notification_role.name
    policy_arn = aws_iam_policy.lambda_password_notification_policy.arn
  }
  

Setup the Lambda function

• Create the new Lambda function with Python (boto3) to generate SES email notifications of expired passwords by generating and parsing the IAM credentials report using Terraform to deploy the resources.

• Python 3.8 will be used for the runtime

• The handler will use the name of the function, ie: password_notification.lambda_handler

• The previously created IAM role will be assigned to the Lambda function

• The Python script will be zipped using the Terraform archive_file data source

• The source_code_hash argument will be applied to the aws_lambda_function resource block to trigger updates in the terraform state when changes are made to the Python script.

• Sample Terraform code:

  # Update below file paths according to your directory architecture
  
  data "aws_lambda_function" "password_notification" {
    function_name = aws_lambda_function.password_notification.function_name
  }
  
  data "archive_file" "password_notification_zip" {
    type        = "zip"
    source_dir  = "src_dir/scripts/aws/lambda/password_notification/"
    output_path = "dest_dir/scripts/aws/lambda/password_notification/password_notification.zip"
  }
  
  resource "aws_lambda_function" "password_notification" {
    filename         = "dest_dir/scripts/aws/lambda/password_notification/password_notification.zip"
    source_code_hash = data.archive_file.password_notification_zip.output_base64sha256
    function_name    = "password_notification"
    description      = "lambda function to send email notifications (SES) to users when passwords are expiring"
    role             = aws_iam_role.lambda_password_notification_role.arn
    handler          = "password_notification.lambda_handler"
    runtime          = "python3.8"
    timeout          = 180
    memory_size      = 128
    depends_on = [
      aws_iam_role_policy_attachment.lambda_policy_attachment,
      aws_cloudwatch_log_group.password_notification_log_group,
    ]
  }
  

Test the Lambda function

• Test the Lambda function manually by triggering it with sample input data to ensure it sends the correct email notifications.

• Use the existing lambda function deployed previously or create a new one using the same settings, IAM role, etc for this test.

• Use the Lambda console to manually test the function.

• Example updated Lambda Python to force a test email notification using custom JSON event values:

  """
  Jenna Sprattler | SRE Kentik | 2023-05-21
  Test lambda function to send email notifications (SES) to specified test user that passwords are expiring
  """
  import time
  from datetime import datetime
  from dateutil import parser
  import boto3
  
  iam_client = boto3.client('iam')
  ses_client = boto3.client('ses')
  
  def lambda_handler(event, context):
      username = event['username']
      password_last_changed = event['password_last_changed']
      email = event['email']
  
      if password_last_changed not in ('N/A', 'not_supported'):
          # Parse the date and time components from the timestamp
          password_last_changed_date = parser.parse(password_last_changed)
          days_since_password_change = (
              datetime.now() - password_last_changed_date.replace(tzinfo=None)).days
          if days_since_password_change > 78:
              message = f'''
                  <html>
                  <body>
                  <p>Hello {username},</p>
                  <p>Your password to access the <a href="https://signin.aws.amazon.com/console">AWS web console</a> has expired or will be expiring within the next 12 days.</p>
                  <p>If your password is still valid, please log into the web console and follow the banner instructions to reset your password now.</p>
                  <p>If you have API access keys configured, you can use the Password Reset Self Service <a href="https://github.com/jksprattler/aws-security/blob/main/scripts/aws_iam_self_service_password_reset.py">script</a>.</p>
                  <p>If you don't use API keys, and your password has passed the expiration date then reply to this email and we'll assist with your password reset.</p>
                  <p>If you don't use your AWS account, reply to this email and we'll remove your account.</p>
                  <br>
                      <p>Thank you,</p>
                      <p>Cloud Admins</p>
                  </body>
                  </html>
                  '''
              ses_client.send_email(
                  Source='cloud-admins@jennasrunbooks.com',
                  Destination={'ToAddresses': [email]},
                  Message={
                      'Subject': {'Data': 'AWS Password Expiry Notification'},
                      'Body': {'Html': {'Data': message}}
                  }
              )
      return 'Password expiry notifications sent to: ' + username
  

  • Add appropriate JSON test event values to your variables and generate a test SES email notification, for example:

    {
      "username": "user1",
      "password_last_changed": "2023-02-10T00:00:00Z",
      "email": "user1@jennasrunbooks.com"
    }
    

  • Validate that the test email was received by the test user's email account. Review the message subject and body contents for accuracy. Sample test email to my AWS IAM user1:

    

  • Feel free to use the Password Reset Self-Service script referenced in the above email body.

  • For users with passwords passed the expiration date, check out my Administrative user password reset script.

Setup the CloudWatch event rule

• Create the CloudWatch Event Rule to trigger the desired schedule for automating the password expiration notification Lambda function which contains the SES send_email operation to your users with expiring passwords.

• Set the target of the CloudWatch Event Rule to the Lambda function you created.

• (Optional) Create the log group for the CloudWatch event log streams. This is created automatically whenever a new Lambda function is deployed however, if/when you destroy your Lambda function in the future, Terraform will not automatically delete the log group that was created. If the log group resource is managed by the Terraform state then you will know to delete that resource and have cleaner infrastructure should the Lambda function be removed in the future.

• There is a hidden Terraform resource called aws_lambda_permission that needs to be created. This is created automatically when a Lambda function is created using the AWS console. This allows the CloudWatch events permissions to access the Lambda function. This is needed for the CloudWatch event rule (cron scheduler), log streams, etc to work properly with the function.

• For testing purposes, you could update the schedule_expression of the event rule to run every 10 min while confirming that the Lambda function and Cloudwatch settings are working properly then update to a daily interval: rate(10 minutes)

• Sample Terraform code:

  resource "aws_lambda_permission" "allow_cloudwatch_for_password_notification" {
    statement_id  = "AllowExecutionFromCloudWatch"
    action        = "lambda:InvokeFunction"
    function_name = aws_lambda_function.password_notification.function_name
    principal     = "events.amazonaws.com"
    source_arn    = aws_cloudwatch_event_rule.password_notification_schedule.arn
  }
  
  resource "aws_cloudwatch_log_group" "password_notification_log_group" {
    name              = "/aws/lambda/password_notification"
    retention_in_days = 7
  }
  
  resource "aws_cloudwatch_event_rule" "password_notification_schedule" {
    name                = "password_notification_schedule"
    description         = "Scheduled rule for password notification"
    schedule_expression = "cron(0 13 * * ? *)" # Schedule for 8am CST / 1pm UTC daily (adjust as needed)
  }
  
  resource "aws_cloudwatch_event_target" "password_notification_target" {
    rule = aws_cloudwatch_event_rule.password_notification_schedule.name
    arn  = data.aws_lambda_function.password_notification.arn
  }
  

Validate functionality

• Monitor the CloudWatch logs in the designated Lambda function's log group and check for any errors or unexpected behavior. Example of a successful run:

  

• Verify that the Lambda function is generating the credential report and sending the email notifications to the intended recipients.

• You should see an IAM credentials report created shortly after the Lambda function's scheduled event, for example:

  

• You can download this report and sort by the password_last_changed column to capture a list of users that should have received the SES notification. Reach out to these users to confirm whether or not they received the email notification.

Conclusion

This implementation provides a comprehensive solution for notifying AWS IAM users of their upcoming password expirations for accessing the AWS web console. With this proactive approach to security, password hygiene is prioritized while also helping reduce the risk of compromised credentials.

Empowering users to self-reset their passwords before expiration not only enhances account security but also streamlines administrative tasks. This implementation showcases the value of leveraging automation and user-centric design to create a more secure and efficient AWS IAM environment. With this solution, organizations can foster a culture of proactive password management.

Resources

• Domain Identities

  • Verified identities in Amazon SES - Amazon Simple Email Service: https://docs.aws.amazon.com/ses/latest/dg/verify-addresses-and-domains.html

  • Using a custom MAIL FROM domain - Amazon Simple Email Service: https://docs.aws.amazon.com/ses/latest/dg/mail-from.html

• Lambda Function

  • aws_lambda_function | Resources | hashicorp/aws | Terraform Registry: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function#iam-role

  • Lambda execution role - AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html

  • Getting started with Lambda - AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html

  • Building Lambda functions with Python - AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html

  • Testing Lambda functions in the console - AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/testing-functions.html

  • Monitoring and troubleshooting Lambda functions - AWS Lambda: https://docs.aws.amazon.com/lambda/latest/dg/lambda-monitoring.html

• CloudWatch Events

  • aws_cloudwatch_event_rule | Resources | hashicorp/aws | Terraform Registry: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule

  • Schedule Expressions for Rules - Amazon CloudWatch Events: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html

• Boto3

  • generate_credential_report - Boto3 1.26.137 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/generate_credential_report.html

  • get_credential_report - Boto3 1.26.137 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/get_credential_report.html

  • list_user_tags - Boto3 1.26.137 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/list_user_tags.html

  • send_email - Boto3 1.26.137 documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ses/client/send_email.html

Perform checkouts against your Azure Kubernetes cluster before and after implementing changes such as upgrades by using this Bash shell script I wrote.

This script provides output for the following checks:

✅ Sets the kubeconfig context based on the provided cluster name (names must match)

✅ Display addresses of the master and services

✅ Checks running version of control master and available upgrade path

✅ Checks the kubernetes version running on both the control plane and the node pools

✅ Lists all node pools in the cluster

✅ Lists all nodes running across both pools in the cluster

✅ Output count for total # pods running

✅ Gets all pods in the cluster, loops through each and runs a verbose describe outputting values from Node to Status

Link to the script: https://github.com/jksprattler/kubernetes/blob/main/aks/scripts/aks-checkouts.sh

Usage & Sample Output

The script output can be useful during instances where you need to submit a vendor ticket for troubleshooting. I'll also be including the output in my internal ticket tracking the upgrade to validate it was completed.

Here's some sanitized sample output with redactions for brevity from running the script:

❯ ./aks-checkouts.sh my-rg my-cluster
Switched to context "my-cluster".
########################################################
Kubernetes control plane is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443
CoreDNS is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://my-cluster-00000000.hcp.westus2.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
########################################################
Name     ResourceGroup    MasterVersion    Upgrades
-------  ---------------  ---------------  -----------------------
default  my-rg        1.24.9           1.24.10, 1.25.5, 1.25.6
########################################################
Name                  Location    ResourceGroup    KubernetesVersion    CurrentKubernetesVersion    ProvisioningState    Fqdn
--------------------  ----------  ---------------  -------------------  --------------------------  -------------------  ---------------------------------------------------
my-cluster  westus2     my-rg        1.24.9               1.24.9                      Succeeded            my-cluster-00000000.hcp.westus2.azmk8s.io
########################################################
Name    OsType    KubernetesVersion    VmSize            Count    MaxPods    ProvisioningState    Mode
------  --------  -------------------  ----------------  -------  ---------  -------------------  ------
user  Linux     1.24.9               Standard_DS13_v2  11       30         Succeeded            User
system  Linux     1.24.9               Standard_DS2_v2   3        30         Succeeded            System
########################################################
NAME                             STATUS   ROLES   AGE   VERSION
aks-user-99999999-vmss000000   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss000005   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss000006   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss000007   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000a   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000d   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000k   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000o   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000p   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000q   Ready    agent   20h   v1.24.9
aks-user-99999999-vmss00000s   Ready    agent   20h   v1.24.9
aks-system-99999999-vmss000000   Ready    agent   21h   v1.24.9
aks-system-99999999-vmss000001   Ready    agent   21h   v1.24.9
aks-system-99999999-vmss000002   Ready    agent   21h   v1.24.9
########################################################
NODE                             POD                                                      NAMESPACE     STATUS    AGE
aks-user-99999999-vmss000000   ama-logs-abcde                                           kube-system   Running   2023-05-18T15:19:14Z
aks-user-99999999-vmss000005   ama-logs-fghij                                           kube-system   Running   2023-05-18T15:46:55Z
aks-system-99999999-vmss000000   ama-logs-klmno                                           kube-system   Running   2023-05-18T14:42:46Z
aks-system-99999999-vmss000001   ama-logs-pqrst                                           kube-system   Running   2023-05-18T14:46:15Z
#### Redacted for brevity ####
########################################################
Total number of pods running:      217
########################################################
Checking pod: botkube/botkube-99999999-abcde
Node:         aks-user-99999999-vmss000000/1.1.1.1
Start Time:   Thu, 18 May 2023 10:24:44 -0500
Labels:       app=botkube
              component=controller
              pod-template-hash=99999999
Annotations:  <none>
Status:       Running
########################################################
Checking pod: default/filebeat-fghij
Node:         aks-user-99999999-vmss000007/1.1.1.2
Start Time:   Thu, 18 May 2023 10:34:15 -0500
Labels:       app=filebeat
              controller-revision-hash=99999999
              pod-template-generation=3
Annotations:  kubectl.kubernetes.io/restartedAt: 2023-04-13T09:36:26-10:00
Status:       Running
########################################################
#### Redacted for brevity ####

Hope you found this useful! What kinds of tools do you use in your Kubernetes environment for running checkouts against changes like upgrades?

Rotate your AWS API access keys using this Boto3 script for IAM users that I wrote. Optional arguments include create, update, delete, and list access keys. When applying the create new 🔑 feature, the user is prompted whether or not to overwrite their existing ~/.aws/credentials file contents with the [default] profile, newly created access key ID and secret. If they enter 'n', the new key is still created however, the user will need to update their credentials file manually with the new key ID and secret when ready to use it locally.

Although this AWS blog post is 10 years old, I still find it to be the most succinct procedure explaining how to rotate your access keys. Highlighting the general steps to follow from the post below which should be considered when using this script:

To rotate access keys, you should follow these steps:

1. Create a second access key in addition to the one in use.

2. Update all your applications to use the new access key and validate applications are working.

3. Change the state of the previous access key to inactive.

4. Validate that your applications are still working as expected.

5. Delete the inactive access key.

Purpose

AWS access keys should be treated like any other security credentials you use daily. Just as it's best practice to change your password regularly, the same treatment should be applied to your access keys. Regularly rotating your keys helps mitigate the potential risk of a security breach occurring within your cloud estate should your keys become compromised.

Script Functionality

Link to script: https://github.com/jksprattler/aws-security/blob/main/scripts/aws_iam_self_service_key_rotation.py

Here's a look at the options available from the -h/--help output:

❯ python aws_iam_self_service_key_rotation.py -h
usage: aws_iam_self_service_key_rotation.py [-h] [-c] [-u UPDATE UPDATE] [-d DELETE] [-l]

create|update|delete|list API access keys

optional arguments:
  -h, --help            show this help message and exit
  -c, --create          create access key
  -u UPDATE UPDATE, --update UPDATE UPDATE
                        update access key; 2 expected inputs: <keyID> <active|inactive>
  -d DELETE, --delete DELETE
                        delete access key; 1 expected input: <keyID>
  -l, --list            list access key ID, status, creation date

Here's a list of my current access key(s):

❯ python aws_iam_self_service_key_rotation.py -l
Key:        ****************4NOP
Status:     Active
Created:    2023-02-19 14:51:31+00:00

Following the general steps outlined in the Overview section above, I'll go ahead and create a new access key since I only have 1 currently and there's a limit of 2 you can have at one time:

❯ python aws_iam_self_service_key_rotation.py -c
New Key:        ****************QJLD
Secret:         ****************Epm2

Metadata response output:
{'AccessKey': {'UserName': 'jslabs', 'AccessKeyId': '****************QJLD', 'Status': 'Active', 'SecretAccessKey': '****************Epm2', 'CreateDate': datetime.datetime(2023, 4, 13, 20, 58, 23, tzinfo=tzutc())}, 'ResponseMetadata': {'RequestId': '12345678-79aa-4f83-a17e-cee94d4d51f2', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '12345678-79aa-4f83-a17e-cee94d4d51f2', 'content-type': 'text/xml', 'content-length': '598', 'date': 'Thu, 13 Apr 2023 20:58:22 GMT'}, 'RetryAttempts': 0}} 

Update /home/jenna/.aws/credentials file w/ new key (y/n)? Warning: File contents will be overwritten! y

Updated /home/jenna/.aws/credentials file contents:
[default]
aws_access_key_id = ****************QJLD
aws_secret_access_key = ****************Epm2

At this point, you should update all of your applications to use the new key. Update your ~/.aws/credentials file manually if needed or run aws configure

Next, we'll update the status of the old key to inactive and check that the status has been updated:

❯ python aws_iam_self_service_key_rotation.py -u ****************4NOP inactive
Metadata response output:
{'ResponseMetadata': {'RequestId': '12345678-5823-4038-be27-63b574b41359', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '12345678-5823-4038-be27-63b574b41359', 'content-type': 'text/xml', 'content-length': '210', 'date': 'Thu, 13 Apr 2023 21:04:12 GMT'}, 'RetryAttempts': 0}}
❯ python aws_iam_self_service_key_rotation.py -l
Key:        ****************QJLD
Status:     Active
Created:    2023-04-13 20:58:23+00:00

Key:        ****************4NOP
Status:     Inactive
Created:    2023-02-19 14:51:31+00:00

Once we've validated our applications are working as expected with the new key, we can proceed with deleting the old/inactive key and validate we're left with the 1 active/in-use key:

❯ python aws_iam_self_service_key_rotation.py -d ****************4NOP
****************4NOP has been deleted!

Metadata response output:
{'ResponseMetadata': {'RequestId': '12345678-21d2-4f42-b0ff-9e26d4ff5a02', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '12345678-21d2-4f42-b0ff-9e26d4ff5a02', 'content-type': 'text/xml', 'content-length': '210', 'date': 'Thu, 13 Apr 2023 21:08:19 GMT'}, 'RetryAttempts': 0}}
❯ python aws_iam_self_service_key_rotation.py -l
Key:        ****************QJLD
Status:     Active
Created:    2023-04-13 20:58:23+00:00

Conclusion

You now have a self-service tool for your AWS IAM users to use for rotating their access keys on an as-needed basis. Although there are more automated approaches out there for AWS key rotation, I like the option of giving users some control over how and when to update their keys.

Hope you've found this useful! Let me know in the comments what you think and how this script could be improved. As an extension to this tool, I'm looking into creating a Lambda function that automatically expires access keys older than 90 days and includes an 📨 notification to users that their keys are expiring via SES so stay tuned for that!

Create GCP compute commitments across an entire project running a fleet of globally distributed resources using my latest bash script. The script is configured to purchase 12-month committed resources based on multiple required input values including project-id, machine-type, commitment-type, # of vcpu's and amount of memory. Ideally, I would have preferred to implement this using Terraform as that is how this environment is managed. While the google_compute_reservation module is available, there is no module for compute commitments at the time of this writing which is what's required for this scenario. I didn't need to configure compute reservations so that capability will not be covered here.

Script Functionality

Link to script: gcp_committed_resources.sh

Based on the machine-type value entered  (i.e. e2-standard-2), the script will loop through all instances across your project running that machine type and create unique-named commitments for each instance using the following format: <machine-type>-<region>-<random#>,  example: e2-standard-2-us-west4-19289. This prevents duplicate commitment names in regions where you might be running multiple instances of the same machine type. The intent is to have each created commitment represent an individual VM's resources from within the residing region.

The script features a dry-run function that allows you to see the results of your input values and how they will be applied to the creation of commitments. Examples of potential conditionals against some inputs (i.e. Commitment Type, vcpu's and memory) can be uncommented and set for your use case. This can prevent allocating too many resources for a particular commitment whether this be a user input error (i.e. fat finger) or an oversight on what's being used by an existing machine type.

The Bash getopts built-in command parses options and enforces input arguments on these flags when a dry run or commitment creation is run: -p project-id, -m machine-type, -t commitment=type, -v vcpus, and -M memory. A dry run can be performed by running these options after passing any set conditionals. The changes will be applied and commitments created by running the dry run values in addition to appending the -G option which is set to dry_run=0 resulting in the implementation of the create_commitments function.

Additional flags are available to assist with capturing details needed for the dry run and subsequent commitment creation. Here's the output of the usage function by invoking the -h (--help flag or null options can be applied for the same output):

./gcp_committed_resources.sh -h
################################################################################
Option descriptions:
-p            Sets project-id
-m            Sets machine type of current running instances for commitments. Double check as gcloud filter is not exact match!
-t            Sets Commitment Type, for available options run -T
-v            Sets the amount of vcpu's for each committed resource per zone
-M            Sets the amount of Memory for each committed resource per zone, default is GB if not specified
-G            Caution! Creates the commitments. Always perform a dry run first without this option!
-P            Lists all existing project-id's to select from
-L            Lists all instances in set project and includes machine type, vm name & zone
-T            Lists possible commitment type options, for details see: https://cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts#commitment_types
-C            Lists current compute commitments
Usage:        [-p project-id] [-m machine-type] [-t commitment-type] [-v vcpus] [-M memory] [-G create-commitments]
Dry run ex.:  ./gcp_committed_resources.sh -p my-project -m e2-standard-2 -t general-purpose-e2 -v 2 -M 8
Creation ex.: ./gcp_committed_resources.sh -p my-project -m e2-standard-2 -t general-purpose-e2 -v 2 -M 8 -G
NOTE:         Check Quotas on Commitments against your regions w/ Limit: 0 vcpu's and submit requests for increases here or you'll encounter gcloud crashed (TypeError)

I've successfully run this script from one of the GCP projects in my environment, purchasing commitments for 40x globally distributed compute instances running two different machine types. Below are the details included in the pre-work analysis steps I performed and a guide with some example outputs from my implementation.

Pre-Work Analysis

Cost

According to Google Cloud docs:

When you purchase vCPUs and/or memory on a 1-year commitment, you get the resources at a discount of 37% over the on-demand prices

This is in alignment with the GCP calculator estimates I ran in a cost savings analysis against my environment. Based on this, my company should see substantial enough savings 💰 for these 1-year commitments associated with 40x instances on two different machine types that are running consistently 24/7/365 to make this a worthwhile implementation.

I recommend performing a similar cost analysis exercise in your environment if you're pursuing purchasing compute commitments. If you're managing multiple GCP projects with a central billing account, then you might be interested in my blog post on performing a GCP BigQuery Expression to capture monthly invoice costs based on labels assigned to resources to assist in your cost analysis.

Google also has a Recommender tool that evaluates the usage of your VM's over 30 days to determine whether or not they're eligible for committed use discounts (CUD) by checking the following:

• The VM was active for the entire duration of the 30 days.

• The VM's SKU is part of an eligible committed use discount bucket.

• The VM's usage was not already covered by an existing commitment.

Machine Type

1. Set your project-id: ./gcp_committed_resources.sh -p my-project

2. Determine the machine type used by your compute instances in a specified project by running the script with the -L flag.

   • Get all machines sorted by machine type:  ./gcp_committed_resources.sh -L

   • Grep for a list of specific machine types: ./gcp_committed_resources.sh -L | grep e2-standard-2

3. Once you have the machine types listed that you want to purchase committed resources for, cross-check them against the GCP list of commitment types that are eligible for commitments.

4. If your machine type is eligible for commitments, then review the page associated with your specific compute engine family to capture the # of vpcu's and amount of memory for that machine type. For example, the e2-standard-2 are in the General purpose machines doc with the following specs:

Note: If the majority of your compute instances are running the same machine type in the same region, then this script may not be well suited for your use case. In that scenario, I would probably calculate the total vcpu's and memory needed for a particular commitment type and create a single commitment in that region running a the gcloud compute commitments create command once.

Commitment & CPU Quota Limits

Before proceeding with creating the new commitments, I highly recommend reviewing both the Commitment and CPU Quota limits on the regions you're working with. I didn't do this step during my pre-work and was affected by a generic gcloud error that broke my while loop while running the script against my second set of machine types which looked like this:

###########################################
Commitment Name:  e2-standard-2-asia-south2-22948
ERROR: gcloud crashed (TypeError): expected string or bytes-like object
		If you would like to report this issue, please run the following command:
  gcloud feedback
		To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

Searching a few posts online led me to discover it was a commitment quota limit affecting a handful of regions in my deployment. Here are a couple references to Google docs that can help you avoid the gcloud crash that I ran into:

• View your quota in the Google Cloud console doc

• Resource usage quotas and permission management doc

Cross-reference the ./gcp_committed_resources.sh -L captured in the Machine Type section against the quota limits set in your regions. If needed, submit a request for quota increases that will cover the number of commitments and/or resources (ie CPU) you'll be creating for that region. If you have multiple instances of the same machine type in the same region, be sure to request quota increases to reflect that.

Here's an example of what a commitment quota limit increase response from Google looks like after submitting the request using the cloud console:

Your quota request for my-project has been approved and your project quota has been adjusted according to the following requested limits:
		
		+-------------+--------------------------------+-------------------------+-----------------+----------------+
		| NAME        | DIMENSIONS                     | REGION                  | REQUESTED LIMIT | APPROVED LIMIT |
		+-------------+--------------------------------+-------------------------+-----------------+----------------+
		| COMMITMENTS | region=asia-south2             | asia-south2             | 2               | 2              |
		|             |                                |                         |                 |                |
		| COMMITMENTS | region=europe-central2         | europe-central2         | 2               | 2              |
		|             |                                |                         |                 |                |
		| COMMITMENTS | region=northamerica-northeast2 | northamerica-northeast2 | 2               | 2              |
		|             |                                |                         |                 |                |
		| COMMITMENTS | region=southamerica-west1      | southamerica-west1      | 2               | 2              |
		+-------------+--------------------------------+-------------------------+-----------------+----------------+
		
After approved, Quotas can take up to 15 min to be fully visible in the Cloud Console and available to you.

It took about 20 minutes for the change to appear for me in the Quota service of the cloud console.

Here's a screenshot of what it looked like after the quota limits were increased and I had successfully created commitments for those affected regions:

Implementation

Purchasing commitments without attached reservations is the Google doc I followed for my scenario which includes the required permissions for applying these changes.

1. Ensure to run all the "Lists" options defined in the Usage as part of your pre-work analysis before the implementation. These should be used to help build out your dry run results followed by the actual implementation with the -G flag set.

2. Perform a dry run with the following options set: [-p project-id] [-m machine-type] [-t commitment-type] [-v vcpus] [-M memory]

   • Example dry run:

     ❯ ./gcp_committed_resources.sh -p my-project -m e2-standard-2 -t general-purpose-e2 -v 2 -M 8
     Updated property [core/project].
     Project set to: my-project
     ###########################################
     Commitment Name:  e2-standard-2-asia-east1-6065
     Region:           asia-east1
     Project:          my-project
     Resources:        vcpu=2,memory=8
     Plan:             12-month
     Type:             general-purpose-e2
     ###########################################
     

3. Perform the implementation of the create commitments function by appending the -G flag to your dry run options: [-p project-id] [-m machine-type] [-t commitment-type] [-v vcpus] [-M memory] [-G create-commitments]

   • Example successful commitment creation run:

     ❯ ./gcp_committed_resources.sh -p my-project -m e2-standard-2 -t general-purpose-e2 -v 2 -M 8 -G
     Updated property [core/project].
     Project set to: my-project
     ###########################################
     Commitment Name:  e2-standard-2-asia-east1-28472
     Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-east1/commitments/e2-standard-2-asia-east1-28472].
     ---
     autoRenew: false
     category: MACHINE
     creationTimestamp: '2023-03-31T07:56:32.947-07:00'
     endTimestamp: '2024-04-01T00:00:00.000-07:00'
     id: 'my-id'
     kind: compute#commitment
     name: e2-standard-2-asia-east1-28472
     plan: TWELVE_MONTH
     region: https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-east1
     resources:
     - amount: '2'
       type: VCPU
     - amount: '8192'
       type: MEMORY
     selfLink: https://www.googleapis.com/compute/v1/projects/my-project/regions/asia-east1/commitments/e2-standard-2-asia-east1-28472
     startTimestamp: '2023-04-01T00:00:00.000-07:00'
     status: NOT_YET_ACTIVE
     statusMessage: The commitment is not yet active (its startTimestamp is in the future).
       It will not apply to current resource usage.
     type: GENERAL_PURPOSE_E2
     ###########################################
     

4. As soon as the committed resources have been purchased they'll display as "Pending" in the cloud portal and "NOT_YET_ACTIVE" from the gcloud cli.

   • Example output of current commitments list using -C flag immediately post-change:

     ❯ ./gcp_committed_resources.sh -C
     NAME                                        REGION                   END_TIMESTAMP                  STATUS
     e2-standard-2-us-central1-7087              us-central1              2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     e2-standard-2-europe-west1-13832            europe-west1             2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     e2-standard-2-us-west1-21069                us-west1                 2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     e2-highcpu-8-asia-east1-14956               asia-east1               2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     e2-highcpu-8-asia-east1-20966               asia-east1               2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     e2-standard-2-asia-east1-28472              asia-east1               2024-04-01T00:00:00.000-07:00  NOT_YET_ACTIVE
     

   • Commitments will move into an Active state at midnight the following day per Google doc:

     After purchasing a commitment, the commitment is effective starting at midnight the following day. For example, a commitment purchased on Monday afternoon at 3 PM US and Canadian Pacific Time (UTC-8, or UTC-7 during daylight saving time) becomes effective on Tuesday at 12 AM US and Canadian Pacific Time (UTC-8 or UTC-7). The discounts are automatically applied to applicable instances in the region you specified, and to the projects in which those discounts are purchased.

   • Output using -C flag 24+ hours post-change:

     ❯ ./gcp_committed_resources.sh -C
     NAME                                        REGION                   END_TIMESTAMP                  STATUS
     e2-standard-2-us-central1-7087              us-central1              2024-04-01T00:00:00.000-07:00  ACTIVE
     e2-standard-2-europe-west1-13832            europe-west1             2024-04-01T00:00:00.000-07:00  ACTIVE
     e2-standard-2-us-west1-21069                us-west1                 2024-04-01T00:00:00.000-07:00  ACTIVE
     e2-highcpu-8-asia-east1-14956               asia-east1               2024-04-01T00:00:00.000-07:00  ACTIVE
     e2-highcpu-8-asia-east1-20966               asia-east1               2024-04-01T00:00:00.000-07:00  ACTIVE
     e2-standard-2-asia-east1-28472              asia-east1               2024-04-01T00:00:00.000-07:00  ACTIVE
     

5. If you happen to run into a gcloud error such as the one referenced in the Commitment & CPU Quota Limits section above, you won't be able to re-run the script as you'll end up creating duplicate commitments for resources you've already purchased commitments for creating unintended additional cloud spend 💸 I've created another script for these scenarios with _manual appended to the name here: gcp_committed_resources_manual.sh

   • The script performs a one-time loop through an array of defined regions that missed getting commitments created.

   • The region array/list, PROJECT_ID, vcpu, memory, and commitment type all need to be hard coded for your use case.

   • If you have duplicate machine types hosted in any of the regions listed in the array, you'll need to account for that. You can rerun the script to create commitments in those scenarios since commitment names are appended with the $RANDOM internal Bash function.

   • Perform a dry run with: ./gcp_committed_resources_manual.sh

   • Perform the implementation with: ./gcp_committed_resources_manual.sh --go

❗️❗️Warning ❗️❗️ Extra caution should be taken using these scripts as you cannot back out of purchasing committed resources. When creating GCP commitments you are agreeing to pay for the resources assigned to your commitments under the length of time defined in the plan, in this case, 12 months.

Conclusion

I've successfully run these scripts from one of the GCP projects in my environment, purchasing commitments for 40x globally distributed compute instances running two different machine types. This should result in a discount of 37% over on-demand pricing resulting in worthwhile savings on resources running in the project.

While running the script to create commitments for the 2nd set of machine types I encountered the gcloud error as mentioned in the Commitment & CPU Quota Limits section above. I was able to successfully create commitments for these remaining regions using the secondary Manual script referenced in Step 5. of the Implementation section. To avoid this in the future, I'll be taking a closer look at Quota limits for my regions so I can run the primary script once through.

At the expiration of a commitment plan, you could renew your commitments with a simple bash script that loops through all the existing commitments capturing the Commitment Name and associated region using the gcloud compute commitments list command or using the -L flag in the script and apply these values to the gcloud compute commitments update $COMMITMENT_NAME --auto-renew --region=$region command referenced here.

This article was originally published in January 2023 on my GitHub io blog here

Overview

After publishing my initial runbook exploring this topic, I decided to test out and implement the HashiCorp for_each meta-argument method for managing the Azure AD User base of a production environment I'm currently working with. I wanted to share my findings from that experience here. In this 2nd part of my series on Azure AD & RBAC with Terraform, I define the requirements necessary for setting up Azure AD User and Group administration using this alternative method. I've also highlighted some tips from issues I ran into during my implementation of this and a Validation section containing helpful commands for post-checkouts and troubleshooting. While the initial runbook and demo in this series still have useful information such as a deep dive into the security behind Azure AD and RBAC and some test login scenarios, I've found this CSV file method to be much more efficient at managing users and group membership. Also, I've refined the security privileges around the GitHub Actions SPN and have implemented the creation and management of the SPN, including its API permissions, using Terraform resource blocks.

Topics Covered

• GH Actions SPN creation and permission configurations are done in Terraform

• GH Actions SPN gets further locked down with reduced privileges from Subscription Owner to Reader and from User Admin to specific API Reader level permissions

• Azure AD Users managed by a Terraform for_each meta-argument using data populated into a CSV file

• Azure AD Group members assigned using Terraform for_each meta-argument against users assigned to a specific Department

Pre-requisites

• See Part 1 runbook: Azure AD & RBAC with Terraform

• Ensure Security defaults are disabled to enable the creation of a Conditional Access policy.

  • From the Portal UI navigate to Azure AD > properties > manage security defaults > enable security defaults toggle to no.

• Not required: if you're interested in how I got the highlighted changes in my GH Actions bot PR comments see my runbook: Silence "Refreshing state…" & Highlight Changes in Github Actions Terraform Plan Output

Design Artifacts

• YouTube demo recording

• Clone/Fork the repo containing the Terraform artifacts for the Azure AD config here: jksprattler/azure-security

  • Relevant files are under /azuread-users-groups-roles-pt2

• Config for the GH Actions SPN, storage account and blob container hosting the terraform backend state files is under /azure-dev-infra

Procedure

1. Authenticate to Azure and configure local environment variables to run terraform commands from your local terminal:

   az login
   export ARM_SUBSCRIPTION_ID=$(az account show --query id | xargs)
   export ARM_ACCESS_KEY="<insert storage account access key used for backend state configs>"
   

   For enterprise environments with multiple subscriptions, run the following to get the specific SubID you need: az account list --query '[].{SubID:id, SubName:name}' -o table

2. Create the gh-actions-runbooks-ad SPN and its required API permissions configured in the apps.tf file under the azure-dev-infra directory. This will be used to perform the terraform plan output in the GitHub PR comments. This config resides under this directory since the SPN is required for all other infrastructure plan outputs in the other directories of the repo.

3. Once created, apply the "Grant admin consent for Default Directory" to the API permissions for the SPN by running the az CLI command: az ad app permission admin-consent --id <app_id value from outputs>

4. Assign the SPN outputs as GitHub Secrets in your repo using the following variables:

   ARM_SUBSCRIPTION_ID="<sub_id>"
   ARM_TENANT_ID="<tenant_id>"
   ARM_CLIENT_ID="<app_id>"
   ARM_CLIENT_SECRET="<auth_client_secret>"
   

   Tip

   In the previous runbook, step 4. mentioned assigning the SPN the User Admin AD role however, I found this is not required with the API Permissions configured to allow AD User, Group and Domain Reader access. The Subscription Owner RBAC role is not needed since we can assign the Subscription Reader RBAC role as we just need it to be allowed to read the data during the terraform plan.

5. Populate the users.csv file with users. If you're importing an existing Azure AD user base into Terraform, navigate in the Portal UI to Azure AD > Users : Download users to capture a csv file of existing users. Extract the user data from the existing CSV file and populate the users.csv Terraform file with the required fields: first_name,last_name,mail_nickname,preferred_language

   Tip

   The usage_location value is required for users that are assigned Microsoft licenses such as O365

6. Assign the users to a department such as Art or Engineering if you'd like to auto-assign them to the Azure AD Groups created in this lab. The Art Group uses Dynamic Membership to assign users requiring the Azure AD Premium P1 license. The Art group also includes a Conditional Access policy assignment enforcing MFA into the portal for all users in the group. The Engineering group resource block includes a for_each argument which loops through all users and for each user assigned to the Engineering department it assigns their membership to the Engineering group. The Engineering group does not require any Azure AD Premium licenses and is a nice option for automating group membership when you want to keep costs down and don't require the use of conditional access policies.

7. Save and commit the changes. Review the github-actions bot output in your PR comments which uses the gh-actions-runbooks-ad SPN created in Step 2. Make adjustments to your code as needed.

8. Perform a manual terraform apply from your local directory.

   Tip

   Terraform will assign each Azure AD user resource a unique ID using the mail_nickname value set in the users.csv file. This identifier can be used for assigning Azure AD group owner/membership. User Principal Names are used for the Azure Portal login username and will require a password reset upon initial login. The new user will be created with an auto-generated password using the following pattern in all lowercase letters in a single string: lastname + first letter of first name + numerical value for length of first name + !123 I added chars 123 as I found the Microsoft password length requirement was not met on users with last names less than 5 chars long. This should fix that issue.

9. To delete users, remove the entire line entry for that user from the users.csv file and be sure to remove any user identifiers (i.e., azuread_user.users["userarose"]) manually assigned to Azure AD groups.

Validations

• List resources managed by Terraform: terraform state list

• Show AD user info: terraform state show 'azuread_user.users["userarose"]'

• List all Azure AD users: az ad user list --query "[].{name:displayName,userPrincipalName:userPrincipalName, ObjectID:id}" -o tsv

• List the 2 groups that were created: az ad group list --query "[?contains(displayName,'Engineering')].{ name: displayName }" -o tsv

• List the users in the groups: az ad group member list --group "Engineering" --query "[].{ name: displayName }" -o tsv

Conclusion

You now have a much more efficient method for managing Azure AD Users using the CSV file and Terraform for_each meta-arguments. Azure AD Group membership is dynamically configured using for_each meta-arguments against department names assigned to users which excludes the requirement for purchasing Azure AD Premium P1 licenses. The SPN used by the GitHub Actions workflow is further locked down to adhere to the principal of least privileges and it's configured in the Terraform code. Having your AD users, groups and SPN's configured as code allows for consistency of settings and permissions across your infrastructure. It also increases a level of security awareness since PR's will require review/approval for code changes.

This query performs the following functions:

• The sum of regular costs, taxes, adjustments and rounding errors for each invoice month using inner CAST

• The CROSS JOIN statement excludes Null arrays/rows and UNNEST operator flattens the array of labels into rows

• Filter by project name

• Group by key/value labels and invoice month

• Order by invoice month

This is just one scenario displaying the value of applying tags/labels on your cloud resources. I've found tagging essential when it comes to cloud cost analysis, especially across a multi-cloud, multi-platform infrastructure!

Link to query on Gist: https://gist.github.com/jksprattler/80519ade571714c4415e103ccc8ad2be

Here are some use cases for enabling VPC flow logs:

• Network monitoring: diagnostics and capacity forecasting

• Network traffic optimization expenses: inter-region and zone traffic analysis

• Network forensics: examine compromised IPs

• Real-time security analysis: SIEM integration

Link to script: https://github.com/jksprattler/gcp-networking/blob/main/scripts/enable-flowlogs.sh

This article was originally published in September 2022 on my GitHub io blog here

Overview

The purpose of this runbook is to demonstrate a potential approach to managing Azure AD users, groups and Role-Based Access Control (RBAC) by following Terraform's declarative model with automated checkouts using GitHub Actions CI/CD Workflows. Both the Azure AD and Azure RM Terraform providers will be used to implement Identity & Access Management as code which will allow for automated provisioning of users, groups, custom role definitions, role assignments and conditional access policies. The principle of least privilege will be followed in the delegation of both Azure AD and Azure RBAC assignments. The procedure will use a combination of AZ CLI, Terraform, python scripts and Azure Portal UI. Users will be assigned to AD Groups dynamically based on their department (ie Art, Engineering). A conditional access policy will be created to demonstrate Multi-factor authentication (MFA) enforcement based on dynamic group assignment. An Azure Service Principal (SPN) will be used to perform the automated GitHub Actions workflow jobs based on code commits generating terraform checks and plan output. Finally, Azure AD self-service password reset (SSPR) will be enabled for all users.

Azure AD & RBAC Topics Covered

• Azure AD identity governance of users and groups (dynamic)

• Azure RBAC group assignment using both built-in and custom roles

• Azure RBAC assignments scoped to dynamic Azure AD groups

• Azure AD Conditional Access Policy enforcing Multi-factor authentication (MFA) for Users based on dynamic Group membership

• Azure Service Principal (SPN) to provision terraform configuration via GitHub Actions automation

• Azure AD self-service password reset (SSPR)

Pre-requisites

• GitHub Account

• Microsoft Account

• Azure Subscription

• Azure AD Premium P1/P2 license required for configuring: Conditional Access policy, MFA with conditional access, Dynamic groups

  • A free trial can be activated from within your Azure AD tenant for a limited time

• Azure CLI installed and credentials for authentication

• Terraform installed

• Azure RG, Storage account and blob container setup if you choose to maintain terraform state files remotely.

Design Artifacts

• YouTube demo recording

• Clone/Fork the repo containing the Terraform artifacts for the Azure AD config here: jksprattler/azure-security

  • Relevant files are under /azuread-users-groups-roles

  • Config for the storage account and blob container hosting the terraform backend state files is under /azure-dev-infra

Diagram

Procedure

1. Authenticate to Azure and configure local environment variables to run terraform commands from your local terminal:

   az login
   export ARM_SUBSCRIPTION_ID=$(az account show --query id | xargs)
   export ARM_ACCESS_KEY="<insert storage account access key used for backend state configs>"
   

2. Create an Azure SPN assigning the RBAC role of owner at the subscription level so it has privileges to both manage all resources and assign RBAC roles to other users. This identity will be used for provisioning the terraform configuration using the GH Actions workflows:

   az ad sp create-for-rbac --name "gh-actions-runbooks-ad" --role owner \
                         --scopes /subscriptions/{subscription-id} \
                         --sdk-auth                        
   # Replace {subscription-id} with the subscription details
   # The command should output a JSON object similar to this:
   {
     "clientId": "<GUID>",
     "clientSecret": "<GUID>",
     "subscriptionId": "<GUID>",
     "tenantId": "<GUID>",
     (...)
   }
   

3. Assign the output of the JSON objects as GitHub Secrets in your repo using the following variables:

   ARM_SUBSCRIPTION_ID="<azure_subscription_id>"
   ARM_TENANT_ID="<azure_subscription_tenant_id>"
   ARM_CLIENT_ID="<service_principal_appid>"
   ARM_CLIENT_SECRET="<service_principal_password>"
   

4. Assign the User administrator Azure AD role to the SPN. From Portal UI navigate to Azure AD > Roles and Administrators blade > "User Administrator" Role > Add Assignments > Select members > Filter by service principal display name.

   As of this writing I couldn’t find an efficient CLI method for applying Azure AD roles to SPN’s as Azure CLI is unsupported and Powershell cmdlets, which are still in preview mode, gave errors leaving the portal as the best option

5. Assign the API permissions from the below screenshot to the SPN to allow read/write access to the conditional access policy. From Portal UI navigate to App registrations > locate and select your SPN > API permissions: Add permission and be sure to select "Grant admin consent for Default Directory" once all the Application type API permissions have been added.

   

6. Disable Security defaults to enable the creation of a Conditional Access policy. From the Portal UI navigate to Azure AD > properties > manage security defaults > enable security defaults toggle to no.

7. Create a new Azure AD user by making a new local branch from the cloned jksprattler/azure-security repo. Run the azuread-create-users.py script to generate the terraform syntax for your new user: python scripts/azuread-create-users.py

   • The script will run an az cli command invoking a custom request through Microsoft Graph to capture your Azure AD default domain using your current az login.

8. Navigate to the azuread-users-groups-roles directory and paste the terraform code for your new user into the main.tf file using either of the existing Engineering or Art AD Groups or create a new group. For example:

   resource "azuread_user" "raybrown" {
     user_principal_name   = "raybrown@jennasrunbooks.com"
     display_name          = "Ray Brown"
     department            = "Art"
     password              = "Super$ecret01@!"
     force_password_change = true
   }
   

9. Save and commit the changes. Review the github-actions bot output from your PR, specifically the terraform plan results, which will perform the following functions on your behalf using the Azure SPN (ie gh-actions-runbooks-ad):

   

10. From the azuread-users-groups-roles directory, perform your terraform apply to create the new Azure AD user.

11. Enable Self Service Password Reset (SSPR) for All users. From Portal UI navigate to: Azure AD > Password reset > Auth methods > SSPR enabled: All

Import Existing Azure AD Users into Terraform

If you have a number of users that already exist in your Azure AD and are looking to start managing this part of your cloud estate using Terraform, you can run the scripts/azuread-import-users.py script which will extract a list of your current Azure AD user's Display Names, Principal Names and Departments associated with the current Azure tenant you are logged into (az login). The script runs an az ad query capturing the user details and copies them to a tsv file which is then read by python and converted into Terraform syntax. Once you have your list of users, follow the Procedure above starting at step 8.

python scripts/azuread-import-users.py
# For much larger lists of users, save the python output to either a txt or tf file
python scripts/azuread-import-users.py > users.tf

Summary

Upon completion of the above procedure, you should now have a basic architecture started for implementing Azure IAM as code using Azure AD and Azure RM Terraform providers. A CI/CD pipeline is implemented using a GitHub Actions workflow generating Terraform format/init/validate/plan results based on PR commits providing a solution for code review by your team before invoking terraform apply locally. Python scripts are available for generating terraform syntax for new Azure AD users (azuread-create-users.py ) and for extracting a list of existing Azure AD users which then generates the terraform code for the list of existing users to be imported into your terraform state (azuread-import-users.py). While this approach might be a good initial start for managing your Azure IAM as code, alternatively there's a link in the references section below to another more elegant solution to try provided by HashiCorp which uses a for_each loop against a CSV file of users. Check out my Part 2 post in this series where I apply the for_each meta-argument for this implementation.

References

• Enable Azure AD Multi-Factor Authentication - Microsoft Entra Microsoft Docs

• Enable Azure Active Directory self-service password reset - Microsoft Entra Microsoft Docs

• Automate Terraform with GitHub Actions Terraform - HashiCorp Learn

• Future improvements: Terraform for_each loop through a list of users defined in a CSV as outlined by HashiCorp to improve your identity governance strategy as your user base grows increasing in complexity of management on a per-user basis.

This article was originally published in November 2022 on my GitHub io blog here

Purpose

After working with GitHub Actions as my Terraform CI pipeline over the past year, I started looking for potential methods to clean up the Plan outputs displayed in PR comments to provide a more streamlined PR review. I was interested in finding a way to redact the "Refreshing state…" messages as I find them distracting and unnecessary for reviewing. These messages can also get quite lengthy for larger infrastructures containing many resources managed by Terraform. Essentially what Terraform is doing when generating these messages is ensuring that your state files are in alignment with the existing infrastructure.

I also found that you can incorporate the diff utility into your Actions pull-request script section to provide color highlights in the plan output. This runbook will cover both the lines of code needed for the diff utility to display results correctly and a method for silencing "Refreshing state…" messages.

I'll only be discussing the GH Actions jobs for Terraform plan, show, reformatting the plan, creating the plan environment variable and incorporating this into the script section of the pull-request plan output.

Code Samples

• Example code snippets will be taken from my workflow on Github here

• The PR I used for testing the config can be reviewed here

GitHub Actions Config

• In the Terraform Plan job ensure that the -no-color flag is set as without it the output is not rendered correctly by the JavaScript and you'll see garbled text/characters. Apply the -out flag which saves the plan output to a local file and assign it a name:

        - name: "Terraform Plan"
          id: plan
          run: terraform plan -detailed-exitcode -no-color -out=plan -input=false
          continue-on-error: true
  

• Create a job for the Terraform Show output. This is going to read the local file of the plan saved in the previous step. Running this job is what will redact all of the "Refreshing state…" messages which get generated by the original terraform plan. Set an if condition to only run the show job when the plan has succeeded or provided exit codes 0 (no changes) or 2 (changes present) and write the contents to a text file:

        - name: Terraform Show 
          id: show 
          if: steps.plan.outcome == 'success' || steps.plan.outputs.exitcode == '0' || steps.plan.outputs.exitcode == '2'
          run: terraform show -no-color plan > plan.txt
          continue-on-error: true
  

• Create a job to Reformat the plan contents of the text file and write it to a newly formatted text file. This will render the plan output in a way that the diff utility recognizes changes within the file as its read during the pull-request script workflow. The sed command in the job uses a Regex statement to apply spaces in front of any symbols next to resource actions by pushing them to the first column of the output. This is required for the diff utility to correctly render the output into color highlights for changes.

        - name: Reformat Plan 
          run: |
            cat plan.txt | sed -E 's/^([[:space:]]+)([-+~])/\2\1/g' > format_plan.txt
          continue-on-error: true
  

• Create a job to assign the new formatted plan output to a Github Environment variable to call the var from within the pull-request script. Note the line containing "${PLAN:0:65536}" is required for very large plan output as the GitHub database sets a limit of 65536 characters on comments. Without setting this limit, if you were to submit a PR over the limit the pipeline would fail. However, with this setting applied a very large plan would be truncated. In a truncated scenario, the reviewer can navigate to the Actions tab of the repo and analyze the full Terraform Plan job contents of the workflow.

        - name: Put Plan in Env Var
          run: |
            PLAN=$(cat format_plan.txt)
            echo "PLAN<<EOF" >> $GITHUB_ENV
            echo "${PLAN:0:65536}" >> $GITHUB_ENV
            echo "EOF" >> $GITHUB_ENV     
  

• Update the pull-request script with the diff utility and the new Plan environment variable:

              <details><summary>Show Plan</summary>
        
              \`\`\`\diff\n
              ${{ env.PLAN }}
              \`\`\`
        
              </details>
  

Conclusion

With the Terraform jobs described above in place, the CI pipeline for the PR comments will no longer display the "Refreshing state…" messages and color highlights will be generated for all changes (ie lines with -+~ symbols) to plan output. This provides for an overall cleaner PR comment for the reviewer as seen in this example:

This article was originally published in July 2022 on my GitHub io blog here

Overview

The purpose of this runbook is to demonstrate the implementation of an AWS Hybrid DNS design and architecture between an AWS region hosting private-only subnets and an on-prem private corporate data center. This design intends to simulate a hybrid DNS cloud connectivity setup to an on-prem environment using AWS DirectConnect (DX) however, the actual implementation will provide private DNS resolution over an established inter-region AWS VPC Peering connection through various Route 53 services and Linux Bind DNS server components as detailed below.

Pre-requisites

• GitHub Account

• AWS Account

• AWS configuration and credentials setup

• Terraform installed

Design Artifacts

• YouTube demo recording

• Diagram: Simulated AWS Hybrid DNS Network Design Architecture

• Diagram: Actual AWS Hybrid DNS Network Design Architecture

• Clone/Fork the repo containing the Terraform artifacts for the AWS Hybrid DNS design here: jksprattler/aws-networking

  • relevant files are under /aws-terraform-hybrid-dns

• Set up integrated DNS resolution for hybrid networks in Amazon Route 53 - AWS Prescriptive Guidance

Diagrams

Architecture

The us-east-2 region will be hosting the micros4l-onprem VPC with a prefix of 192.168.10.0/24 also containing 2 private subnets. Two t2.micro EC2 instances will be deployed here in separate subnet/availability zones called micros4l-onpremdnsa/b and configured with Bind and the zone corp.microgreens4life.org. Additionally, the zone for aws.microgreens4life.org will be configured on these servers with forwarders set to the AWS Inbound endpoint IP addresses to forward requests for the Web subdomain to the AWS Route 53 resolvers. A third basic t2.micro EC2 instance called micros4l-onpremapp will be deployed which we'll use for testing DNS resolution from "on-prem" to AWS (us-east-1).

The us-east-1 region will be hosting the micros4l-aws VPC with a prefix of 10.10.0.0/16 containing 2 private subnets - note the non-overlapping private IP space between the environments as this is a requirement for VPC peering in addition to DirectConnect (DX) connectivity which is being simulated. Two basic t2.micro EC2 instances, micros4l-awsec2b/b will be deployed here for testing DNS resolution into our simulated Corporate on-prem data center. Each instance is deployed in a separate subnet/availability zone. The Route 53 private hosted zone for aws.microgreens4life.org with an A record of web.aws.microgreens4life.org is configured here. Additionally, Route 53 Inbound and Outbound endpoints are hosted here where each endpoint gets associated with both of the us-east-1 private subnets. The Outbound endpoint will have a forwarding rule for the corp.microgreens4life.org zone associated with it which targets the micros4l-onpremdnsa/b Bind servers hosted "on-prem" (us-east-2) to resolve outbound requests to the Corporate subdomains.

All instances will be deployed with settings configured to allow Systems Manager connectivity as this is the only way to connect to these private instances in this environment as none of them will be deployed with a public IP address nor is there any internet gateway created - these are completely isolated private environments.

Procedure

1. Navigate to the /global/iam directory and run terraform plan/apply:

   cd aws-terraform-hybrid-dns/global/iam
   terraform plan
   terraform apply
   

   Resources deployed in this terraform module:

   • roles.tf - IAM instance policy, roles and policy attachments which all EC2 instances in this design will utilize

   • s3.tf - S3 bucket for storing the terraform state files. Update your bucket name here as it must be globally unique

2. Navigate to the /us-east-2 directory and run terraform plan/apply:

   cd aws-terraform-hybrid-dns/us-east-2
   terraform plan
   terraform apply
   

   Resources deployed in this terraform module:

   • ec2.tf - micros4l-onpremdnsa/b simulating on-prem Linux Bind/DNS servers and micros4l-onpremapp Linux server

   • vpc.tf - VPC with prefix 192.168.10.0/24, 2x private subnets, private route table associated with the 2x subnets, Security Group and rules allowing SSM access and DNS requests, VPC Endpoints for SSM connectivity

3. Capture the outputs from the /us-east-2 module deployment and save them in a temp text file for use as input in the next step. For example:

   onprem-private-rt_id = "rtb-0fae5266503453da8"
   onpremdnsa_ip = "192.168.10.11"
   onpremdnsb_ip = "192.168.10.236"
   onpremvpc_id = "vpc-0ab74c12320891aa3"
   

4. Navigate to the /us-east-1 directory and run terraform plan/apply:

   cd ../us-east-1
   terraform plan
   terraform apply
   

   Resources deployed in this terraform module:

   • ec2.tf - micros4l-awsec2a/b AWS instances

   • route53.tf - aws.microgreens4life.org Route 53 hosted private zone, web.aws.microgreens4life.org A record, Route 53 Inbound endpoint, Route 53 Outbound endpoint for the corp.microgreens4life.org domain with a Forwarding rule pointing to the Corp on-prem environment (us-east-2)

   • vpc.tf - VPC with prefix 10.10.0.0/16, 2x private subnets, private route table associated with the 2x subnets, VPC Peering connectivity between the AWS us-east-1 region to the "on-prem" us-east-2 region, Security Group and rules allowing SSM access and DNS requests, VPC Endpoints for SSM connectivity

5. Capture the outputs from the /us-east-1 module deployment and save them in a temp text file for use as input in the next step. Note you'll only need the "ip" address output from each of the 2 endpoints. For example:

   aws_route53_resolver_inbound_endpoint_ips = toset([
     {
    "ip" = "10.10.0.90"     <---- INBOUND_ENDPOINT_IP1
    "ip_id" = "rni-2bc122c23384d09af"
    "subnet_id" = "subnet-0e6a97614d0833b47"
     },
     {
    "ip" = "10.10.10.221"     <---- INBOUND_ENDPOINT_IP2
    "ip_id" = "rni-75c2ecfc30094b3a9"
    "subnet_id" = "subnet-0dce015d7ba12e0de"
     },
   ])
   

6. In the awszone.forward file, replace the INBOUND_ENDPOINT_IP1 and INBOUND_ENDPOINT_IP2 values of the forwarders with the Endpoint IP addresses from the outputs of the previous step.

7. From the AWS console, navigate to the EC2 instances in the us-east-2 region, select micros4l-onpremdnsa and initiate a connection to it via Session Manager. Enter sudo -i and with your editor of choice, vi or nano into the /etc/named.conf file. Scroll to the end of the file and paste the contents of your updated awszone.forward file, save and exit. Run the following command to restart the bind service:

   systemctl restart named 
   systemctl status named
   

   Using dig or nslookup, test that your local DNS server is resolving the AWS Route 53 private zone/domain for aws.microgreens4life.org now that you've applied the Route 53 endpoint IP addresses into the Bind server's named.conf file. You should see it resolve to the 10.10.x.x private IP space of the us-east-1 AWS VPC where the Route 53 inbound endpoints are hosted. For example:

   sh-4.2$ dig web.aws.microgreens4life.org @127.0.0.1 +short
   10.10.0.172
   10.10.10.31
   

   Repeat above steps for micros4l-onpremdnsb From the AWS console, navigate to the Route 53 service in the us-east-1 region and validate that the A record hosted in your private Route 53 zone is using the same IP addresses that your DNS server in us-east-2 just resolved to.

8. Navigate to the EC2 instances in us-east-2 and select micros4l-onpremapp and initiate a connection to it via Session Manager. Enter sudo -i and with your editor of choice, vi or nano into the /etc/sysconfig/network-scripts/ifcfg-eth0 file. Scroll to the end of the file and paste the following contents replacing the THE_PRIVATE_IP_OF_ONPREM_DNS_A/B values with the actual private IP addresses of the on-prem DNS servers which were given in the outputs of your terraform apply for the us-east-2 implementation in step 3.):

   DNS1=THE_PRIVATE_IP_OF_ONPREM_DNS_A
   DNS2=THE_PRIVATE_IP_OF_ONPREM_DNS_B
   

   Restart the network services: systemctl restart network Run a test ping/dig from the micros4l-onpremapp instance to the AWS route 53 hosted subdomain:

   ping web.aws.microgreens4life.org
   dig web.aws.microgreens4life.org +short
   

9. Navigate back to the ec2 instances in us-east-1 and initiate a systems manager session on micros4l-awsec2a/b and test DNS resolution of the on-prem hosted subdomain:

   ping app.corp.microgreens4life.org
   dig app.corp.microgreens4life.org +short
   

10. Cleanup! Run a terraform destroy in each region/module starting with us-east-1 - Note I configured ignore lifecycle rules on the accepter_route_table_id and accepter_vpc_id prompts so just hit Enter here to bypass these. You'll need to input the onpremdnsa/b_ip private IP's as I couldn't get a lifecycle rule to work here:

    terraform destroy
    var.accepter_route_table_id
      Route table id of the accepter that you want to peer with it
      Enter a value: <Enter>
    var.accepter_vpc_id
      VPC id that you want to peer with it
      Enter a value: <Enter>
    var.onpremdnsa_priv_ip
      Private IP Address of micros4l-onpremdnsa
      Enter a value: 192.168.10.53 <-----onpremdnsa_ip
    var.onpremdnsb_priv_ip
      Private IP Address of micros4l-onpremdnsb
      Enter a value: 192.168.10.243 <-----onpremdnsb_ip
    

    Do the same for us-east-2 - No prompts for input on this one, simply just terraform destroy it. I've left the /global/iam resources in tact since it's just an IAM role/policy and S3 bucket storing my terraform state files.

Summary

Upon completion of the above procedure, you should now have 2 separate private environments with fully integrated DNS resolution between them. The private AWS VPC instances in us-east-1 are successfully resolving the Corporate subdomain hosted in the private "on-prem" VPC in us-east-2 via the outbound endpoint and forwarding rule for the Corporate domain which gets routed via the VPC Peering connection. Conversely, the private "on-prem" instances in us-east-2 are successfully resolving the web subdomain hosted in the private AWS VPC us-east-1 via the inbound endpoint resolver.

Note the original idea for this design came from Cloud Trainer, Adrian Cantrill. You can find the CFT stack, procedure steps, and videos for his lab here

Differences between deployments where I:

• Coded all infrastructure steps in Terraform including vpc peering, vpc peering inter-region routes, route 53 inbound and outbound endpoints and forward rules, etc. instead of CloudFormation (HybridDNS.yaml) used for initial/base infrastructure.

• Coded outputs.tf files to provide values for the input variable strings for the deployments to the separate regions/modules to prevent needing to hunt down id's and ip addresses from within the AWS console (ie vpc peering id, route table id for peering connection, onpremdnsa/b private ip addresses for the aws zone file)

• Deployed the simulation of the on-prem / DX connected environment in a completely separate region (us-east-2) instead of all in the same us-east-1 region in an attempt to increase complexity, validate inter-region vpc peering works with DNS resolution against private Route 53 endpoints and just for overall better visualization of connectivity between 2 isolated environments/regions

• Chose microgreens4life for my Domain/zone instead of animals4life - nothing against animals, I just really love microgreens. I also replaced all resource names in my code with my variation of m4l/micros4l/microgreens4life which allowed me the opportunity to deeply review the code line by line so I didn't just copy/paste pieces of Adrian's CFT stack code (ie a4l/animals4l/animals4life).

This article was originally published in July 2021 on my GitHub io blog here

Overview

The purpose of this runbook is to define the steps needed to deploy a secure static website hosted on an AWS S3 Bucket served by an AWS CloudFront Distribution (CDN) with automated deployment of updates using a GitHub Actions Workflow.

Our use case for this runbook will be following the build of my personal resume website: jennasprattler.com | www.jennasprattler.com

Pre-requisites

• Static website

  • For ideas, there's a curated directory of Jekyll Themes

  • I'm using this Jekyll theme for my website: modern-resume-theme

• GitHub Account and Repo

  • To store your website files and run GitHub Actions Workflows

• AWS Account

  • To implement Route 53 DNS records, S3 Bucket storage, CloudFront CDN and AWS Certificate Manager for SSL Certificates

High-level Overview - Traffic Flow

• Mobile and Desktop browsers are supported by the Jekyll website

• AWS Route 53 performs DNS resolution for jennasprattler.com and www.jennasprattler.com

• AWS CloudFront CDN serves up any cached content of your website from one of its Edge Locations closest to you

• AWS Certificate Manager hosts the SSL Certificate for your website which gets assigned to your CloudFront CDN encrypting all user traffic

• AWS S3 Bucket hosts all of your website files whereas the CloudFront CDN retrieves anything that it doesn't have cached on its Edge Location

High-level Overview - CI/CD Flow

• Developer writes the code for the Jekyll Website

• GitHub hosts the repo for the Jekyll website files

• GitHub Actions kickoff the CI/CD Workflow whenever a push is made to the main branch

• CI/CD Workflow uploads the output files from the Jekyll _site directory to the AWS S3 bucket

• CloudFront invalidation is run to clear out any cached content and immediately serve the updated S3 website content

Procedure

Create Route 53 Domain

• I'll be creating my domain using Route 53 however, you can use another domain provider if you like just note that you'll need to follow a slightly different procedure than what I've defined in this runbook.

1. Navigate to the AWS Route 53 service and check the availability of your domain name - if available purchase it. At the time of this writing, it cost me $12 per year for my new .com domain.

Your new domain should now show a status of "Domain registration in progress." It will take approximately 30 minutes for your new domain to be registered.

Proceed with the next section to create the S3 bucket. We'll create DNS records for our new site in a later section.

Create S3 Bucket

1. Copy the Cloudformation stack YAML code below and replace all FIXME values per your environment.

   • BucketName must match your domain name exactly

   • PublicAccessBlockConfiguration properties are all set to false to allow public access to your website

   • BucketPolicy has s3:GetObject action set to allow anyone can read the object data and view the website

   • WebsiteConfiguration enables the static website capability in S3

   • WWWBucket creates an empty bucket only used to redirect www.FIXME.com traffic to your FIXME.com bucket; only needed if you decide to not use a CloudFront CDN and just host unencrypted content from S3 only (which is what I did initially)

   ---
   AWSTemplateFormatVersion: '2010-09-09'
   
   
   Description: Simple S3 Bucket to host static public website.
   
   
   Resources:
   
   
     S3Bucket:
       Type: AWS::S3::Bucket
       Properties:
         BucketName: FIXME.com
         PublicAccessBlockConfiguration:
           BlockPublicAcls: false
           BlockPublicPolicy: false
           IgnorePublicAcls: false
           RestrictPublicBuckets: false
         Tags:
           -
             Key: Description
             Value: FIXME
           - Key: Project
             Value: FIXME.com
         VersioningConfiguration:
           Status: Enabled
         WebsiteConfiguration:
           ErrorDocument: 404.html
           IndexDocument: index.html
             
     BucketPolicyDataSync:  
       Type: 'AWS::S3::BucketPolicy'  
       Properties:  
         Bucket:  !Ref S3Bucket
         PolicyDocument:
           Statement:  
           -  
             Sid: "AllowAccesToIAMRole"  
             Action:  
               - "s3:GetObject"
               
             Effect: "Allow"  
             Resource:  
               Fn::Join:  
                 - ""  
                 -  
                   - "arn:aws:s3:::"  
                   -  
                     Ref: "S3Bucket"  
                   - "/*"  
             Principal: "*"            
   
     WWWBucket:
       Type: AWS::S3::Bucket
       Properties:
         BucketName: www.FIXME.com 
         AccessControl: BucketOwnerFullControl
         WebsiteConfiguration:
           RedirectAllRequestsTo:
             HostName: FIXME.com
   
   Outputs:
   
   
     S3BucketName:
       Value: !Ref S3Bucket
       Description: S3 Bucket for object storage
   
   
     S3BucketARN:
       Value: !GetAtt S3Bucket.Arn
       Description: S3 bucket ARN
   

   1. Navigate to the region closest to you and go to the CloudFormation service.

   2. Upload your updated CFT stack to create your new S3 buckets for hosting your static website files and monitor the event progress.

      

   3. Once created, your S3 bucket permissions and policy should look similar to this:

Capture your S3 Endpoint URL - you can find this by navigating to your new S3 Bucket > Properties > Static web site hosting > Endpoint. We'll need this for the CloudFront section.

Create Route 53 Hosted Zone and DNS Records

• Proceed once the Domain registration has been completed

1. Navigate to the Route 53 service where you should see your new domain has moved from Pending requests to Registered domains.

2. Create a Hosted zone for your new domain by entering your Domain Name and selecting Public Hosted Zone in the Type dropdown.

3. The NS and SOA records for your new domain will auto-create for you.

4. Proceed to the next section to create your new CloudFront distribution as you'll need to point your new A records to the new CloudFront Domain Name for your website. You can also go through the ACM certificate process to automatically validate your new domain which will automatically create a new CNAME record for you in your hosted zone. Once that's completed, return here to step 5.

5. In your new public hosted zone, you should now see a CNAME record pointing to your ACM Certificate in addition to the NS and SOA records.

6. Create 2 new A records, one for your root domain and the other for any sub-domains you want pointed to your root domain (ie www.FIXME.com). Each A record should point to your new CloudFront Domain Name created in the Create CloudFront Distribution section.

   

Create CloudFront Distribution

1. Navigate to the AWS Cloudfront service and select Create distribution > Web > Get started

2. Under Origin Domain name, paste your S3 Endpoint and remove the prefix, "http://" from the URL. Under Default Cache Behavior Settings > Viewer Protocol Policy select "Redirect HTTP to HTTPS". Leave the remaining Origin settings as default.

   

3. Under Distribution Settings > Alternate Domain Names (CNAMEs):

   • Enter your new domain and any other subdomains (ie www.FIXME.com) you plan to use each on a new line.

   • Select "Custom SSL Certificate" to secure your website using a stored in Amazon Certificate Manager (ACM) in the US East (N. Virginia) Region. Select the "Request or Import a Certificate with ACM button" which will take you through the process of validating your domain. I went through the DNS validation process since my domain was registered using Route 53.

• NOTE: Your Route 53 domain must have completed registration before you can request to validate it with a new ACM cert. It can take 30+ minutes for your SSL cert to be validated, meanwhile, you can move onto the IAM policy/user creation steps below. Don’t forget to request your ACM cert from us-east-1 which is the only region supported by Cloudfront!

• For the Default Root Object, enter: index.html

• Once your new CDN has been created, copy the 14-character alphanumeric Cloudfront ID and Domain Name (ie FIXME.cloudfront.net).

Create IAM Policy and IAM User for Github Actions

1. Navigate to IAM and create a new IAM Policy using the JSON editor.

2. Paste the following policy contents into the JSON editor and update all "FIXME" values for your bucket name, AWS account ID and CloudFront ID:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Resource": [
                    "arn:aws:s3:::FIXME-bucket-name",
                    "arn:aws:s3:::FIXME-bucket-name/*"
                ],
                "Sid": "VisualEditor1",
                "Effect": "Allow",
                "Action": [
                    "s3:*"
                ]
            },
            {
                "Sid": "VisualEditor2",
                "Effect": "Allow",
                "Action": "cloudfront:*",
                "Resource": "arn:aws:cloudfront::FIXME-aws-account-number:distribution/FIXME-distribution-id"
            }
        ]
    }

1. Create a new IAM User with programmatic access and attach the IAM Policy you just created above.

2. Copy the AWS access key ID and Secret access key for your user to a safe location such as secrets manager or key vault, as these will be used in the GitHub Action Workflow setup below.

Create GitHub Action Workflow

1. From within your GitHub repo navigate to: .github/workflows/ and create a new file called build-and-deploy.yml

2. Copy and paste the following into your newly created GitHub Action workflow and update the FIXME value for the region that your S3 bucked was deployed in.

name: CI / CD

# Controls when the action will run. 
on:
  # Triggers the workflow on push for the main branch
  push:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
  
env:
  AWS_ACCESS_KEY_ID: $
  AWS_SECRET_ACCESS_KEY: $
  AWS_DEFAULT_REGION: 'FIXME'

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: 2.7
        bundler-cache: true
    - uses: actions/cache@v2
      with:
        path: vendor/bundle
        key: $-gems-$
        restore-keys: |
          $-gems-
    - name: Install dependencies
      run: |
        gem install bundler
        gem install jekyll
        bundle config path vendor/bundle
        bundle install --jobs 4 --retry 3        
    - name: "Build Site"
      run: bundle exec jekyll build
      env:
        JEKYLL_PAT: $
    - name: "Deploy to AWS S3"
      run: aws s3 sync ./_site/ s3://$ --acl public-read --delete --cache-control max-age=604800
    - name: "Create AWS Cloudfront Invalidation"
      run: aws cloudfront create-invalidation --distribution-id $ --paths "/*"

Create GitHub Action Secrets

1. Navigate to Your Repo > Settings > Secrets > Actions

2. Configure The GitHub Action Secrets for the following:

• AWS_ACCESS_KEY_ID - The AWS access key ID associated with the programmatic IAM User.

• AWS_SECRET_ACCESS_KEY - The AWS secret key ID associated with the programmatic IAM User.

• AWS_S3_BUCKET_NAME - Your AWS bucket name hosting your website, for example jennasprattler.com or FIXME.com

• AWS_CLOUDFRONT_DISTRIBUTION_ID - The 14-character alphanumeric Cloudfront distribution ID fronting your S3 bucket's website.

• JEKYLL_PAT - Set up a GitHub token that can be used by the workflow to build the Jekyll _site pages.

Deploy your Jekyll website using GitHub Actions

1. Make a change to your Jekyll site > commit > push the changes to main.

2. Navigate to Your Repo > Actions and under the CI / CD Workflow select the latest running build to monitor the progress:

   

3. Once the Workflow has completed, browse to your website in a new tab or refresh any open tabs to see the updated content.

You now have a functioning, secure, serverless, static website configured for automatic updates upon code commit to your repository!

This article was originally published in July 2021 on my GitHub io blog here

Purpose

I gave a presentation on DevOps terminology, roles and strategy to a group of leaders in the networking group of my employer. The purpose of the presentation was to engage in open dialogue and support facilitating the path to a successful DevOps transformation for our network teams.

The catalyst for my presentation was based on understanding who would be assigned to some very technical-specific DevOps roles. I took some time to put together a slide deck on DevOps topics (linked below) which led to a very fruitful conversation during our call. Based on the dialogue and feedback I received from the presentation, it was apparent that there was a much clearer understanding amongst the group on what a Network DevOps transformation might look like for our specific scenario.

Background

Over the past several months a small group of us embarked on a journey into uncharted territory with goals to design a complete CI/CD infrastructure using Azure DevOps by building out the architecture completely from scratch. A key driver for this was automating the upcoming deployments for our global cloud firewall solution which was going to be hosted in all of our landing zones.

We were highly collaborative amongst ourselves and also reached out to engineers on different teams for valuable insight as we progressed. Once we nailed down the design and architecture for our ADO CI/CD infrastructure our next task was to train our network support teams on how to use it all. We held weekly/monthly demos reviewing ADO Git, Wiki, build/release pipelines, etc. and built out lab environments for our different cloud providers to encourage a learning environment within the network teams.

Presentation

I've done a mock-up of the "Network DevOps Transformation" presentation as the feedback I received from the group was very positive. My colleagues found the information to be beneficial in their understanding of DevOps and what a transformation from a traditional Operations team into a highly performing DevOps team might look like as an entry point.

I've removed any employer branding and company-specific details from the presentation. What you'll find in the youtube recording and slide deck links below are the core contents of the presentation that I gave originally. This is all publicly available information based on my research and personal DevOps experience over the past couple of years.

• YouTube recording: Network DevOps Transformation

• Slide deck: Network DevOps Transformation pdf presentation

Final Thoughts

I hope you've found this presentation helpful as you move along in your DevOps journey. As I've learned in my research and experience, every DevOps transformation is going to look a little different - there is no one size fits all approach and not everyone will agree on everything. Although we may not always agree, the main goal is to arrive at that self-organizing system of collaboration where all members of the team feel valued and respected.

The config in this gif is from a lab I'm working on deploying AWS IAM objects which include local custom modules, meta-arguments, variable types and expressions:

I only recently started looking into terraform visualization tools and have experimented with this one and 𝘁𝗲𝗿𝗿𝗮𝗳𝗼𝗿𝗺 𝗴𝗿𝗮𝗽𝗵 so far. Terraform graph adds a bit of granularity to the visualization as it includes all components such as variables, providers, etc. However, it can be a bit more difficult to follow the code logic at first glance depending on complexity. Here's the png output from my AWS IAM lab using terraform graph:

If you're on Ubuntu/Debian 🖥 check it out by running the following:

sudo apt-get update
sudo apt install graphviz 
terraform graph -type=plan | dot -Tpng -o graph.png

Both of these tools 🔧 offer a helpful way to better understand your Terraform infra code particularly if the config is complex, there's little to no documentation on the code logic, or you're newer to Terraform in general. I also like how easy they are to spin up and use plus they're open-source 😎 That said, always be cautious about uploading config files containing private/secure data to a remote website like terraform visual. In those scenarios, it'd be best to run something locally like terraform graph.

Are there other tools you've found useful for visualizing terraform infra?

Pain points I was looking to address with this one:

✅ No more logging into the AWS console UI

✅ No more crafting the "aws iam update-login-profile …" command

✅ No more manually generating passwords to match the strict password policy enforced

Link to Script: https://github.com/jksprattler/aws-security/blob/main/scripts/aws_iam_user_password_reset.py

"𝗬𝗼𝘂𝗿 𝗹𝗼𝗰𝗮𝗹 𝗰𝗵𝗮𝗻𝗴𝗲𝘀 𝘁𝗼 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝗳𝗶𝗹𝗲𝘀 𝘄𝗼𝘂𝗹𝗱 𝗯𝗲 𝗼𝘃𝗲𝗿𝘄𝗿𝗶𝘁𝘁𝗲𝗻 𝗯𝘆 𝗺𝗲𝗿𝗴𝗲"

I didn't have any changes to commit locally, particularly not on the specified file, nor was anything stashed in my local branch. Despite this, Git continued to prevent me from pulling my latest commits to my master branch so that I could perform a rebase.

I tried the following to rectify this:

git pull
git pull --force
git pull origin --force
git stash pop
git stash
git stash drop
git merge --ff-only origin/master
git pull origin master
git reset HEAD

But none of these ⬆️ worked.

The solution for me was to:

git checkout path/to/file/to/revert
git reset HEAD path/to/file/to/revert
git pull

I was then able to switch back to my local branch, rebase the master branch, fix some conflicts and move on with my day 😎

Have you run into this particular scenario before? Any other suggested commands that worked for you?