Install and Configure Amazon CloudWatch Agent — Using Command Line and AWS Systems Manager

There are various tools and services for monitoring and logging in near real-time. In AWS, Amazon CloudWatch is a service to collect metrics and logs with tracking, monitoring and alerting capabilities from resources, applications and services running on AWS and on-premises.

In this blog, I’ll cover the steps to install and configure the unified CloudWatch agent for collecting metrics and logs from the Amazon EC2 instance. The blog will have two parts to install and configure the CloudWatch agent. The first part covers using the command line, and the second part covers using AWS Systems Manager.

As prerequisites, an Amazon EC2 instance is created running Ubuntu Server 20.04 and installed NGINX web server.

Install and configure the CloudWatch agent using the command line

Install and configure the CloudWatch agent using the command line

Step 1: Create an IAM role to run the CloudWatch agent

Create an IAM role with AWS managed policy type — CloudWatchAgentServerPolicy, which allows the CloudWatch agent to send metrics and logs to CloudWatch.

Attach the IAM role to the Amazon EC2 instance where you want to install the CloudWatch agent.

Step 2: Download the CloudWatch agent package

Find the appropriate architecture and platform and copy the download link from Amazon CloudWatch docs.

With the copied download link, use the following command to download the package.

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Step 3: Install the CloudWatch agent package

After successfully downloading the package, use the following command to install the package.

sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Step 4: Create the CloudWatch agent configuration file

Use the following command to start the CloudWatch agent configuration wizard to create the CloudWatch agent configuration file.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Answer the questions to customize the CloudWatch agent configuration file.

For this blog, I have entered the following parameters:

CloudWatch Agent Configuration File Wizard Parameters

After entering the appropriate parameters, the content of the config file and config file location will be shown in the output, and the program will exit.

Here’s the CloudWatch agent configuration file generated based upon the above parameters:

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "cwagent"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/nginx/access.log",
                        "log_group_name": "web-server",
                        "log_stream_name": "{instance_id}/{ip_address}/access.log",
                        "retention_in_days": -1
                    },
                    {
                        "file_path": "/var/log/nginx/error.log",
                        "log_group_name": "web-server",
                        "log_stream_name": "{instance_id}/{ip_address}/error.log",
                        "retention_in_days": -1
                    }
                ]
            }
        }
    },
    "metrics": {
        "aggregation_dimensions": [
            [
                "InstanceId"
            ]
        ],
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            }
        }
    }
}

For more information, visit Create the CloudWatch agent configuration file.

Step 5: Start the CloudWatch agent

Use the following command, which contains the configuration file location created in the previous step.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Check the status of the Amazon CloudWatch agent.

systemctl status amazon-cloudwatch-agent

Step 6: Verify metrics and logs in the Amazon CloudWatch

Check and verify the result in the CloudWatch Metrics and CloudWatch Logs.

CloudWatch Metrics — All metrics

CloudWatch Agent — Custom metrics

CloudWatch Agent — Logs

Install and configure the CloudWatch agent using AWS Systems Manager

Install and configure the CloudWatch agent using AWS Systems Manager

Prerequisites

As we utilize AWS Systems Manager, ensure the AWS SSM Agent is installed.

Create and attach an IAM role to the Amazon EC2 instance with AWS managed policy type — AmazonSSMManagedInstanceCore to use AWS Systems Manager service core functionality.

For more information on SSM Agent, visit Working with SSM Agent.

Step 1: Create an IAM role to run the CloudWatch agent

Add AWS managed policy type — CloudWatchAgentServerPolicy to the IAM role created as a part of prerequisites, which allows the CloudWatch agent to send metrics and logs to CloudWatch.

Step 2: Download and install the CloudWatch agent using Systems Manager

• Open Systems Manager console.
• Navigate to Run Command under Node Management.
• Choose Run command.

• In the Command document, choose AWS-ConfigureAWSPackage.

• In the Command parameters,
  - Choose Install in the Action list.
  - Enter AmazonCloudWatchAgent in the Name field.
  - Enter latest in the Version field.

• In the Targets, choose the appropriate method for selecting targets.

• In the Output options, you can choose the Amazon S3 bucket and Amazon CloudWatch logs to send command output.
• Choose Run.
• In the Targets and outputs, verify the status and choose View Output. Additionally, you can view command output in the previously selected output options (Amazon S3 bucket and Amazon CloudWatch logs).

Step 3: Store CloudWatch agent configuration file in the Systems Manager Parameter Store

• Open Systems Manager console.
• Navigate to Parameter Store under Application Management.
• Choose Create parameter.
• In the Name, enter the appropriate name. Use the ‘AmazonCloudWatch-’ prefix if you used AWS managed policy — CloudWatchAgentServerPolicy.
• In the Type, choose String.
• In the Data type list, choose text.
• In the Value, insert valid CloudWatch agent configuration. For this blog, I am using the following agent configuration generated in the previous part, step 4 — Create the CloudWatch agent configuration file.

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "cwagent"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/nginx/access.log",
                        "log_group_name": "web-server",
                        "log_stream_name": "{instance_id}/{ip_address}/access.log",
                        "retention_in_days": -1
                    },
                    {
                        "file_path": "/var/log/nginx/error.log",
                        "log_group_name": "web-server",
                        "log_stream_name": "{instance_id}/{ip_address}/error.log",
                        "retention_in_days": -1
                    }
                ]
            }
        }
    },
    "metrics": {
        "aggregation_dimensions": [
            [
                "InstanceId"
            ]
        ],
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            }
        }
    }
}

• Choose Create parameter.

Step 4: Start the CloudWatch agent

• Open Systems Manager console.
• Navigate to Run Command under Node Management.
• Choose Run command.

• In the Command document, choose AmazonCloudWatch-ManageAgent.

• In the Command parameters,
  - Choose configure in the Action list.
  - Choose ec2 in the Mode list.
  - Choose SSM in the Optional Configuration Source list.
  - Enter the SSM parameter name in the Optional Configuration Location created in the previous step.
  - Choose yes in the Optional Restart list.

• In the Targets, choose the appropriate method for selecting targets.

• In the Output options, you can choose the Amazon S3 bucket and Amazon CloudWatch logs to send command output.
• Choose Run.
• In the Targets and outputs, verify the status and choose View Output. Additionally, you can view command output in the previously selected output options (Amazon S3 bucket and Amazon CloudWatch logs).

Step 5: Verify metrics and logs in the Amazon CloudWatch

Check and verify the result in the CloudWatch Metrics and CloudWatch Logs.

CloudWatch Metrics — All metrics

CloudWatch Agent — Custom metrics

CloudWatch Agent — Logs

In this blog, you have learned how to install and configure Amazon CloudWatch Agent using the command line and AWS Systems Manager.

Further, there are two more popular ways to install CloudWatch agents. The first is to create a golden AMI, in which the CloudWatch agent can be installed along with other latest security patches, software, configuration, and software agents, and the second is to install the CloudWatch agent on new instances using AWS CloudFormation.

To learn more, please visit Collecting metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent.