Deployment using Terraform

The Terraform template creates AWS resources to enable the monitoring of AWS API gateway logs. The template creates an EC2 instance where Traceable services are running. Traceable fetches the cloudwatch logs every 5 minutes, parses the data, and sends it to the Traceable platform. The processed data is displayed in the Traceable UI. The following flow diagram displays how the traffic flows through:

Before you begin

Make a note of the following points before proceeding with configurations:

• Keep Traceable's access token handy. It will be used when configuring the variables in the tfvars (*.tfvars) file. You can copy the access token by logging into your Traceable account and then navigating to Settings ()→Account → Agent Token.

• Make sure that Terraform is already installed. For more information on installing Terraform, see Download Terraform.

REST API gateway

To monitor the REST API gateway, complete the following:

1. Navigate to the console. aws.amazon.com/apigateway/.

2. Select the API and then select the stage.

3. Navigate to Logs/tracing. 

4. Select Full Request and Response Logs from Cloudwatch settings → Cloudwatch Logs drop-down list.

5. Mark Enable Access Logging as true under Custom Access Logging and append the following JSON in Log Format:

   JSONJSON

   {
     "requestId":"$context.requestId","ip":"$context.identity.sourceIp","httpMethod":"$context.httpMethod","path":"$context.path","status":"$context.status","responseLength":"$context.responseLength","domainName":"$context.domainName"
   }

HTTP API gateway

To monitor the HTTP API gateway, enable access logging and append the JSON mentioned above in the Log Format.

Configure AWS

Configure AWS in your shell and verify that the region is set correctly. Enter the following command to set up your AWS CLI installation:

aws configure

The following example shows sample values. Replace them with your values to configure the credentials correctly.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

For more information on the credentials file, see Configuration and credential file settings.

If you have configured named AWS profiles, export the environment variable AWS_PROFILE=myprofile where the profile named myprofile has the credentials that you wish to use to deploy the Traceable mirroring resources.

Finally, run the following command and verify that the AWS region is set to the region where you wish to install Traceable:

aws configure get region

Alternatively, you can configure the following environment variables:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_REGION or AWS_DEFAULT_REGION

• AWS_DEFAULT_OUTPUT

For more configuration information, see AWS documentation.

Multiple accounts

A single AWS API Gateway account can support multiple accounts across different regions. However, each region within an account must be configured separately. This setup requires establishing cross-account IAM roles in each account, allowing the host account to authenticate and access the necessary resources.

Cross account IAM role

An IAM role is required to authenticate the host account to monitor the API gateways from another account. The following are the required permissions:

  "logs:describeLogGroups"
  "logs:FilterLogEvents"
  "apigateway:GET"

Alternatively, you can use the following Terraform template to create cross account role. Override the account_id variable with the host account’s ID.

data "aws_iam_policy_document" "traceable" {
  statement {
    sid = "readAccess"
    actions = [
      "logs:describeLogGroups",
      "logs:FilterLogEvents",
      "apigateway:GET"
    ]
    effect    = "Allow"
    resources = ["*"]
  }
}

resource "aws_iam_policy" "traceable" {
  name        = "traceable-cross-account-policy"
  description = "IAM policy for traceable instance"
  path        = "/"
  policy      = data.aws_iam_policy_document.traceable.json
}

resource "aws_iam_role" "traceable" {
  name                  = "traceable-cross-account-role"
  description           = "IAM role for traceable instance"
  path                  = "/"
  force_detach_policies = true
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:root"
        }
        Condition = {}
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "traceable" {
  depends_on = [aws_iam_policy.traceable, aws_iam_role.traceable]
  role       = aws_iam_role.traceable.name
  policy_arn = aws_iam_policy.traceable.arn
}

variable "account_id" {
  type = string
}

Download

Traceable provides AWS API gateway traffic mirroring tarball. Complete the following steps to download and untar the tarball: 

1. Enter the following command to download the tarball:

   ActionScriptActionScript

   curl -O https://downloads.traceable.ai/install/aws-api-gateway/terraform/latest/aws-api-gateway-tf.tar.gz

2. Untar the tarball. Enter the following command:

   ActionScriptActionScript

   tar xvzf aws-api-gateway-tf.tar.gz

3. Change directory. Enter the following command:

   ActionScriptActionScript

   cd aws-api-gateway-tf/

Create tfvars file

Create a terraform.tfvars file with terraform variables as shown below:

subnet_id                = "subnet-0fbc2fd166a8e94eb"
key_name                 = "traceable-local"
traceable_refresh_token  = "***"
traceable_agent_endpoint = "https://1.2.3.4:5443"
traceable_environment    = "awsapigw"
traceable_service_name   = "awsapigw"
traceable_api_endpoint   = "api.apac2.traceable.ai"
accounts = [
  {
    region                 = "us-east-1"
    api_list               = ["625vetvy0f/test"]
    cross_account_role_arn = ""
    exclude                = false
  },
  {
    region                 = "us-east-1"
    api_list               = []
    cross_account_role_arn = "arn:aws:iam::1234567890:role/aws-apigw-cross-account-role"
    exclude                = true
  }
]

tags = {
  owner             = "Traceable"
  reason            = "aws apigw"
  expected-end-date = "07/05/2024"
}

service_version = "1.45.0"

Configuration variables

The following tables describe the various terraform variables.

IAM roles for ECS deployment

If you choose ECS deployment, the following IAM roles are created:

1. IAM Role for the ECS Task to Call AWS SDK APIs

Key Permissions:

• API Gateway Access (apigateway:GET): This allows the ECS task to perform GET operations on API Gateway. This could be for retrieving API configurations or monitoring API statuses.

  {
      "Action": "apigateway:GET",
      "Effect": "Allow",
      "Resource": "*",
      "Sid": "getApiGateways"
  }

• CloudWatch Logs Access (logs:describeLogGroups, logs:FilterLogEvents): These permissions allow the task to describe CloudWatch log groups and filter log events. This is useful for the ECS task to monitor logs if needed.

  {
      "Action": [
          "logs:describeLogGroups",
          "logs:FilterLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*",
      "Sid": "filterLogEvents"
  }

• Cross-Account Role Assumption (sts:AssumeRole): This permission allows the task to assume a cross-account role. This is important when the ECS task needs to access resources from a different AWS account.

  {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Resource": "arn:aws:iam::909318972178:role/jacob-aws-apigw-cross-account-role"
  }

2. IAM Role for Task Execution

Key Permissions:

• Pulling Docker Images from ECR (ecr:GetDownloadUrlForLayer, ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:BatchCheckLayerAvailability): These permissions allow the ECS service to pull container images from AWS ECR, which is required to run the task.

  {
      "Action": [
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetAuthorizationToken",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
      ],
      "Effect": "Allow",
      "Resource": "*"
  }

• CloudWatch Logs Access (logs:PutLogEvents, logs:CreateLogStream): These permissions allow the ECS task to write logs to CloudWatch by creating log streams and putting log events into them.

  {
      "Action": [
          "logs:PutLogEvents",
          "logs:CreateLogStream"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:logs:us-east-1:141020838:log-group:traceable-awsapigw-77722ddbacd:*"
  }

3. IAM Role for EventBridge Scheduler

Key Permissions:

• ECS Task Execution (ecs:RunTask): This permission allows EventBridge to trigger the execution of ECS tasks. The resources specified here are the specific ECS task definitions that can be run.

  {
      "Action": [
          "ecs:RunTask"
      ],
      "Effect": "Allow",
      "Resource": [
          "arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-9093963563178-us-east-1-77722ddba6:1",
          "arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-base-us-east-1-77722ddba6:1"
      ]
  }

• IAM Role Passing (iam:PassRole): This permission allows EventBridge to pass the necessary IAM roles (the task execution role and instance role) to the ECS tasks when they are triggered. Without this permission, EventBridge would not be able to assign the correct roles to the tasks it runs.

  {
      "Action": [
          "iam:PassRole"
      ],
      "Effect": "Allow",
      "Resource": [
          "arn:aws:iam::141020914838:role/traceable-task-exec-role-77722ddba6",
          "arn:aws:iam::141020914838:role/traceable-instance-role-77722ddba6"
      ]
  }

accounts

The accounts object has the following values:

Example

The following example explains the configuration. The configuration is for 3 regions across 2 accounts.

accounts = [
  {
    region                 = "us-east-1"
    api_list               = ["625vetvy0f"]
    cross_account_role_arn = ""
    exclude                = false
  },
  {
    region                 = "us-east-2"
    api_list               = []
    cross_account_role_arn = ""
    exclude                = true
  },
  {
    region                 = "us-east-1"
    api_list               = []
    cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
    exclude                = true
  },
  {
    region                 = "us-west-1"
    api_list               = ["example-api/example-stage"]
    cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
    exclude                = true
  }
]

1. The first object is for the region us-east-1 in Account 1. It will capture traffic only from the API  gateway instance with the ID 625vetvy0f in this account. It is assumed that Account 1 is the host account (where the agent instance is deployed).

2. The second object is for the region us-east-2 in Account 1. It will capture traffic from all the API gateway instances in this region in this account.

3. The third object is for the region us-east-1 in Account 2. It will capture traffic from all the API gateway instances in this region in this account. The cross_account_role_arn in this case, it is the ARN of the IAM Role authenticating Account 1 to access required objects in Account 2.

4. The fourth object is for region us-west-1 in Account 2. It will capture traffic only from the stage example-stage of the API gateway instance with the ID example-api in this account.

   Note

   The IAM role ARN is the same as that of case number 3.

Traceable Platform Agent

Configure these parameters only when you wish to deploy the Traceable Platform agent alongside the tracing agent instance. The Traceable Platform Agent will be deployed alongside the agent instance only if the parameter traceable_agent_endpoint of above is configured as http://localhost:5442.

Apply terraform

As a last step to configure, run the following command to apply the terraform changes:

terraform init
terraform apply

Verification

Log into your Traceable Platform account and navigate to API Catalog → Services to view the service name (traceable_service_name) that you configured earlier.

Uninstall

To uninstall, run the following command from the same directory to destroy all the resources created in the installation step:

terraform destroy

Troubleshooting

Loss of traffic

If your setup is experiencing a loss of traffic or observing less amount of traffic, you may try doing the following:

1. Login to the Traceable EC2 instance you created using the Terraform template earlier. The instance's name should be with the prefix traceable-instance-.

   Note

   Make sure to update the security group attached to this instance to allow SSH connection from your machine.

2. After you log in, check the service status of the running service. Enter the following command:

   ActionScriptActionScript

   sudo systemctl status traceable logstash

You may also view the logs generated by Traceable by navigating to the logs directory:

cd /var/traceable/log/aws-api-gateway/
less gateway.err