AWS traffic mirroring

AWS CloudFormation is a service offered by Amazon Web Services (AWS) that enables users to create and manage AWS resources through templates written in JSON or YAML. It allows users to provision and manage AWS infrastructure as code, which can help automate the deployment and scaling of resources and manage the configuration of those resources over time.

Traceable provides an AWS Cloud Formation template that automates data collection by using traffic mirroring. Traffic mirroring can capture a copy of the original data from the source network interface without disrupting your existing infrastructure and adding latency to your requests. For example, this mirrored data can be used for telemetry, attack detection, etc. Traceable uses open-source (AWS recommended) Suricata to capture mirrored traffic for further processing. The Cloud Formation template creates an EC2 instance and installs a Traceable platform agent and Suricata. The topic explains deploying Traceable's platform agent using the AWS CloudFormation template. The template also sets up traffic mirroring.

Traffic mirroring for out-of-band data collection

Before you begin

Make a note of the following before you proceed with the deployment:

Make sure that you have an AWS account.
You have an understanding of CloudFormation. For more information, see AWS CloudFormation.
Keep Traceable's access token handy. You will use it when you configure the CloudFormation stack parameters. You can copy the access token by logging into your Traceable platform and navigating to Settings → Account → Agent Token.

Support matrix

The following is a list of the traffic mirroring sources and traffic types that Traceable supports:

Traffic mirroring sources
- Load balancer - ALB, CLB, and NLB
- Target groups
- ECS (EC2, FARGATE)
Supported traffic types
- HTTP
- gRPC-web

The following table lists the types of EC2 instances supported.

EC2 Instance type	Supported instances
Xen-based hypervisor EC2 instance	C4, D2, G3, G3s, H1, I3, M4, P2, P3, R4, X1, and X1e
Virtualized Nitro-based hypervisor EC2 instance	C5, C5a, C5ad, C5d, C5n, C6g, C6gd, D3, D3en, G4, I3en, Inf1, M5, M5a, M5ad, M5d, M5dn, M5n, M5zn, M6g, M6gd, `p3dn.24xlarge`, P4, R5, R5a, R5ad, R5b, R5d, R5dn, R5n, R6g, R6gd, T3, T3a, T4g, high memory (u-*), X2gd, and z1d.

Permissions required

You can use the following JSON to create an IAM policy which is sufficient to install Traceable AWS traffic-mirroring templates:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "cloudformation",
      "Effect": "Allow",
      "Action": [
        "cloudformation:Get*",
        "cloudformation:Detect*",
        "cloudformation:Describe*",
        "cloudformation:List*",
        "cloudformation:CreateStack",
        "cloudformation:DeleteStack",
        "cloudformation:RollbackStack",
        "cloudformation:UpdateStack",
        "cloudformation:CancelUpdateStack",
        "cloudformation:ContinueUpdateRollback",
        "cloudformation:CreateChangeSet",
        "cloudformation:DeleteChangeSet",
        "cloudformation:ExecuteChangeSet",
        "cloudformation:EstimateTemplateCost",
        "cloudformation:RecordHandlerProgress",
        "cloudformation:SignalResource",
        "cloudformation:UpdateTerminationProtection",
        "cloudformation:ValidateTemplate",
        "cloudformation:TagResource",
        "cloudformation:UntagResource"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ec2",
      "Effect": "Allow",
      "Action": [
        "ec2:Describe*",
        "ec2:Get*",
        "ec2:AllocateAddress",
        "ec2:AssociateAddress",
        "ec2:DisassociateAddress",
        "ec2:ReleaseAddress",
        "ec2:MonitorInstances",
        "ec2:RunInstances",
        "ec2:RebootInstances",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances",
        "ec2:AttachNetworkInterface",
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:DetachNetworkInterface",
        "ec2:CreateSecurityGroup",
        "ec2:DeleteSecurityGroup",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:UpdateSecurityGroupRuleDescriptionsEgress",
        "ec2:UpdateSecurityGroupRuleDescriptionsIngress",
        "ec2:CreateTrafficMirrorFilter",
        "ec2:DeleteTrafficMirrorFilter",
        "ec2:ModifyTrafficMirrorFilterNetworkServices",
        "ec2:CreateTrafficMirrorFilterRule",
        "ec2:DeleteTrafficMirrorFilterRule",
        "ec2:ModifyTrafficMirrorFilterRule",
        "ec2:CreateTrafficMirrorSession",
        "ec2:DeleteTrafficMirrorSession",
        "ec2:ModifyTrafficMirrorSession",
        "ec2:CreateTrafficMirrorTarget",
        "ec2:DeleteTrafficMirrorTarget",
        "ec2:CreateVolume",
        "ec2:DeleteVolume",
        "ec2:DetachVolume",
        "ec2:ModifyVolume",
        "ec2:AssociateIamInstanceProfile",
        "ec2:DisassociateIamInstanceProfile",
        "ec2:ReplaceIamInstanceProfileAssociation",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyInstanceMetadataOptions",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:ModifySecurityGroupRules",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:CreateLaunchTemplate",
        "ec2:DeleteLaunchTemplate",
        "ec2:ModifyLaunchTemplate",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:DeleteLaunchTemplateVersions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "fullAccess",
      "Effect": "Allow",
      "Action": [
        "application-autoscaling:*",
        "ecs:*",
        "elasticloadbalancing:*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "events",
      "Effect": "Allow",
      "Action": [
        "events:List*",
        "events:DescribeRule",
        "events:DeleteRule",
        "events:DisableRule",
        "events:EnableRule",
        "events:PutRule",
        "events:PutTargets",
        "events:RemoveTargets",
        "events:TagResource",
        "events:UntagResource"
      ],
      "Resource": "*"
    },
    {
      "Sid": "iam",
      "Effect": "Allow",
      "Action": [
        "iam:Get*",
        "iam:List*",
        "iam:CreateInstanceProfile",
        "iam:DeleteInstanceProfile",
        "iam:CreatePolicy",
        "iam:CreatePolicyVersion",
        "iam:DeletePolicy",
        "iam:DeletePolicyVersion",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:UpdateRole",
        "iam:UpdateRoleDescription",
        "iam:AttachRolePolicy",
        "iam:DetachRolePolicy",
        "iam:DeleteRolePolicy",
        "iam:PutRolePolicy",
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:UpdateAssumeRolePolicy",
        "iam:TagInstanceProfile",
        "iam:UntagInstanceProfile",
        "iam:TagPolicy",
        "iam:UntagPolicy",
        "iam:TagRole",
        "iam:UntagRole",
        "iam:TagUser",
        "iam:UntagUser"
      ],
      "Resource": "*"
    },
    {
      "Sid": "iamPassRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": [
			"application-autoscaling.amazonaws.com",
			"autoscaling.amazonaws.com",
			"ec2.amazonaws.com",
            "ecs-tasks.amazonaws.com",
            "events.amazonaws.com",
            "lambda.amazonaws.com"
          ]
        }
      }
    },
    {
      "Sid": "lambda",
      "Effect": "Allow",
      "Action": [
        "lambda:Get*",
        "lambda:List*",
        "lambda:CreateFunction",
        "lambda:DeleteFunction",
        "lambda:InvokeFunction",
        "lambda:AddPermission",
        "lambda:RemovePermission",
        "lambda:UpdateFunctionCode",
        "lambda:UpdateFunctionConfiguration",
        "lambda:TagResource",
        "lambda:UntagResource"
      ],
      "Resource": "*"
    },
    {
      "Sid": "logs",
      "Effect": "Allow",
      "Action": [
        "logs:Describe*",
        "logs:Get*",
        "logs:List*",
        "logs:CreateLogStream",
        "logs:CreateLogGroup",
        "logs:DeleteLogGroup",
        "logs:FilterLogEvents",
        "logs:PutLogEvents",
        "logs:PutRetentionPolicy",
        "logs:TagLogGroup",
        "logs:UntagLogGroup",
        "logs:TagResource",
        "logs:UntagResource"
      ],
      "Resource": "*"
    },
    {
      "Sid": "route53",
      "Effect": "Allow",
      "Action": [
        "route53:Get*",
        "route53:List*",
        "route53:AssociateVPCWithHostedZone",
        "route53:DisassociateVPCFromHostedZone",
        "route53:ChangeResourceRecordSets",
        "route53:ChangeTagsForResource",
        "route53:CreateHostedZone",
        "route53:DeleteHostedZone",
        "route53:UpdateHostedZoneComment"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ssm",
      "Effect": "Allow",
      "Action": [
        "ssm:Get*",
        "ssm:Describe*",
        "ssm:List*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "autoscaling",
      "Effect": "Allow",
      "Action": [
        "autoscaling:Describe*",
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup",
        "autoscaling:AttachLoadBalancers",
        "autoscaling:DetachLoadBalancers",
        "autoscaling:AttachLoadBalancerTargetGroups",
        "autoscaling:DetachLoadBalancerTargetGroups",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:DeletePolicy",
        "autoscaling:ExecutePolicy",
        "autoscaling:PutScalingPolicy",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:DeleteTags"
      ],
      "Resource": "*"
    }
  ]
}

Installation

Traceable platform agent can be installed using CloudFormation by completing the following steps:

Configure traffic mirroring source - This is typically your traffic load-balancer or traffic groups or ECS Cluster
Configure traffic mirroring target - Amazon EC2 instance where traceable agent and Suricata will be installed
Traceable platform agent configuration

Step 1 - Navigate to the CloudFormation template

Click on the button. Traceable's CloudFormation template is displayed.

Alternatively, you can download the CloudFormation template from Traceable’s download site. Navigate to Install → traffic-mirroring → cf-template → latest. Download template.yml file.

Login to your AWS account. Search for CloudFormation from the search bar.
On the CloudFormation page, click on the Create stack button.

On the Create stack page, select the Upload a template file option in the Specify template section. Then, upload the template you downloaded from Traceable’s download site.

Click Next. The CloudFormation template is displayed.

Amazon S3 URL - Alternatively, if you wish to use the Amazon S3 URL, use https://s3.amazonaws.com/downloads.traceable.ai/install/traffic-mirroring/cf-template/latest/template.yaml

Step 2 – Configure CloudFormation parameters

Configure the CloudFormation parameters. The parameters are in the following categories:

Traffic mirroring source configuration

MirrorSourceType — Select the source traffic type from where traffic must be mirrored. Possible values are:
- LOAD_BALANCER
- LOAD_BALANCER_TAGS
- TARGET_GROUP
- ECS_CLUSTER
- MANUAL
  Note
  When you use this option, the template only creates a Traceable Platform agent instance. The rest of configurations need to be completed manually.
- FARGATE
LOAD_BALANCER_TAGS — You can use this to choose load balancers based on tags. All load balancers that match at least one of the tags provided in the MirrorSource are mirrored. The tag can be either in the form of key=value or key.
Example: For the following input:
MirrorSourceType: LOAD_BALANCER_TAGS
MirrorSoure: app=developement, traceable-mirror=true, traceable-app
All load balancers satisfying at least one of the following conditions will be mirrored:
- The load balancer has a tag with the key app and value development.
- The load balancer has a tag with the key traceable-mirror and value true.
- The load balancer has a tag with a key traceable-app and any value.
MirrorSource — Based on the MirrorSourceType selected above, provide a string as follows:
- LOAD_BALANCER: A comma-separated list of load balancers.
- LOAD_BALANCER_TAGS: A comma-separated list of tags.
- TARGET_GROUP: A comma-separated list of target groups.
- ECS_CLUSTER: Name of exactly one ECS cluster.
- MANUAL: Leave this field empty.
- FARGATE: Leave this field empty.
MirrorSourceVpcId — A comma-separated list of VPC IDs of the mirrored source. This is only applicable when MirrorSourceType is "LOAD_BALANCER_TAGS."

Traffic mirroring target deployment configuration

DeployTraceableAgentInECS — Deploy the Traceable Platform agent in an ECS cluster. The default value is false.
SubnetIds — Select SubnetIds from the drop-down list where you wish to deploy the Traceable Platform agent. If you are selecting a private subnet, make sure that the private subnet is configured to access the public internet.
AutoscalingMaxSize — Defines the maximum number of EC2 instances in the Traceable autoscaling group or ECS tasks in the Traceable ECS cluster.
AutoscalingTargetValue — Target threshold value in percentage for average CPU and memory utilization in the autoscaling policy of Traceable Instance group or ECS service.

Traceable EC2 instance configuration

Traceable EC2 instance configuration is applicable only when you have set DeployTraceableAgentInECS to false.

InstanceType- Select the appropriate instance type from the drop-down list. The default value is m4.xlarge.
Note
Make sure that the VM instance where the Traceable Platform agent will be installed has at least four vCPUs and 8 GB RAM.
CustomInstanceType - Enter a valid instance name you wish to use for Traceable instances. Enter the name if it is not in the drop-down list of InstanceType.
CustomAMI - Custom AMI ID for the Traceable instance to use. By default, Traceable instances use the latest Amazon Linux AMI.
Note
Traceable only supports Ubuntu, CentOS 7, and Amazon Linux 2 AMIs in the CustomAMI field. If you are using an ARM64-based AMI, make sure that the chosen instance type is also based on ARM64 architecture.
KeyName - Select the appropriate key name from the drop-down list to enable SSH access to the EC2 instance.

Traceable configuration

TraceableRefreshToken - Enter Traceable’s access token that you had copied and saved in the Before you begin section.
TraceableRefreshTokenSecretArn - ARN of secret where refresh token is stored. The key for the secret must be refresh_token. For more information on Secrets Manager, see AWS Secrets Manager.
TraceableEnvironment — Provide the Traceable environment name, such as, QA, dev, production, and so on.
TraceableServiceName—Provide the service name for mirrored traffic. This name will appear on the Traceable Dashboard for the mirrored traffic.
TraceableAPIEndpoint — Provide the Traceable endpoint. The default value is api.traceable.ai.

Route53 configuration

Route53 configuration is applicable only when you set MirrorSourceType to FARGATE. By default, Traceable creates a route53 private hosted zone to add an alias record for the Traceable NLB (network load balancer). If you want to use an existing private route53 zone instead, you need to associate the Traceable VPC to the existing zone and provide the Zone ID in Route53ZoneId.

Route53ZoneName - Defines the route53 private hosted zone created by Traceable. The default value is private.traceable.ai.
Route53ZoneId - Zone ID for existing route53 private hosted zone to be used for Traceable NLB. The default value is an empty string.
Route53SubdomainName - Subdomain for route53 alias record created for the Traceable NLB. The default value is mirroring-fargate.

Mirroring session lambda configuration

MirroringSessionLambdaTimeout - Timeout is the maximum time in seconds that a Lambda function can run. AWS only allows a lambda to run for a maximum of 900 seconds. The default value is 600 seconds.
MirroringSessionLambdaInterval - The time interval in minutes at which the lambda function runs periodically. This time interval must be greater than MirroringSessionLambdaTimeout. The default value is 15 minutes.
MirrorUnhealthyTargets - Mirror unhealthy targets. If this is set to true, Traceable creates traffic mirror sessions and filter rules for targets even if their health status is unhealthy. The default value is false.

Step 3 – Acknowledge capabilities

Acknowledge that the stack might create IAM capabilities along with the capability to create nested stack.

Note
Traceable's CloudFormation template takes care of autoscaling.

Verification

You can verify a successful installation of the Traceable platform agent using CloudFormation by completing the following steps:

Log into the Traceable platform.
Select TraceableEnvironment from the Environment ( ) drop-down list.

This would display the mirrored data on the Traceable platform.

Note
It may take a few minutes for the data to show up the first time in the Traceable platform.

You can also see the list of installed traffic mirroring agents by navigating to the Settings → Configuration → Data Collection section and searching for TMM agents.

Troubleshooting

If you do not see any data in Traceable's platform after a few minutes:

Make sure that traffic is coming through your load-balancers, source-target groups, and source ECS cluster.
Navigate to /var/traceable/log to view the log files. The log files are rotated and are timestamped. View the latest log file. All the stats are printed in the log file at a 5-minute duration.
{"level":"info","time":"2021-09-03T13:19:13.302Z","message":"Stats", "service":"ext_cap","total":"412","post":"0","get":"387","put":"25", "delete":"0","head":"0","other":"0","20x":"412","30x":"0","40x":"0", "50x":"0","RCother":"0","header_summary_blobs_read":"0", "body_summary_blobs_read":"0","total_blobs_read":"496","queued_work": "0","active_routines":"0","queue_capacity":"5000","http1x":"412", "http2":"0","grpc":"0","errors":"0"}
Check whether the correct access token is configured. You can copy the access token by logging into your Traceable platform and navigating to Settings → Account → Access token.
Enter the following commands to check the status of the Traceable and Suricata services. Then, SSH to the Traceable ec2 instance created using the CloudFormation stack.

Suricata service

sudo systemctl status suricata

Traceable service

sudo systemctl status traceable

If you need further assistance, reach out to Traceable support.

Upgrade

To upgrade your current deployment, delete the CloudFormation stack and rerun the CloudFormation template. For more information on deleting the CloudFormation stack, follow the steps mentioned in the Deleting a stack on the AWS CloudFormation console page.

Uninstall

To uninstall, delete the CloudFormation stack by following the steps mentioned in the Deleting a stack on the AWS CloudFormation console page.