AWS traffic mirroring

AWS CloudFormation is a service offered by Amazon Web Services (AWS) that enables users to create and manage AWS resources through templates written in JSON or YAML. It allows users to provision and manage AWS infrastructure as code, which can be useful for automating the deployment and scaling of resources, as well as for managing the configuration of those resources over time.

Traceable provides an AWS Cloud Formation template that automates data collection by using traffic mirroring. Traffic mirroring can be used to capture a copy of the original data from the source network interface without disrupting your existing infrastructure and without adding any latency to your requests. For example, this mirrored data can be used for telemetry, attack detection, and so on. Traceable uses open-source (AWS recommended) Suricata to capture mirrored traffic for further processing. The Cloud Formation template creates an EC2 instance and installs Traceable platform agent and Suricata on it. The topic explains deploying Traceable's platform agent using AWS CloudFormation template. The template also sets up traffic mirroring.

 Traffic mirroring for out-of-band data collection

Before you begin

Make a note of the following before you proceed with the deployment:

• Make sure that you have an AWS account.
• You have an understanding of CloudFormation. For more information, see AWS CloudFormation.
• Keep Traceable's access token handy. It will be used when you configure the CloudFormation stack parameters. You can copy the access token by logging into your Traceable platform and navigating to Administration > Account > Agent Token.

Support matrix

The following is a list of the traffic mirroring sources and traffic types that Traceable supports:

• Traffic mirroring sources
  • Load balancer - ALB, CLB, and NLB
  • Target groups
  • ECS (EC2, FARGATE)
• Supported traffic types
  • HTTP
  • gRPC-web

The following table lists the types of EC2 instances supported. 

Permissions required

The following table lists the permissions required for AWS traffic mirroring using CloudFormation.

• CloudFormation
• EC2
• Logs
• Events
• IAM
• SSM
• Lambda

Installation

Traceable platform agent can be installed using CloudFormation by completing the following steps:

1. Configure traffic mirroring source - This is typically your traffic load-balancer or traffic groups or ECS Cluster
2. Configure traffic mirroring target - Amazon EC2 instance where traceable agent and Suricata will be installed
3. Traceable platform agent configuration

Step 1 - Navigate to CloudFormation template

 Click on the button. Traceable's CloudFormation template is displayed.

Alternatively, you can also download the CloudFormation template from Traceable’s download site. Navigate to Install → traffic-mirroring → cf-template → latest. Download template.yml file. 

• Login to your AWS account. Search for CloudFormation from the search bar.
• On the CloudFormation page, click on Create stack button.

• On the Create stack page, select Upload a template file option in the Specify template section. Upload the template that you downloaded from Traceable’s download site.

Click on Next. The CloudFormation template is displayed. 

Amazon S3 URL - Alternatively, if you wish to use Amazon S3 URL, use https://s3.amazonaws.com/downloads.traceable.ai/install/traffic-mirroring/cf-template/latest/template.yaml.

Step 2 – Configure CloudFormation parameters

Configure the CloudFormation parameters. The parameters are in the following categories:

• Traffic mirroring source configuration
• Traffic mirroring target deployment configuration
• Traceable EC2 instance configuration
• Traceable configuration
• Route53 configuration
• Mirroring session lambda configuration

Traffic mirroring source configuration

• MirrorSourceType - Select the type of source traffic from where traffic has to be mirrored. Possible values are:
  •  LOAD_BALANCER
  •  LOAD_BALANCER_TAGS
  • TARGET_GROUP
  • ECS_CLUSTER
  • MANUAL
• LOAD_BALANCER_TAGS - You can use this if you wish to choose load balancers based on tags. All load balancers that match at least one of the tags provided in the MirrorSource are mirrored. Tag can be either in the form of key=value or key.

  Example: For the following input:
  MirrorSourceType: LOAD_BALANCER_TAGS
MirrorSoure: app=developement, traceable-mirror=true, traceable-app
  All load balancers satisfying at least one of the following conditions will be mirrored:

  • The load balancer has a tag with key app and value development.
  • The load balancer has a tag with key traceable-mirror and value true.
  • The load balancer has a tag with a key traceable-app and any value.
• MirrorSource - Based on the MirrorSourceType selected above, provide a string as follows: 
  • LOAD_BALANCER: A comma-separated list of load balancers.
  • LOAD_BALANCER_TAGS: A comma-separated list of tags.
  • TARGET_GROUP: A comma-separated list of target groups.
  • ECS_CLUSTER: Name of exactly one ECS cluster.
  • MANUAL: Leave this field empty.
  • FARGATE: Leave this field empty.

Traffic mirroring target deployment configuration

• DeployTraceableAgentInECS - Deploy the Traceable Platform agent in an ECS cluster. The default value is false.
• SubnetIds - Select SubnetIds from the drop-down list where you wish to deploy the Traceable Platform agent. If you are selecting a private subnet, make sure that the private subnet is configured to access the public internet.
• AutoscalingMazSize - Defines the maximum number of EC2 instances in the Traceable autoscaling group or ECS tasks in the Traceable ECS cluster.
• AutoscalingTargeValue - Target threshold value in percentage for average CPU and memory utilization in the autoscaling policy of Traceable Instance group or ECS service.

Traceable EC2 instance configuration

Traceable EC2 instance configuration is applicable only when you have set DeployTraceableAgentInECS to false.

• InstanceType- Select the appropriate instance type from the drop-down list. The default value is m4.xlarge.
  Note

  Make sure that the VM instance where Traceable Platform agent will be installed has at least 4 vCPUs and 8 GB RAM.
• CustomInstanceType - Enter a valid instance type name that you wish to use for Traceable instances. Enter the name if it is not present in the drop-down list of InstanceType.
• CustomAMI - Custom AMI ID for the Traceable instance to use. By default, Traceable instances use the latest Amazon Linux AMI.
  Note

  Traceable only supports Ubuntu, CentOS 7, and Amazon Linux 2 AMIs in the CustomAMI field. If you are using an ARM64-based AMI, then make sure that the chosen instance type is also based on ARM64 architecture.
• KeyName - Select the appropriate key name from the drop-down list to enable SSH access to the EC2 instance.

Traceable configuration

• TraceableRefreshToken - Enter Traceable’s access token that you had copied and saved in the Before you begin section.
• TraceableRefreshTokenSecretArn - ARN of secret where refresh token is stored. Key for secret must be refresh_token. For more information on Secrets Manager, see AWS Secrets Manager.
• TraceableEnvironment - Provide the Traceable environment name, for example, QA, dev, production, and so on.
• TraceableServiceName - Provide the service name for mirrored traffic. This is the name that would appear on the Traceable Dashboard for the mirrored traffic.
• TraceableAPIEndpoint - Provide the Traceable endpoint. The default value is api.traceable.ai.

Route53 configuration

Route53 configuration is applicable only when you set MirrorSourceType to FARGATE. By default, Traceable creates a route53 private hosted zone to add an alias record for the Traceable NLB (network load balancer). If you want to use an existing private route53 zone instead, you need to associate the Traceable VPC to the existing zone and provide the Zone ID in Route53ZoneId.

• Route53ZoneName - Defines the route53 private hosted zone created by Traceable. The default value is private.traceable.ai.
• Route53ZoneId - Zone ID for existing route53 private hosted zone to be used for Traceable NLB. The default value is an empty string.
• Route53SubdomainName - Subdomain for route53 alias record created for the Traceable NLB. The default value is mirroring-fargate.

Mirroring session lambda configuration

• MirroringSessionLambdaTimeout - Timeout is the maximum time in seconds that a Lambda function can run. AWS only allows a lambda to run for a maximum of 900 seconds. The default value is 600 seconds. 
• MirroringSessionLambdaInterval - The time interval in minutes at which the lambda function runs periodically. This time interval must be greater than MirroringSessionLambdaTimeout. The default value is 15 minutes. 
• MirrorUnhealthyTargets - Mirror unhealthy targets. If this is set to true, Traceable creates traffic mirror sessions and filter rules for targets even if their health status is not healthy. The default value is false.

Step 3 – Acknowledge capabilities

Acknowledge that the stack might create IAM capabilities along with the capability to create nested stack.

Verification

You can verify a successful installation of Traceable platform agent using CloudFormation by completing the following steps:

1. Log into the Traceable platform.
2. Select TraceableEnvironment from the Environment ()drop-down list.

 This would display the mirrored data on the Traceable platform.

You can also see the list of installed traffic mirroring agents by navigating to Administration → Configuration → Data Collection section and searching for TMM agents.

Troubleshooting

If you do not see any data in Traceable's platform after a few minutes:

1. Make sure that traffic is coming through your load-balancers, source-target groups, and source ECS cluster.
2. Navigate to /var/traceable/log to view the log files. The log files are rotated and are timestamped. View the latest log file. All the stats are printed in the log file at a 5-minute duration.
    {"level":"info","time":"2021-09-03T13:19:13.302Z","message":"Stats", "service":"ext_cap","total":"412","post":"0","get":"387","put":"25", "delete":"0","head":"0","other":"0","20x":"412","30x":"0","40x":"0", "50x":"0","RCother":"0","header_summary_blobs_read":"0", "body_summary_blobs_read":"0","total_blobs_read":"496","queued_work": "0","active_routines":"0","queue_capacity":"5000","http1x":"412", "http2":"0","grpc":"0","errors":"0"}
3. Check whether the correct access token is configured. You can copy the access token by logging into your Traceable platform and navigating to Administration ()>Account > Access token.
4. Check the status of the Traceable and Suricata services by entering the following commands. SSH to the Traceable ec2 instance created using the CloudFormation stack.  

Suricata service

sudo systemctl status suricata

 Traceable service

sudo systemctl status traceable

If you need further assistance, reach out to Traceable support.

Upgrade

To upgrade your current deployment, delete the current CloudFormation stack and rerun the CloudFormation template. For more information on deleting the CloudFormation stack, follow the steps mentioned in the Deleting a stack on the AWS CloudFormation console page.

Uninstall

To uninstall, delete the CloudFormation stack by following the steps mentioned in the Deleting a stack on the AWS CloudFormation console page.