HAProxy

  • Updated on Feb 11, 2022
  • Published on Feb 7, 2022
  • 7 minute(s) read
Prev Next

HAProxy is a reverse-proxy offering high availability, load balancing, and proxy services for TCP and HTTP-based applications that spreads requests across multiple servers. Traceable supports HAProxy deployments by using ext_cap in Traceable agent. Traceable takes advantage of HAProxy's Stream Processing Offload Engine (SPOE) Golang agent library. SPOE is a filter that talks to servers managed by Stream Processing Offload Agent (SPOA) to offload the stream processing. Following is a simple high-level deployment diagram to deploy Traceable agent with HAProxy. The topic details information about installing Traceable agent on a virtual machine (VM).

Configuring HAProxy consists of two steps:

  1. Deploy and configure Traceable agent
  2. Configure HAProxy

Before you begin

  • The deployment is supported for HAProxy 2.4.x and later.
  • Make sure that you understand HAProxy configuration, for example HAProxy configuration (*.cfg) file structure.
  • Make sure that port number 5444 is available.
  • The setup uses frontend-config-snippet.cfg, backend-config-snippet.cfg, and backend.cfg. The backend.cfg defines Traceable platform agent as a backend that the HAProxy can connect to as an SPOE agent. These files are available in the Appendix section of this topic. You can copy-paste these files to configure HAProxy.

Step 1  - Deploy HAProxy on a virtual machine

The first step to deploy HAProxy on a virtual machine is to deploy and configure Traceable agent. For more information on deploying Traceable agent on a virtual machine, see Virtual Machine. Make a note of the following points to customize Traceable agent for HAProxy:

  • By default Traceable agent should be configured to start SPOE server and listen on port number 5444.
  • The default service name is ext_cap, however, if you wish to change it something else. To configure a different service name, configure the value in service_name inside the ext_cap section.
  • Check whether the values inside ha_proxy section are configured. The default values configured are:
    • enabled - Set to true
    • request_capture_message_name - Set to traceable-reqcap
    • response_capture_message_name - Set to traceable-rescap

Following is sample file:

global:
  # ...
  haproxy_spoe_server:
    endpoint: "0.0.0.0:5444"
# ...
ext_cap:
  # ...
  service_name: ext_cap
  # ...
  ha_proxy:
    enabled: true
    # Request Capture message name
    request_capture_message_name: traceableai-reqcap
    # Response Capture message name
    response_capture_message_name: traceableai-rescap

Step 2 - Configure HAProxy

Traceable provides a traceable.conf file that you can download from Traceable's download site. Navigate to install > haproxy > latest to download the latest files. The download site also provides HAProxy configuration snippets for frontend and backend. Add these frontend and backend snippets to your haproxy.config file. The traceable.conf file defines the configuration for the SPOE agent and the snippets contain config to be added to the frontends and backends you wish to instrument. Complete the following steps:

  1. Download the traceable.conf file as mentioned above. Make a note of files listed in the Appendix section.
  2. Create the directory to put traceable.conf in /usr/local/etc/traceable. Make sure the HAProxy can access this folder.
  3. Copy the traceable.conf file in /usr/local/etc/traceable folder.

Configure Traceable's SPOE filter

Configure Traceable's SPOE filter in your haproxy.cfg file. This involves adding the contents of frontend-config-snippet.cfg to each frontend that you wish to monitor and backend-config-snippet.cfg to each backend to correlate with. This is required to capture the request and response information that travels through your HAProxy. Make sure to configure the haproxy.cfg file for both frontends and backends. Also, add the contents of backend.cfg at the bottom of your haproxy.cfg file in order for your HAProxy to connect to Traceable Platform Agent. 

Following is an example haproxy.cfg file. The section that you need to configure are marked by #**TRACEABLE CONFIG**

global  
        daemon

        # The capture size is also limited by haproxy's bufsize
        # so increase from the default of 16384 to support larger 
        # bodies
        tune.bufsize 131072
        ssl-server-verify none
  
defaults  
        log global  
        mode    http  
        timeout connect 5000  
        timeout client 5000  
        timeout server 5000  
  
frontend main  
        bind *:80  
        # Copied from frontend-config-snippet.cfg in appendix section
        # ********** TRACEABLE CONFIG ***********
        # Load traceable's config file. The customer will download it
        # from https://downloads.traceable.ai/install/haproxy/latest
        filter spoe engine traceable config /usr/local/etc/traceable/traceable.conf

        # add these two lines so that we can capture the body of http requests and responses
        # The line below means wait for the body for at most 10seconds or until you get 128K of the body.
        http-request wait-for-body time 10s at-least 128k
        http-response wait-for-body time 10s at-least 128k

        # instruct haproxy to make an RPC call to the traceable agent on request
        # and responses
        http-request send-spoe-group traceable traceableai-reqcap-grp
        http-response send-spoe-group traceable traceableai-rescap-grp

        # deny requests that the traceable-agent tells us to block 
        http-request deny deny_status 403 if { var(sess.traceable.block_request) -m bool }
        
        # *********** END OF TRACEABLE CONFIG *************

        default_backend app  

backend app  
        balance roundrobin  
        mode http  
        server srv1 web:5000
        
        # Copied from backend-config-snippet.cfg in appendix section
        # ********** TRACEABLE CONFIG ***********
        # add the trace context headers generated by traceable agent to the request sent
        # upstream.
        #
        # This example uses w3 trace-context; if b3 or a different propagation format is used,
        # the line needs to be modified accordingly.
        http-request set-header traceparent %[var(sess.traceable.trace_context.traceparent)]

        # *********** END OF TRACEABLE CONFIG *************

# Configuring a special backend for the traceable SPOE agent
# Copied from backend.cfg in appendix section
# ********** TRACEABLE CONFIG ***********
backend traceable
    mode tcp
    balance roundrobin
    timeout connect 5s  # greater than hello timeout
    timeout server  3m  # greater than idle timeout
    option spop-check

    # set up connectivity to the traceable agent. using localhost since it's a VM deployment
    server agent localhost:5444 check
# *********** END OF TRACEABLE CONFIG *************

Restart HAProxy

Restart HAProxy for the changes to apply.

Verify Deployment

Complete the following steps to verify a successful deployment:

  1. Make sure that HAProxy restarts successfully without any errors.
  2. Send some traffic to the frontends that you have configured with Traceable config in the haproxy.cfg.
  3. Navigate to Traces on Traceable platform and verify that the traces of service that you configured is visible in the Traceable UI.

You can also check the agent version by navigating to Settings () > Data Collection > Tracing agent.

Upgrade

Complete the following steps to upgrade:

  1. Upgrade Traceable agent for virtual machines.
  2. Navigate to install > haproxy > latest on Traceable's download site and download traceable.conf file.
  3. Make a note of the following files from the Appendixsection:
    • frontend-config-snippet.cfg
    • backend-config-snippet.cfg 
    • backend.cfg
  4. Copy the new traceable.conf file to where you copied it during installation, that is in /usr/local/etc/traceable folder.
  5. Follow the same installation instructions in the Install section to update your haproxy.cfg with the snippets for your monitored frontends and backends and the backend.cfg
  6. Restart HAProxy for the new changes to be picked up

Uninstall

Complete the following steps to uninstall Traceable's HAProxy agent:

  1. Undo the changes made to your haproxy.cfg during the installation process. This involves:
    1. Deleting the frontend and backend config snippets from haproxy.cfg file. 
    2. Delete the backend traceable section.
  2. Delete traceable.conf file from the folder where you copied it during the installation, for example, /usr/local/etc/traceable.
  3. Restart HAProxy for the new changes to apply. 
  4. Uninstall Traceable Agent.

Troubleshooting

Following are a few troubleshooting steps:

  • haproxy.cfg file - Make sure that haproxy.cfg file is well formed. HAProxy will not start if there are errors in the haproxy.cfg file. 
  • traceable.conf file location - If during the installation process, you are not able to create /usr/local/etc/traceable, then:
    1. Copy the traceable.conf file to another folder that HAProxy has access to. For example, you can copy the traceable.conf file to the same location as haproxy.cfg file, that is, /usr/local/etc/haproxy.
    2. If you change the location of traceable.conf file, change the path of traceable.conf file in frontend-config-snippet.cfg before copying it to haproxy.cfg. Configure the path of traceable.conf as a value for filter spoe engine traceable.
  • Traces - If you are not seeing traces for a frontend you wish to monitor, check your haproxy.cfg to make sure that it contains the frontend-config-snippet and that it has the correct path for traceable.conf. Also, make sure that  check that haproxy.cfg file has traceable backend, that is, backend traceable section.
  • Traceable agent and HAProxy host - It is expected that since this a VM installation, the TA and HAProxy are running on the same host. For the same reason, backend traceable, the configuration for Traceable agent host and port is server agent localhost:5444 check. If Traceable agent is running on a different host, change localhost in that line to the host name of where Traceable agent is installed. It is not recommended to run Traceable agent on a different host.

Appendix

The following three files are used as part of the configuration.

frontend-config-snippet.cfg

        # ********** TRACEABLE CONFIG ***********
        # Load traceable's config file. The customer will download it
        # from https://downloads.traceable.ai/install/haproxy/latest
        filter spoe engine traceable config /usr/local/etc/traceable/traceable.conf

        # add these two lines so that we can capture the body of http requests and responses
        # The line below means wait for the body for at most 10seconds or until you get 128K of the body.
        http-request wait-for-body time 10s at-least 128k
        http-response wait-for-body time 10s at-least 128k

        # instruct haproxy to make an RPC call to the traceable agent on request
        # and responses
        http-request send-spoe-group traceable traceableai-reqcap-grp
        http-response send-spoe-group traceable traceableai-rescap-grp

        # deny requests that the traceable-agent tells us to block 
        http-request deny deny_status 403 if { var(sess.traceable.block_request) -m bool }
        
        # *********** END OF TRACEABLE CONFIG *************

backend-config-snippet.cfg

        # ********** TRACEABLE CONFIG ***********
        # add the trace context headers generated by traceable agent to the request sent
        # upstream.
        #
        # This example uses w3 trace-context; if b3 or a different propagation format is used,
        # the line needs to be modified accordingly.
        http-request set-header traceparent %[var(sess.traceable.trace_context.traceparent)]

        # *********** END OF TRACEABLE CONFIG *************

backend.cfg

# ********** TRACEABLE CONFIG ***********
backend traceable
    mode tcp
    balance roundrobin
    timeout connect 5s  # greater than hello timeout
    timeout server  3m  # greater than idle timeout
    option spop-check

    # set up connectivity to the traceable agent. using localhost since it's a VM deployment
    server agent localhost:5444 check
# *********** END OF TRACEABLE CONFIG *************