How to Connect AWS Lambdas within VPC to Datadog

Published on April 8, 2024

By Hyuntaek Park

Senior software engineer at Twigfarm

Recently, as part of a Proof of Concept (PoC) at Twigfarm, we have been adapting Datadog to efficiently identify bugs and issues. Our backend is primarily serverless, incorporating elements such as API Gateway, S3, MongoDB Atlas, and numerous Lambda functions spread across different AWS accounts.

While integrating these serverless components with Datadog proved feasible, we encountered difficulties in tracking logs for certain Lambda functions on the Datadog console. Upon investigation, we discovered that the Lambda functions within the VPC were not connecting.

As you might know already, lambda functions inside VPC cannot connect to the internet without additional settings such as NAT gateway or NAT instance.

We addressed this problem using VPC Peering.

Since we had to deal with several Datadog private links, we had some repetitive jobs and they were easy to make mistakes. Be careful.

Prerequisites

For simplicity, consider the setup as follows:

  • A Lambda function resides in the ap-northeast-2 region, within a VPC.
  • Datadog has been configured on datadoghq.com.
  • Another VPC is established in the us-east-1 region, ensuring its IP range does not overlap with the VPC in the ap-northeast-2 region.
  • VPC IP CIDR for ap-northeast-2 is 172.28.0.0/16
  • VPC IP CIDR for us-east-1 is 172.10.0.0/16

Security Group

Create a security group within the us-east-1 VPC that permits traffic from the 172.28.0.0/16, (IP range of ap-northeast-2 VPC) network on port 443.

alt text

VPC Endpoints

The table below lists the PrivateLink service names. Not all endpoints need to be created; I have selected 7 that I intend to use.

Datadog PrivateLink service name and DNS name

  • Select N.Virginia (us-east-1) region in AWS console
  • Go to VPC
  • Select Endpoints
  • Click Create endpoint
  • Input as the following. Make sure input name tag. I used private DN name for the name.

alt text

  • After you click Verify service button, you get to choose VPC, Subnets, and Security groups.
  • Click Create endpoint

al text

Repeat the same process for all the private link services that you want to connect. Here are the results:

Results after creation of EndPoints

Amazon Route 53 Hosted Zones

  • Go to Route 53
  • Click Create hosted zone
  • Fill in Domain name
  • Choose Private h osted zone
  • Choose Region as US East (N. Virginia)
  • Choose your VPC that you created for Datadog integration
  • Press Create hosted zone

create route 53 hosted zone

Repeat the same process for all of the private link services that you want to connect. Here are the results:

hosted zones

Amazon Route 53 A Record

Now we need create an A record for each of the private hosted zones we just created.

  • Choose one of the hosted zones. e.g.) agent-http-intake.logs.datadoghq.com
  • Press Create record
  • Leave the record name part empty
  • Choose A - Routes traffic to an IPv4 address and some AWS resources for Record type
  • Turn on Alias
  • Route traffic to Alias to VPC endpoint and US East (N. Virginia)
  • Carefully choose the VPC endpoint name for the record
  • Press Create records

a record

Again, repeat this for all of the private link services

Edit Amazon Route 53 Hosted Zone

We need to associate VPCs which reside in both us-east-1 and ap-northeast-2.

  • Choose one of the hosted zones. e.g.) agent-http-intake.logs.datadoghq.com
  • Press Edit hosted zone
  • Make sure add regions and the appropriate VPC IDs
  • Press Save changes button

alt text

Again, repeat this for all of the hosted zones. Then we are done with the Route 53 setup.

Create Peering Connection

A VPC peering connection consists of requester and acceptor. In our case we create a requester first in ap-northeast-2 region and acceptor in us-east-1.

  • Go to VPC in us-east-1 region
  • Select and copy VPC ID
  • Change the region to ap-northeast-2
  • Go to Peering connections
  • Press Create peering connection button
  • Fill in two VPC information in ap-northeast-2 and us-east-1

alt text

Now the request has been made. Time to accept the request.

  • Change the region back to us-east-1
  • Make sure you are in VPC -> Peering connections
  • Choose the peering connection that you just created in ap-northeast-2 region
  • Press Action button and choose Accept request
  • Soon the status will be changed to Active.

Status is Active now

Update Route Tables

We need to update Route tables for VPCs in both regions. Since we are in us-east-1 region, let’s do this region first.

  • Go to VPC -> Route tables
  • Choose the route table for the Datadog VPC
  • Press Edit routes and press Add route button
  • Add IP address of ap-northeast-2 region as the destination. In our case, it is 172.28.0.0/16
  • Choose Peering Connection for the target and select the peering connection ID
  • Then press Save changes button

Route tables for acceptor

Updating route table in ap-northeast-2 region is the last step. Repeat the same but choose the destination appropriately.

  • Switch back the the ap-northeast-2 region
  • Go to VPC -> Route tables
  • Choose the route table for the Datadog VPC
  • Press Edit routes and press Add route button
  • Add IP address of ap-northeast-2 region as the destination. In our case, it is 172.10.0.0/16
  • Choose Peering Connection for the target and select the peering connection ID
  • Then press Save changes button

Testing

If you’ve completed the steps so far, the peering connection should now be established. Next, navigate to the Datadog console to verify whether the logs from the Lambda function are visible in a few minutes. However, if your peering connection is not established correctly, you need to check the connection.

The following commands would be useful for you to test the peering connection. Run the commands from EC2 instances in both us-east-1 and ap-northeast-2 regions.

dig +short http-intake.logs.datadoghq.com
nc w1 -vz http-intake.logs.datadoghq.com 443

References