Building Lambda Functions with MongoDB Atlas via VPC Peering
I don’t know how many of you took your networking classes in undergrad seriously but I didn’t. You generally don’t take it, until you need it. This was exactly one of those cases. One of the best in class in Networking, I failed miserably at understanding Virtual Private Cloud and always ran away from it, until one day I was left with no option.
It all started with a requirement to connect my production database to Lambda function we had just deployed. Which was easy except for the part that, in the Network Access tab, I had to give 0.0.0.0/0 for my production database as lambdas don’t have a specific IP address. Any decent security practice would disallow that because now anybody could access my production database with the right credentials from anywhere in the world and whole everything to ransom or delete everything for fun. If you have read my blogs before you might have come across this article.
Which illustrates my tryst with security. I sure as hell was not going to take chance on this one. So I got determined to put my lambda inside a VPC and connect it to Mongo.
A tiny bit of googling on the topic will land you here.
I followed the link to the bits and of course, it didn’t work. Remember
AWS, is a death by thousand cuts.
So if you don’t understand something fully, it’s really difficult to get it to work in AWS. Especially a twisted concept like VPC, which has lots of networking stuff to it. So I decided to put an end to my misery.
One fine morning I sat down and started looking at AWS VPC Videos, to understand what the hell was going on. I watched a lot of videos and created a special playlist of videos that helped me learn and wrap my head around this. These lectures are around 30 mins, except for the last 2. Should give you a head start as well.
Once equipped with knowledge of what a VPC is and how it works. I set out on the mission. There was still one unknown called VPC Peering. What the hell was that? I read about the fact that MongoDB Atlas clusters live inside of a VPC and when two friends VPC talk to each it’s called VPC Peering. Simple right.
So the task was simple. Create my own demon, I mean VPC which I can control, through the knowledge I just acquired about Route tables, Subnets, and Internet Gateways and connect it to Atlas Cluster so that they can both talk to each other.
Let me describe all those moving parts in a concise manner so that we can understand everything better.
Internet Gateway
VPC is designed to be hidden away from the Public Internet. Internet Gateway is how we make it discoverable.
Route Table
Glorified Key Value Pair, determines where traffic to be directed. Like in the diagram, when lambda tries to connect to the database, it goes out with the address of database to the route table, which directs us to go via the VPC Peering to the database.
Network Access Control List (NACL)
Optional security layer for the VPC acts as a firewall for controlling traffic in and out of subnets. NACL List has rules and rules with the lowest rule number is applied.
Security Groups(SGs)
I have been using SGs for a long time but two things I didn’t know about them
- Work on instance level rather than subnet level.
- All rules of security groups are evaluated from all SGs associated with that instance. Everything is denied until it’s explicitly allowed.
NAT Gateways
Do opposite of Internet Gateways, they cannot accept anything from outside but can help access the internet from within the VPC, they come with an elastic IP. They live inside of a subnet. Use NAT Gateways if lambda needs to call an external API(Send SMS, Send Push Notification, Send events)
The basic principle of learning
Everything look tough, until you accomplish it.
I am sitting here telling you like a know-all. But in reality, I learned this stuff like a day ago. Before that, every second it seemed like I won’t be able to learn this stuff. I am writing this blog post so that somebody else is not stuck like me.
So let’s go through the diagram, it’s annotated so that you know what I am talking about.
HTTP Request from Lambda comes through via Internet Gateway, which goes to route table 1, which goes to Subnet 3
Subnets within a VPC are configured to allow communication by default. The route table entry 10.0.0.0/16 → Local means all subnets can route to all other subnets.
So that’s how your HTTP request reaches either Lambda 1 or Lambda 2 which accepts the request. Let’s say Lambda 2 accepts the request, Now Lambda 2 wants to connect to the database. Lambda 2 goes to the route table and via the VPC peering reaches Atlas Cluster and thus the connection is established.
After say you are done with the final DB operation and want to communicate the result to the user via SMS. So you call the SMS API, the connection goes to subnet 3 from there to NAT Gateway, from there to route table 1 and from there to the internet gateway.
Response from API comes to the internet gateway, to route table 1 to subnet 3 and then to subnet 2.
Post which API sends back the response from the same path it came.
Seems easy right. It is, now read all of it over again and try it out sometimes.
The sucker is you can only do this on a production cluster of MongoDB Atlas. There are a lot of free credits lying around for MongoDB. I can hook you up some, in case you need one. Just DM me on twitter.
I hope you never have to try this on your own, I got a lot of help from AWS Business support, which not everybody has access to. But in case you do, they are nice folks and help you out every time you get into trouble.
Feel free to slide into DMs for further clarifications.
