How to install custom packages to AWS Lambda — with EC2

5 min readApr 11, 2021

Lambda is a important service offered by AWS which comes in handy while fully automated and server-less pipelines in AWS cloud.

With AWS Lambda we can deploy our own python scripts and program its execution based on occurrence of a certain event (like file landing in S3, etc.)

However, one of the limitations of this service is that we do not get the liberty to use any python package while scripting. AWS Lambda comes with only a python packages, and we are not allowed to import packages other than these. Pandas, numpy, etc. are some of the popular packages which are not preinstalled into AWS Lambda service.

Thus in this blog post we will see how get python packages of our choice to work with AWS Lambda, and we will be doing this with the help of AWS EC2 and AWS cli.

So lets get started!

For the sake of this demo, I will be installing three packages and it is worthy to note that, all the three packages are pip install able:
1) Pandas
2) numpy
3) psycopg2

Step 1a: To setup an EC2 instance navigate to the AWS EC2 service and click on launch instances.

Step1b: Search and select the following instance.

Step1c: Just go with the default settings and click Review and Launch >> Launch.

Next you will be prompted to select a keypair, this is important for SSH to our EC2 instance (I will use PuTTy here).

I already have a keypair configured so I will simply select that and click “Launch Instances”

To connect with new key-pair see steps here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Step1c: Configure Security group.

Once the instance is created select the check box and click on “Security” and then “Security Groups”

Next, on the “Inbound rules” tab select “edit inbound rules”

Select, Type >> All traffic,
Source >> Any where
And “Save rules”

Step1d: Now connect to our newly created instance using PuTTY

From Connect >> SSH client >> copy the “Public DNS” of our EC2 instance

Connect to our EC2 instance using PuTTY by entering the “Public DNS” as the “Host Name” and select the “dhirajusescases.ppk” file in “Auth”

Then login as “ubuntu”

Step 2a: Now we need to install the following:
1) pip3
2) zip
3) virtual env
4) aws cli

Type the following commands line by line

sudo apt update
sudo apt install python3-pip
sudo apt install zip
sudo apt install python3-virtualenv
sudo apt install awscli
aws configure

Step2c: Once the awscli is installed we need to configure it. For that we need the following:
1) AWS access key ID (get from your AWS account)
2) AWS security key (get from your AWS account)
3) AWS region (this may be any region of our interest [ap-south-1])
4) Default output format (keep it [None])

Step 3a: Type the following commands line by line

mkdir folder
cd folder
virtualenv v-env
source v-env/bin/activate [“better to go with tabs”]
pip3 install pandas
pip3 install numpy
git clone https://github.com/jkehler/awslambda-psycopg2
mv awslambda-psycopg2/psycopg2–3.8 v-env/lib/python3.8/site-packages/
mv v-env/lib/python3.8/site-packages/psycopg2–3.8 v-env/lib/python3.8/site-packages/psycopg2
deactivate

if you get an error in the third line “No such directory” then

(1) please check the python version

(2) please check the path to site-packages, by manually going into each folder till you reach site-packages

Step3b: Run the following commands line by line

mkdir python
cd python
cp -r ../v-env/lib/python3.8/site-packages/* . ["python version!"]
cd ..
zip -r mypackages_layer.zip python
aws lambda publish-layer-version --layer-name mypackages --zip-file fileb://mypackages_layer.zip --compatible-runtimes python3.8

Once done, you must see the following json on the command line.