Thanks to the amazing assistance of AWS Support, I finally was able to install
1. Launch a new EC2 instance:
1) On “Choose AMI” screen, select “Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type” as your AMI.
2) On “Step 2: Choose an Instance Type” screen, I suggest to select a medium or large type (mine is t2.large). I tried to use t2.micro but the performance is not good enough for compiling
3) Click “Review and Launch”, then click “Launch”.
4) Use your key pair to finalize the configuration.
2. Use ssh to connect to your new EC2 instance with your private kay pair.
3. On your new EC2 Amazon Linux instance, please do following things:
1) Install gcc-c++: sudo yum install gcc-c++ -y
2) Install cmake: sudo yum install cmake -y
3) Install python36: sudo yum install python36 -y
4) Install python36-devel: sudo yum install python36-devel -y
4. Install Python dependencies:
1) Install Pillow: sudo python3 -m pip install Pillow
It will also install “PIL” for you.
2) Install face_recognition: sudo python3 -m pip install face_recognition
It will also install “face_recognition_modes”, “numpy”, and “dlib”.
Above modules will be installed to “/usr/local/lib64/python3.6/site-packages/”. Here is my file list for your reference:
—————————————————————————————————
[[email protected] ~]$ ls /usr/local/lib64/python3.6/site-packages/ -l
total 9652
drwxr-xr-x 2 root root 4096 Feb 22 08:08 dlib-19.16.0.egg-info
-rwxr-xr-x 1 root root 9852368 Feb 22 08:08 dlib.cpython-36m-x86_64-linux-gnu.so
drwxr-xr-x 18 root root 4096 Feb 22 08:08 numpy
drwxr-xr-x 2 root root 4096 Feb 22 08:08 numpy-1.16.1.dist-info
drwxr-xr-x 4 root root 4096 Feb 22 08:00 PIL
drwxr-xr-x 2 root root 4096 Feb 22 08:00 Pillow-5.4.1.dist-info
—————————————————————————————————
5. Zip above mentioned “/site-packages” directory and download your zip file.
6. On your local machine, unzip above mentioned zip file. Then make some new folders for each module.
Here is my file structure for your reference:
dlib
└ python
└ dlib.so (I renamed “dlib.cpython-36m-x86_64-linux-gnu.so” to “dlib.so”)
face_recognition
└ python
│ face_recognition (directory and its contents)
└ face_recognition-1.2.3.dist-info (directory and its contents)
face_recognition_models
└ python
│ face_recognition_models (directory and its contents)
└ face_recognition_models-0.3.0.egg-info (directory and its contents)
numpy
└ python
│ numpy (directory and its contents)
└ numpy-1.16.1.dist-info (directory and its contents)
PILPillow
└ python
│ PIL (directory and its contents)
└ Pillow-5.4.1.dist-info (directory and its contents)
7. Zip each “python” directory under above five folders. You will get 5 “python.zip” files. These files are for Lambda layers.
8. Open AWS Lambda console. Create 5 new lambda layers for above zip files. Upload these zip files to each layer correspondingly. Here are some details:
——————————————————————–
Name Runtime
——————————————————————–
PILPillow Python 3.6
dlib Not specified
face_recognition Python 3.6
face_recognition_models Python 3.6
numpy Python 3.6
——————————————————————–
Please note, “python.zip” of “face_recognition_models” is too large (approx. 100.6MB) to be uploaded from console webpage, you have to upload it to a S3 bucket first, mark it public, and then upload to Lambda layer from your S3 bucket.
9. Create a new Lambda function, add above 5 layers to your Lambda function, select Python 3.6 as runtime. Then upload your Python code with a test jpeg picture. Here is my “lambda_function.py” for your reference:
——————————————————————–
import face_recognition
import json
print(“cold start”)
def lambda_handler(event, context):
fileName = “1.jpeg”
print(“start face recognition…”)
image = face_recognition.load_image_file(str(fileName))
encoding = face_recognition.face_locations(image)
print(“face recognition finish…”)
return {
‘statusCode’: 200,
‘body’: json.dumps(encoding[0])
}
——————————————————————–
Please note, in my case, I use “1.jpeg” for testing only.
10. Increase your Lambda function’s memory to about 512 MB. The default 3s timeout is enough for simple request. In my case, the running status are:
——————————————————————–
Duration: 174.37 ms
Billed Duration: 200 ms
Memory Size: 512 MB
Max Memory Used: 400 MB
——————————————————————–
Please note, the memory and timeout should be updated as your functionality needs.
Explanations:
————————————————————————–
1. When
2. I firstly build/install everything on my EC2 with its pre-installed Python 2, but some libraries can’t be invoked by Lambda. Then I switch to Python 3, that’s what we did in step 3 and 4.
3. To build
4. After
——————————————————————————————
import os
directories = os.popen(“find /opt/* -type d –
return {
‘directories’: directories
}
——————————————————————————————
5. Increasing memory allocation is important,
6. Be aware of Lambda total unzipped deployment files (includes your code/dependencies) size limitation, which is 250 MB. If your Lambda function has multiple layers, the total deployment size then is the sum of all referenced layers plus the Lambda itself. Please make sure the total size is within this limitation.
7. In production, the image file should be stored in storage services like S3. The jpeg file I upload to my Lambda function is just for testing.
8. Your functionality may vary from my POC, so you may need to increase the timeout and memory for advance processing.
When you create the 5 zip files, when do you get “dace_recognation” and “face_recognition_models” from? I cant find it on the zip I downloaded.
Hi
I found your article on “installing dlib on aws lambda” very useful. Thank you. However, I faced two issues:
1. While installing PIL, it got installed under /usr/local/lib instead of /usr/local/lib64.
2. I am trying to upload a picture to S3 and enabled a lambda trigger to S3 put event. When I try to read the image using face_recognition.load_image_file or face_encodings, I get following error:
‘dict’ object has no attribute ‘read’: AttributeError
Traceback (most recent call last):
File “/var/task/lambda_function.py”, line 17, in lambda_handler
img = fr.load_image_file(resp)
File “/opt/python/face_recognition/api.py”, line 83, in load_image_file
im = PIL.Image.open(file)
File “/opt/python/PIL/Image.py”, line 2776, in open
fp = io.BytesIO(fp.read())
AttributeError: ‘dict’ object has no attribute ‘read’
My lambda_function.py is as follows:
import json
import numpy as np
from PIL import Image
import PIL.Image
import os
import io
import face_recognition as fr
import boto3
s3=boto3.client(‘s3’)
def lambda_handler(event, context):
for record in event[‘Records’]:
bucket=record[‘s3’][‘bucket’][‘name’]
key = record[‘s3’][‘object’][‘key’]
resp = s3.get_object(Bucket=bucket, Key=key)
img = fr.load_image_file(resp)
imgenc=fr.face_encodings(resp)[0]
print(imgenc)
Do you have any thoughts?
Hi,
Pls ignore my earlier question. I figured it out. Thank you again for an excellent tutorial blog.
Hi. Great post. But I am unable to import face_recognition module. Can you help me solving this issues. I followed the same steps mentioned.
This is an amazing solution for dlib with Lambda. I have been struggling with for a couple of days. Much appreciated.
🙂
Thanks for a detailed blog. However, I seem to have run into an issue.
Just adding the statement
“`
import face_recognition
“`
in my lambda function leads to timeout.
I have tried increasing the memory and timeout but it does not help. If you are in my place, where would you look?
Thanks,
Vivek
Do you have the dlib and face_recognition package files in a lambda layer? When mind failed in the past it was because I was not properly including the lambda layer.
Hello Steven, thank you for this wonderful tutorial! I followed everything, however, the face_recognition package is missing from the site_packages folder. I verified that the library is installed in the same location, yet the folder is simply invisible.
I also notice that it is not present in your tutorial when you list the contents of the site_packages folder. Where did you find the face_recognition folder? Thank you!