Sideproject Story: Building Face Authentication in Lisbon

The problem

Typing in username and password every time you want to access the system can be tiresome. Ideally, you want to use some password manager. At Mews, we love 1Password! And we wish also our clients would use a similar solution. Unfortunately, we have noticed there are cases of freely accessible unlocked systems in the hotel lobby or that multiple users are at the computer while signed in with a single account.

Former is a security issue and, with the latter, it’s difficult to keep track of who did what and run some investigation in our action logs when we cannot distinguish the actual users.

The solution

We decided to solve this with a lock screen giving the users an easy way to switch between accounts and there were several steps we decided to work on:

• user can lock their account and see list of other locked accounts on the sign in screen, they can easily choose their account, fill in just the password and get back to the system,
• user can set up an optional PIN code that can be used instead of the password when unlocking the system,
• and, side by side with the PIN code, users can enable face recognition and use their face to unlock the system.

We have built our Mews platform on top Azure infrastructure. It has many great services which also includes Cognitive Services with Face API.

The backend part of our solution was mostly about implementing Azure Face API and following the process of setting up person groups, adding persons, their images and finally training the model. Check the how-to guide.

The way how it is designed on Azure side is that you have person groups where you organise your persons between whom you want to match your image data. So the question was whether to use single person group for all our system users or use partitioning and split the users based on selected key. Using single group would be easier to implement, since we can keep the groupId in the application configuration, but we weren’t sure how the service actually behaves when the number of persons growth fast. With new person added or their training images updated, the service re-trained the whole group which suggest that the training time grows too and eventually we’d need to implement a queue of re-train requests.

We did not get to confirming verifying that and decided to have more groups. The challenge was how to define the partitioning? The security setting and enabling face unlock is at the level of user profile and not at the each account in different hotels, so we cannot partition it by properties or chains. In the end, we went ahead with having one group per one person. This is sufficient, since we always know whom we are verifying.

To give a brief insight into how simple the code can look using our FuncSharp library, here are snippets from unlock operation:

private bool VerifyUnlock(UserUnlockVerification verification, AccessToken token) {
    var user = token.User;
    return verification.Match(
        password => Users.VerifyPassword(user, password.Value), 
        pin => Users.VerifyUnlockPin(user, pin), 
        face  => Users.VerifyFaceImage(user, face)
    );
}

public bool VerifyUnlockPin(User user, UserUnlockPin pin) {    
    return user.PinUnlockEnabled.Match(        
        t => VerifyEncrypted(pin.Value, user.UnlockPin),        
        f => false    
    );
}

public bool VerifyFaceImage(User user, FaceImage faceImage){    
    return FaceDetector.Verify(faceImage, user).Match(        
        success => true,        
        error => error.Match(            
            VerifyError.Failed, _ => false,               
            VerifyError.FaceUnlockNotEnabled, _ => throw new MewsException(l => l.RecognitionNotEnabledErrorMessage)        
        )    
    );
}

On the Frontend, we reused the React component that we have built when working on Identity document scanner – another great project, but let’s leave it for a different article.

The side project

Originally, we used to have the well known concept of Friday side projects. It was limited to developers and we encouraged them to build something cool to improve our product. We had few successes with that. But after a while, it phased out. In an attempt to restart this, and this time we went big, we extended this to the whole company. Promoting cross department collaboration on solving challenging ideas either inside our company or within the products we build.

To get the lock screen feature built, we submitted our suggestion to the company-wide side projects poll. Everyone had a chance to vote and choose what project should have been built. After getting most of the votes, we could go off-site for a couple of days to focus on the solution and we could pick one of the Mews properties around the world.

We enjoyed Lisbon. It’s a wonderful city and when we were done with the work for the day, we went out to discover what the city had to offer us. Pasteis de Nata FTW!

We have managed to leave with a working prototype and now it’s already out in production for all our clients to use.

It does not stop there. We have still several ideas that would make the lock screen better for our customer. For example, we can detect whether there is a face in the picture without any requests to backend. We used face-api.js with the Tiny Face Detector model that has only about 250kB and works just fine for our use case.

Another side project is already in progress, so stay tuned for what is coming next. 😉