There are many instances when the good photos are ruined by red-eye, lens flare or closed eyes. There are many applications or settings where the red eye and lens flare can be corrected. But, there is no solution for the blinked eyes, yet. But, Facebook might have changed it with its new AI research tool. The new research from Facebook helps to ‘open’ the closed eyes in any photos.
The technique is called ‘in-painting’ where the program fills in the space with what it thinks belongs there. Previously, Adobe has used the feature in its ‘context-aware fill’ which allows the users to replace the undesired things in the image. Suppose, if there is a protruding branch, the feature can identify what would be there if it were not. The tool is great however it did not have the capacity to replace closed eyes. The eyes are very detailed and variable so the tool was not able to create it realistically.
Facebook’s Take On Blinked Eyes
Facebook has taken steps to change the existing tool. It can be of great use because probably it has the highest repository of blinking eyes photos. The new tool works on a Generative Adversarial Network, which is a Machine Learning tool. The GAN has two parts – one recognizes the face and other tries to create an image based on that. It uses the feedback of the recognition part to make the new image seem realistic. It fools the system to believe that its creation is real.
The GAN is trained to work with both recognizing and replicating open eyes images. This was possible before also with some of the tools available but it was not as accurate as the new Facebook tool. The previous tools used to paste the eyes directly without considering much about the matching with the rest of the image. The machines do not understand much regarding the eyes. The machines do not know that opening one’s eye does not change the color of the skin near it.
How The Algorithm Works
Facebook researchers found another way around the problem. They included the need for exemplar data of the subject. It asked the user to feed in the target image of the person with their eyes open. From the image, GAN identified the details of eyes such as what shape it is, how it looks, what color it is and so on. From these data, the tool was able to rectify the blinking eyes image with open eyes.
Surprisingly, the results were too good. There was no mismatch in colors and it understands how a person should look with their eyes open. It avoided the stitching which was done by other existing tools.
When the tool was tested, the results were good enough. Most of the people confused the created image with real images. Many times the users were not able to differentiate between the two images. At the first sight, no one will know that the photo was changed and they would scroll past the photo after a look. However, the sample photo of Mahatma Gandhi did seem a little weird.
Moreover, Facebook can use the tool to automatically apply on its website. It can improve the group photo by opening the eyes of people with reference to their other photos. It can be a little weird but too can save many good pictures.