He used facial and image recognition to enable the system to detect if his daughter is awake and moving around in her crib, if that's the dog in the living room or if it's a rug, and if a friend or relative is at the door or if it's a stranger.
"About one-third of the human brain is dedicated to vision, and there are many important A.I. problems related to understanding what is happening in images and videos," Zuckerberg wrote. "Face recognition is a particularly difficult version of object recognition because most people look relatively similar compared to telling apart two random objects -- for example, a sandwich and a house. But Facebook has gotten very good at face recognition for identifying when your friends are in your photos. That expertise is also useful when your friends are at your door and your A.I. needs to determine whether to let them in."
To figure out who's at his door and possibly let them into his house, Zuckerberg said he installed a few cameras to get images of his visitors from different angles, along with a server to monitor the cameras and run facial recognition and check a list of people allowed entry to his home. The system also tells him when a guest has been let in.
"This type of visual A.I. system is useful for a number of things, including knowing when Max is awake so it can start playing music or a Mandarin lesson, or solving the context problem of knowing which room in the house we're in so the A.I. can correctly respond to context-free requests like 'turn the lights on' without providing a location," Zuckerberg wrote.
"Like most aspects of this A.I., vision is most useful when it informs a broader model of the world, connected with other abilities like knowing who your friends are and how to open the door when they're here. The more context the system has, the smarter is gets overall."
Zuckerberg noted that he was disappointed that some of his appliances aren't smart and connected, and the ones that are use different languages and protocols. This made coding his A.I. system more difficult.
One positive, though, is that he was able to use a Messenger bot to communicate with Jarvis.
"I programmed Jarvis on my computer, but in order to be useful I wanted to be able to communicate with it from anywhere I happened to be," he wrote. "That meant the communication had to happen through my phone, not a device placed in my home."
He used the Messenger bot because it was easier than building a separate app.
"I can text anything to my Jarvis bot, and it will instantly be relayed to my Jarvis server and processed," he added. "I can also send audio clips and the server can translate them into text and then execute those commands. In the middle of the day, if someone arrives at my home, Jarvis can text me an image and tell me who's there, or it can text me when I need to go do something."
Sign up for Computerworld eNewsletters.