Mark Zuckerberg checks in on his year building A.I. for his home

Challenges and accomplishments for setting up A.I. to answer his door, entertain his daughter

artificial intelligence
geralt (CC0)

Mark Zuckerberg's personal 2016 challenge to build an artificial intelligence system to run his home has been a learning experience for the co-founder and CEO of Facebook.

Some parts were simpler than expected and others were a surprising challenge, Zuckerberg said in a blog post.

"My goal was to learn about the state of artificial intelligence -- where we're further along than people realize and where we're still a long ways off," Zuckerberg wrote. "These challenges always lead me to learn more than I expected, and this one also gave me a better sense of all the internal technology Facebook engineers get to use, as well as a thorough overview of home automation."

Zuckerberg's personal home A.I. challenge is one of a string of New Year's resolutions that he has made for himself.

Over the past few years, he's challenged himself to learn to speak Mandarin, read two books a month, and meet a new person every day.

a.i. facebook zuckerberg Facebook

Mark Zuckerberg's home A.I. system is set up to recognize friends and family at the front door. It will let them in and notify Zuckerberg that someone has arrived.

Inspired by Jarvis, the home computer system in the Iron Man comics and movies, Zuckerberg wrote in a blog post last January that for 2016 he was going to focus on using A.I. to run his home and help him with his work.

At the time, Zeus Kerravala, an analyst with ZK Research, said he thought Zuckerberg's focus on A.I. could spur other researchers to do the same.

Today, though, Kerravala said that doesn't seem to have happened.

"if you were working on A.I., it's unlikely that Zuckerberg doing it got you more interested," he added. "If you weren't, then it's unlikely it caused you to jump in."

However, Kerravala still is happy that A.I. was Zuckerberg's focus for the year.

"Enough small advancements in A.I. will mean a big leap one day," he said. "More leaders and companies should be focused on moon shots. I think he's doing what he should be doing. "

So here's what Zuckerberg said he accomplished this past year.

He reported in his post that he built a simple A.I. system that -- by using natural language processing, speech, facial recognition, and reinforcement learning -- can control his home's lights, temperature, security, music, and appliances.

The system, written in Python, Objective C, and PHP, is able to learn new words and concepts, he added, noting that it's also able to entertain his daughter Max and play Mandarin lessons for her. His system is also named Jarvis.

He used facial and image recognition to enable the system to detect if his daughter is awake and moving around in her crib, if that's the dog in the living room or if it's a rug, and if a friend or relative is at the door or if it's a stranger.

"About one-third of the human brain is dedicated to vision, and there are many important A.I. problems related to understanding what is happening in images and videos," Zuckerberg wrote. "Face recognition is a particularly difficult version of object recognition because most people look relatively similar compared to telling apart two random objects -- for example, a sandwich and a house. But Facebook has gotten very good at face recognition for identifying when your friends are in your photos. That expertise is also useful when your friends are at your door and your A.I. needs to determine whether to let them in."

To figure out who's at his door and possibly let them into his house, Zuckerberg said he installed a few cameras to get images of his visitors from different angles, along with a server to monitor the cameras and run facial recognition and check a list of people allowed entry to his home. The system also tells him when a guest has been let in.

"This type of visual A.I. system is useful for a number of things, including knowing when Max is awake so it can start playing music or a Mandarin lesson, or solving the context problem of knowing which room in the house we're in so the A.I. can correctly respond to context-free requests like 'turn the lights on' without providing a location," Zuckerberg wrote.

"Like most aspects of this A.I., vision is most useful when it informs a broader model of the world, connected with other abilities like knowing who your friends are and how to open the door when they're here. The more context the system has, the smarter is gets overall."

Zuckerberg noted that he was disappointed that some of his appliances aren't smart and connected, and the ones that are use different languages and protocols. This made coding his A.I. system more difficult.

One positive, though, is that he was able to use a Messenger bot to communicate with Jarvis.

"I programmed Jarvis on my computer, but in order to be useful I wanted to be able to communicate with it from anywhere I happened to be," he wrote. "That meant the communication had to happen through my phone, not a device placed in my home."

He used the Messenger bot because it was easier than building a separate app.

"I can text anything to my Jarvis bot, and it will instantly be relayed to my Jarvis server and processed," he added. "I can also send audio clips and the server can translate them into text and then execute those commands. In the middle of the day, if someone arrives at my home, Jarvis can text me an image and tell me who's there, or it can text me when I need to go do something."

Though Zuckerberg's annual year challenge is just about over, he noted in his blog that he'll continue working on Jarvis, including building an Android app, setting up Jarvis voice terminals in more rooms, and connecting more appliances.

"In the longer term, I'd like to explore teaching Jarvis how to learn new skills itself rather than me having to teach it how to perform specific tasks," he said. "If I spent another year on this challenge, I'd focus more on learning how learning works."

Copyright © 2016 IDG Communications, Inc.

Bing’s AI chatbot came to work for me. I had to fire it.
Shop Tech Products at Amazon