Google Assistant now equipped with a new capability. In all probability, in the near future, it would also make phone calls on the users’ behalf. For explaining this, CEO Sundar Pichai played back a phone call recording that he said was placed by the Assistant to a hair salon. So, here the voice sounded incredibly natural and in fact, the person on the other end had no idea that they were talking to a digital AI helper. Google Assistant has even dropped in a super casual “mmhmmm” early in the conversation.

Pichai then confirmed that this was a real call using Assistant and not some staged demo. So, he added that “The amazing thing is that Assistant can actually understand the nuances of conversation.  We’ve been working on this technology for many years. It’s called Google Duplex”.

Duplex really gave the feeling of next-level AI stuff, but Google’s chief executive said it’s still very much under development. Further, Google also plans to conduct early testing of Duplex inside Assistant this summer “to help users make restaurant reservations, schedule hair salon appointments, and get holiday hours over the phone.”

Pichai added that the Assistant can react intelligently even when a conversation “doesn’t go as expected” and veers off course a bit from the given objective. A blog post detailing about the soundbites of Duplex in action, also stated that “We’re still developing this technology, and we want to work hard to get this right,” he said. “We really want it to work in cases, say if you’re a busy parent in the morning and your kid is sick and you want to call for a doctor’s appointment.”

The company here also added that “The technology is directed towards completing specific tasks, such as scheduling certain types of appointments. For such tasks, the system makes the conversational experience as natural as possible, allowing people to speak normally, like they would to another person, without having to adapt to a machine.” Google has, in fact, tried to use this Google Assistant in other cases like having Assistant call businesses and inquire about their hours to help keep Maps listings up to date. The company also added that it wants to be transparent about where and when Duplex is being used, as a voice that sounds this realistic and convincing is certain to raise some questions.

Talking about the current testing, Google notes that Duplex successfully completes most conversations and tasks on its own without any intervention from a person on Google’s end. But there are cases where it gets overwhelmed and hands off to a human operator. This section on the ins and outs of Duplex is very interesting.

hair-cut-google-assistant

The Google Duplex system is capable of carrying out sophisticated conversations and it completes the majority of its tasks fully autonomously, without any human involvement. The system has a self-monitoring capability, which allows it to recognize the tasks it cannot complete autonomously, for instance, scheduling an unusually complex appointment. In these cases, it signals to a human operator, who can complete the task.

So, for training this system in a new domain, real-time supervised training is used. This is comparable to the training practices of many disciplines, where an instructor supervises a student as they are doing their job, providing guidance as needed, and making sure that the task is performed at the instructor’s level of quality. Now, in this Duplex system, experienced operators act as the instructors. By monitoring the system as it makes phone calls in a new domain, they can affect the behavior of the system in real time as needed. This continues until the system performs at the desired quality level, at which point the supervision stops and the system can make calls autonomously.