Homie - A language understanding bot for smart homes

Smart homes, smart vehicles, smart everything - nowadays all devices which we daily use have to be smart otherwise we wouldn’t use them, would we? There are so many different approaches to make a device smart either you can control it remotely, speak to it or it even sets actions autonomously which should ease our lifes here and there. The key to achieve all this is nearly the same everywhere, no matter if you use smart devices and services provided by Google, Amazon or Microsoft. Behind all that is alwaysAI (Artifical Intelligence).

And as I do a lot of research on Microsoft’s AI stack (Cognitive Services, Bot Framework, Machine Learning, …) within Azure, I wanted to develop something intelligent, which could make my everyday life a bit easier. So I thought of a service, which could take actions whenever I tell it to do so. And this was the birth of “Homie”, who is my personal assistant for doing smart stuff. But now you most probably want to know who or what Homie is, right? Well, I’ll try to explain you what and who he is within the next few paragraphs.

The Solution Design

Let’s start. Imagine you are sitting in your office at home or at work and are really deepened in your work and you have to be concentrated in order to get your work done. And as time flies by, it gets dark outside. In the beginning you won’t notice that it gets dark as you only star into your monitors and do your work. But after a few minutes, when it’s really dark, you are sitting in a dark room and we all know, this doesn’t work out very well for a longer time. But as you are a lazy person, like me, you don’t want to stand up and go over to the light switch to turn the light on. But what if I told you that Homie could turn the light on for you and you can remain seated? All you need to do is to follow the following steps and get all services connected to each other like in the picture below and you can tell Homie what to do for you ;)

The design of the given solution as well as the flow of communication is shown above

The Bot and Cognitive Services Stuff

First of all we need to create an Azure Bot service, which is responsible for communicating with the user. So head over to the Azure portal and create a new bot. In my case I created a NodeJS LUIS bot, as there is already some LUIS related stuff preconfigured. Now when the bot is ready you need to add some intents to your LUIS app in order for the bot to actually be smart. So go to luis.ai and login (or register if you did not already) and add the following pre-defined intents:

These are the pre-defined LUIS intents used in this Cognitive Services app

Now test and train your LUIS app and go back to your bot in the Azure portal, as we now will insert some magic. Now the following piece of code, let’s your bot become smart as this code is responsible for doing actions based on the detected intent of a received messages.

intents.matches('HomeAutomation.TurnOn',function(session, args){ 
    session.sendTyping();
    if(args.entities[0]) {
        var deviceEntity = builder.EntityRecognizer.findEntity(args.entities, 'HomeAutomation.Device');
        var roomEntity = builder.EntityRecognizer.findEntity(args.entities, 'HomeAutomation.Room');
        if (roomEntity){
            session.send("Ok, I will turn the %s on in the %s", deviceEntity['entity'], roomEntity['entity']);
            session.endDialog();
            var commandJSON= {
                "device": deviceEntity['entity'],
                "room": roomEntity['entity'],
                "action": "Turn on"
            };
            request({
                url: "<URL of your Azure Function>",
                method: "POST",
                json: true,   // <-- This is very important!
                body: commandJSON
            }, function (error, response, body){
                console.log(response);
            });
        }
    }
});

The IoT Stuff

Now that your bot is ready to receive messages, we need to create an Azure IoT Hub which is used to send and receive messages from and to devices. So create one in your portal and add your devices to the hub according to your needs. In this example I used a device for my office, which I called “office_controller”. Make sure to note the keys of your device, as you will need them later on:

There are only a few clicks needed to create a new IoT device in the IoT Hub blade

The Azure Functions Stuff

Now that we have our bot and the IoT Hub up and running, we need to connect those services with each other. And I thought of doing so by using a JavaScript HTTP trigger function. So create a function in the Functions portal and make sure to install the following dependencies by using the command line within your function’s settings:

npm install azure-iothub --save

Now to reference your created IoT Hub in the function’s code you can use the following piece of code:

var Client = require('azure-iothub').Client;
var Message = require('azure-iot-common').Message;
var connectionString = '<yourIoTHubConnectionString>';
var targetDevice = 'office_controller';
var serviceClient = Client.fromConnectionString(connectionString)

In order to send a message to your IoT Hub, use the following code snippet:

var message = new Message('lightsOnOffice');
message.ack = 'full';
message.messageId = "ID_001";
serviceClient.send(targetDevice, message, printResultFor('send'))

The Device Stuff

Now this is the last piece of work to do in order to actually make your hard work visible. In this case I used a Raspberry Pi 3 in combination of a breadboard and a normal LED light, as I don’t have a smart light bulb at hand, which I could have used instead. But I guess this is ok, as I want to show you that the solution is working rather than the light bulb is IoT enabled.

So now it’s time to assemble your PI in order to get the LED connected to it. Below you’ll find the actual construction manual by Microsoft, which I followed to connect the LED on the breadboard with my Pi via GPIO.

This is the pinout manual on the Raspberry PI’s breadboard for connecting a LED light taken from GitHub

When I finished that, I installed NPM and Node JS on my PI in order to run a little Node JS app, which is actually listening on the IoT hub’s device commands. Whenever there is a new command in the queue for this specific device, a function gets triggered, which does something. In my case I wanted to turn the lights on or off in a specific room. The actual code for lightening up your LED is something like this:

var wpi = require('wiring-pi');
// GPIO pin of the led
var configPin = 7;
wpi.setup('wpi');
wpi.pinMode(configPin, wpi.OUTPUT);
// Turn LED on
var isLedOn = 0;
isLedOn = +!isLedOn;
wpi.digitalWrite(configPin, isLedOn )

And that’s basically it. Now it’s time to tell Homie to turn the light on and off in the office and see if it’s working. Take a look at the video to find out the answer. And yes you’re right, this is only a LED light, which Homie turned on and off, but the process is actually the same and I wanted to proof that the given problem can be solved by a bot. But please feel free to adapt my solution and add real light bulbs or other things to it in order to make it even more realistic…

The Teams Stuff

Now that your bot is fully functional and doing what you want him to do, it’s time to add the desired channels to that bot in order to publish it into your favourite apps. I decided to go for Microsoft Teams, as this is one of my favourite communication tools. But you can choose whatever you like from the following list of channels:

These are the available channels, in which the bot can be added

After you have added the Teams channel you can communicate with the bot via Teams, just as simple as that:

The communication with your bot within Teams is like communicating with your co-workers

The Conclusion

• Azure, Cognitive Services and the Bot Framework fit perfectly together, as the interaction between the services is working out of the box
• Azure Functions are always a good use for hosting logic and code which is consumed by other services
• Language Understanding Bots will likely to be introduced more and more in consumer as well as business solutions as they make life easier a lot
• With less effort,Cognitive Services can be used to make a service more human and lets therefore interact people with a service like a bot in a more natural way, which is quite promising

Now that this solution is working, I’ll update you with more features concerning this scenario as well as publishing the code and other stuff which was used in this solution as an open-source solution so everyone of you can set up a personal Homie as well. If you have any questions, notes or any other use cases for this bot (or other use cases), which may be interesting, please let me know down below or via any social media channel as I would like to hear your feedback on this one…

By the way, this blog post was also released on the official MSDN Team Blog Austria in German if you are interested take a look at MSDN Team Blog Austria (Codefest)