In part four we’ve created an empty AWS lambda application to answer to the Google Smart Home requests, now it’s time to really implement it.
To fullfill the requests coming from the Google Assistant we need to implement 4 REST API in our AWS fullfillment .net Core Web API c# solution:
SYNC
QUERY
EXECUTE
DISCONNECT
REPORT STATE
There is an official library for .net (nuget), but since is bad documented we’ll implement a minimal set of the Google Smart Home SDK, just for the thermostats.
SYNC
The first call from Google is sync, which asks for the list of thermostats. The token for the thermostat backend is in the Authorization header, and the body contains a JSON with the SmartHomeRequest object:
public class SmartHomeRequest
{
public string requestId { get; set; }
public List<RequestInput> inputs { get; set; }
public SmartHomeRequest()
{
inputs = new List<RequestInput>();
}
}
public class RequestInput
{
public string intent { get; set; }
}
With the authorization bearer token in the header (coming from the account linking phase) we can retrieve the list of thermostat, and pack it a SmartHomeResponse object:
With this c# models we can elaborate the answer needed by Google in the POST fullfillment call:
We have given Google the list of thermostats. If we test now in the smartphone Home app, when we do the account pairing with [test] Smart Fullfill we get the list of thermostats:
QUERY
Ask the fullfillment server the current real-time status of one or more devices.
NOTE: there are two ways to know the status of device:
QUERY call to fullfillment server
Report State: the thermostats are constantly connected to the Google Home Graph sending statuses
When the Google Assistant is asked for the status of a thermostat, it issues the fullfillment server a QUERY POST call, asking the current real-time status. We just need to put together a JSON like this with data coming from the IoT backend:
Send one or more commands to the fullfillment server for one or more thermostats.
DISCONNECT
Stop reporting to this user.
Report state
The Home Graph is a database of your house devices, and it will contains our thermostats. To show the temperature of this thermostats, especially in the visual-assistant devices, the database must be kept updated with the latest temperatures coming from the thermostats. Every now and then we should call the report-state-api updating the Home Graph. The best moments to call it are:
After the SYNC fullfillment request
After each QUERY fullfillment request
After every EXECUTE fullfillment request
To call report state API we need a private key. To see the code and how to call the report state API look at the JWT post.
Once the fullfillment API is ready and published on AWS, we insert its public address in the intents tab:
In a Google Smart Home project I needed my .net core server application to call the Report State API. The call needs to be authorized by an OAUTH 2.0 access token.
Access to the Smart Home account in the GCP console
Add a Service account of type service account token creator
Download the private key of the service account (a JSON file)
Enable the API you want to call
Then in the .net application you need to:
Craft a JWT for the request
Sign the JWT with the service account JSON private key
Call the Google OAUTH 2 service to get the access token
Use the fresh access token to call the API
Let’s begin!
1- Create/select the GCP account (the right one!)
WARNING: this first step can be tricky!
The GCP console contains all of your projects. And it should contains also the project related to the Google Smart Home action. To be sure to select the right GCP project go in the Actions console, select the action you are working on, and click on the upper right menu, then click Manage user access on Google Cloud Platform.
You will be redirected to the IAM page of the relative GCP account, with all the resource listed. Click on Service Account:
2- Add a service account
The service account is needed in the machine-to-machine flow, exactly when our .net server application call the Report State API. We need to create a service account and give it the Service Account Token Creator role:
3-The JSON private key
Click on ‘Create Key‘ to download the JSON private key file:
Select the JSON file type:
Keep safe the JSON file.
4-Enable the Report State API for this GCP account
To call the API it must be granted. In the GCP dashboard click the API arrow:
Add the HomeGraph API:
.net application
The official Google guide is here. This tutorial is for a .net project that needs to call the Report State API: https://www.googleapis.com/auth/homegraph
There is a little-to-none documented .net library for the HomeGraph API available via nuget: Google.Apis.HomeGraphService.v1.
With this library is possible to create the JWT, sign it with the secret key directly from its JSON file, ask for access token and call the ReportState API in just one call!
Google will send all its requests to the fullfillment URI.
We will serve this URI with a ASP.NET Core Web API, hosted on an AWS Lambda.
To create a Visual Studio solution that deploys in an AWS Lambda follow my previous post. In this post we will focus on the ASP.NET project. The only difference here, is that we use the ASP.NET Core Web API template:
Delete from the templated project the two sample APIs S3ProxyController and ValuesControlles. Right click on Controllers folder and select Add new: Controller (of type API):
It’s a single command of our voice system. In case of thermostats could be:
Rise temperature
What’s the temperature?
Turn off heating
The Google Assistant will listen to the voice command, translate it into an Intent, and send the intent in JSON format to our fullfillment server (ASP.NET Web API).
Google Smart Home has four type of intents:
SYNC: require the list of devices. It’s the first request done by Google to our fullfillment server. Right after the OAuth exchange. In our case will return the list of thermostats.
QUERY: ask for the current state of the system. It’s a read-only request to know thermostat’s state: temperature, setpoints, power.
EXECUTE: require a command to be executed. For example: rise temperature by 2 degrees.
DISCONNECT: Issued by Google when the user unlink it’s thermostat account from Google Assistant account.
Thermostats
The Google Smart Home ecosystem features a lot of Device Type:
There is no limit on the traits a device can support. For example our thermostat could support also Brightness trait.
TemperatureSetting
This trait defines the most used features of a typical thermostat: temperatures and modes. It’s composed by attibutes, states and commands. These are the ones supported by our thermostat:
Attributes:
availableThermostatModes:
off: turn off thermostat itself, disabling heating/cooling.
heat: set thermostat in heating mode (Winter mode)
cool: set thermostat in cooling mode (Summer mode)
on: turn on thermostat. Restore its last functioning mode.
auto: set thermostat in scheduled mode
States:
thermostatMode: current functioning mode
thermostatTemperatureSetpoint: single target temperature in °C
thermostatTemperatureAmbient: observed temperature in °C
Commands:
ThermostatTemperatureSetpoint
thermostatTemperatureSetpoint: single target temperature to set [float type]
ThermostatSetMode
thermostatMode: set a functioning mode (heat/cool/auto/on/off)
TemperatureRelative
thermostatTemperatureRelativeDegree: rise or lower temperature b a number of degrees (Turn down 5 degrees)
Fullfillment server
Now that we have clear which intents to support, we can start developing our fullfillment server. It basically is:
ASP.NET Core Web API server
A single REST API that accepts JSON input
AWS Lambda to host the Web API server
AWS API Gateway to expose the lambda on internet
The fullfillment server is where the biggest development effort is required, and it’s described in part 4.
In the previous post we saw what a Google Smart Home Action is, and how to perform Account Linking.
Create a Google Smart Home Action
All Actions development happens in the Action Console. Just login with your Google Account and click New project:
Define a Display name:
Then set the Account Linking URLs:
Since my OAuth server is following the Authorization Code flow, I will set up this Action accordingly:
Then fill the client ID, secret and URLs as defined in your OAuth server:
The OAuth server needs also to know the PROJECT_ID, that can be found in the Action Settings page:
Fullfillment and authentication
What is a Google Smart Home Action for a c# developer? Back to the bone is someone that pronounce some voice commands, and the corrisponding c# methods will be executed.
In the Google jargon would be: every Smart Home Intent will be fullfilled by a webhook. In other words the Google Assistant listen to the user’s words, and send the request to the fullfillment server that elaborates a response.
So we need a fullfillment server. It’s a web server that expose APIs that responds to user voice commands and do something on the IoT thermostat.
We will use a c# ASP.NET Web API to implement the fullfillment server. This server will be published in an AWS Lambda, but for now let’s focus on the Smart Home Intents.
I’m working on an IoT thermostat project that needs integration with Alexa and Google Assistant.
The IoT architecture is quite standard:
The IoT backend server offers a number of APIs to manage thermostats. It requires user authentication; once logged in a long-lived access token is issued and used in all subsequent API calls. It is a commercial server provided by a third-party.
Google Assistant / Smart Home
The technology behind “Ok google” is Google Assistant, a voice driven AI that interact with smart home devices. The same devices control can be achieved in the Google Home app.
The first step to connect the thermostat to Google Assistant is to create a smart home Action. [glossary]
A tipical Google voice Action requires these steps:
Define the grammars and keywords the users can pronounce, in every language you want to support
Handle all different paths of voice conversations
Trigger the right intent
Fullfill the intent (API calls, etc…)
Being in the smart home framework give us a big advantage: the conversation is already managed. You don’t need to define grammars, keywords, languages etc… The smart home framework will call directly our intents.
Account linking OAuth
When you say “Ok Google” you’re always logged in a Google account under the hood. It can be Google mini, our smartphone, or an Android TV; whatever device you’re using, it is always connected with some Google account. You can even setup Voice Match to automatically map your voice with your very personal Google account.
To use the thermostat you also need to be logged in the IoT backend server. So we need to link our personal Google accounts to our IoT account.
There is a (heavy) prerequisite: OAuth 2.
Minimal OAuth tutorial
We must provide Google an OAuth service. The quickest way is to use the so called implicit code flow, that is basically a single HTTPS endpoint:
GET https://myservice.example.com/auth
This endpoint accepts the following parameters:
client_id
redirect_uri
state
response_type=token
Then display a login/password GUI to let user authenticate in the IoT backend. If the user successfully login, the long-lived access token generated by the IoT backend is passed back to Google, redirecting the user’s browser to the redirect_uri.
OAuth authorization flow
To provide more security, and to be also Alexa-compatible, you can choose the authorization code flow.
This flow uses two endpoints:
Login page endpoint
Token exchange endpoint
If your IoT backend authentication system is already OAuth-compliant you can continue and configure the Google Action with your OAuth URLs.
If your authentication system is not OAuth-compliant you can develop your own OAuth server. Look at my other post about creating an OAuth server in Amazon AWS.