Categories
Coding

Ok google 101 in c# – part 6

Testing the intents

Note: be sure to enable the HomeGraph API for this project in the GCP:

In the action console there is a Test tab, but for the Smart Home device this page is not usable:

We need an Android phone, logged in with the same account as in the Action console, and the Google Home app.

Click on the upper left +, and select Already configured something. Then search for the new draft Action:

The linked thermostats pops up:

Click on the thermostats to see its state and change temperature:

Deploy the release

Go to the Action console and clic on the deploy tab. You are invited to test the action using the test suite. Open it and fill the fields:

  • Agent User Id: the account you want to test
  • Service Account Key: the private Json key generated in the GCP

The test page will play sounds using the computer loudspeaker. An english voice will speak commands to your Google device, that must be near.

Once the test is passed a test_result.txt is generated. This file will be uploaded in the Deploy section of the Actions console.

The approval process is quite slow, and requires proper privacy policy!

Mind the privacy policy. Must meet Google guidelines and must be in the correct language for the country deployed!

If all is ok the action will be approved and published in the Assistant Directory:

Seitron Smart

Categories
Coding

Ok google 101 in c# – part 5

In part four we’ve created an empty AWS lambda application to answer to the Google Smart Home requests, now it’s time to really implement it.

To fullfill the requests coming from the Google Assistant we need to implement 4 REST API in our AWS fullfillment .net Core Web API c# solution:

  • SYNC
  • QUERY
  • EXECUTE
  • DISCONNECT
  • REPORT STATE

There is an official library for .net (nuget), but since is bad documented we’ll implement a minimal set of the Google Smart Home SDK, just for the thermostats.

SYNC

The first call from Google is sync, which asks for the list of thermostats. The token for the thermostat backend is in the Authorization header, and the body contains a JSON with the SmartHomeRequest object:

public class SmartHomeRequest
    {
        public string requestId { get; set; }
        public List<RequestInput> inputs { get; set; }

        public SmartHomeRequest()
        {
            inputs = new List<RequestInput>();
        }
    }
public class RequestInput
    {
        public string intent { get; set; }
    }

With the authorization bearer token in the header (coming from the account linking phase) we can retrieve the list of thermostat, and pack it a SmartHomeResponse object:

With this c# models we can elaborate the answer needed by Google in the POST fullfillment call:

We have given Google the list of thermostats. If we test now in the smartphone Home app, when we do the account pairing with [test] Smart Fullfill we get the list of thermostats:

QUERY

Ask the fullfillment server the current real-time status of one or more devices.

NOTE: there are two ways to know the status of device:

  • QUERY call to fullfillment server
  • Report State: the thermostats are constantly connected to the Google Home Graph sending statuses

When the Google Assistant is asked for the status of a thermostat, it issues the fullfillment server a QUERY POST call, asking the current real-time status. We just need to put together a JSON like this with data coming from the IoT backend:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "devices": {
      "123": {
        "online": true,
        "thermostatMode": "cool",
        "thermostatTemperatureSetpoint": 23,
        "thermostatTemperatureAmbient": 25.1,
        "thermostatHumidityAmbient": 45.3
      }
    }
  }
}

EXECUTE

Send one or more commands to the fullfillment server for one or more thermostats.

DISCONNECT

Stop reporting to this user.

Report state

The Home Graph is a database of your house devices, and it will contains our thermostats. To show the temperature of this thermostats, especially in the visual-assistant devices, the database must be kept updated with the latest temperatures coming from the thermostats. Every now and then we should call the report-state-api updating the Home Graph. The best moments to call it are:

  • After the SYNC fullfillment request
  • After each QUERY fullfillment request
  • After every EXECUTE fullfillment request

To call report state API we need a private key. To see the code and how to call the report state API look at the JWT post.

Once the fullfillment API is ready and published on AWS, we insert its public address in the intents tab:

Now it’s time to test.

End of part 5 — Part 6

Categories
Coding

Sign a Google API JWT in .net C#

In a Google Smart Home project I needed my .net core server application to call the Report State API. The call needs to be authorized by an OAUTH 2.0 access token.

There are many steps to perform on the Google Cloud Platform console (GCP) to obtain this token:

  1. Access to the Smart Home account in the GCP console
  2. Add a Service account of type service account token creator
  3. Download the private key of the service account (a JSON file)
  4. Enable the API you want to call

Then in the .net application you need to:

  • Craft a JWT for the request
  • Sign the JWT with the service account JSON private key
  • Call the Google OAUTH 2 service to get the access token
  • Use the fresh access token to call the API

Let’s begin!


1- Create/select the GCP account (the right one!)

WARNING: this first step can be tricky!

The GCP console contains all of your projects. And it should contains also the project related to the Google Smart Home action. To be sure to select the right GCP project go in the Actions console, select the action you are working on, and click on the upper right menu, then click Manage user access on Google Cloud Platform.

You will be redirected to the IAM page of the relative GCP account, with all the resource listed. Click on Service Account:

2- Add a service account

The service account is needed in the machine-to-machine flow, exactly when our .net server application call the Report State API. We need to create a service account and give it the Service Account Token Creator role:

3-The JSON private key

Click on ‘Create Key‘ to download the JSON private key file:

Select the JSON file type:

Keep safe the JSON file.

4-Enable the Report State API for this GCP account

To call the API it must be granted. In the GCP dashboard click the API arrow:

Add the HomeGraph API:

.net application

The official Google guide is here. This tutorial is for a .net project that needs to call the Report State API: https://www.googleapis.com/auth/homegraph

There is a little-to-none documented .net library for the HomeGraph API available via nuget: Google.Apis.HomeGraphService.v1.

With this library is possible to create the JWT, sign it with the secret key directly from its JSON file, ask for access token and call the ReportState API in just one call!

Categories
Coding

Ok google 101 in c# – part 4

Fullfillment server

Google will send all its requests to the fullfillment URI.

We will serve this URI with a ASP.NET Core Web API, hosted on an AWS Lambda.

To create a Visual Studio solution that deploys in an AWS Lambda follow my previous post. In this post we will focus on the ASP.NET project. The only difference here, is that we use the ASP.NET Core Web API template:

Delete from the templated project the two sample APIs S3ProxyController and ValuesControlles. Right click on Controllers folder and select Add new: Controller (of type API):

The controller will have only one POST endpoint:

 [Route("api/v1/[controller]")]
    public class GoogleController : Controller
    {
// POST: api/v1/Google
        [HttpPost]
        public IActionResult Post([FromBody]object intent)
        {
            try
            {
                Console.WriteLine("Google.POST. Intent=" + intent);
            }
            catch (Exception ex)
            {
                Console.Error.WriteLine("Google.POST Generic: " + ex);
            }
        }

For now we just log what we receive to be sure we are receiving something.

Let’s publish the Web API on AWS. Once published we will use the URL to configure the Action:

We can start testing.

Test the draft version

The action is ready (in DRAFT mode), we can test it. Go to Console▶Develop▶Test:

Make sure that Testing on device is enabled:

Once the test is enabled we can go to our Android phone, that must be logged with same account, and open Google Home app:

Click + on the upper left ▶ Setup device ▶ Have something already set up?

Search for your action name:

Click on it to test the Account Linking.

Warning: this test will result in an error, but we should receive a REST API call to our fullfillment REST API (check on AWS CloudWatch):

[Information] Microsoft.AspNetCore.Hosting.Internal.WebHost: Request starting  POST https://XXXXX.execute-api.XXXXXX.amazonaws.com/Prod/api/google application/json;charset=UTF-8
Google.POST. Intent={
"inputs": [
{
"intent": "action.devices.SYNC"
}
],
"requestId": "10224884186323447126"
}

This means that the Account linking is working! ???

The next step is to implement the real fullfillment.

End of part 4Part 5

Categories
Coding

Ok google 101 in c# – part 3

What is a Smart Home Intent

It’s a single command of our voice system. In case of thermostats could be:

  • Rise temperature
  • What’s the temperature?
  • Turn off heating

The Google Assistant will listen to the voice command, translate it into an Intent, and send the intent in JSON format to our fullfillment server (ASP.NET Web API).

Google Smart Home has four type of intents:

  • SYNC: require the list of devices. It’s the first request done by Google to our fullfillment server. Right after the OAuth exchange. In our case will return the list of thermostats.
  • QUERY: ask for the current state of the system. It’s a read-only request to know thermostat’s state: temperature, setpoints, power.
  • EXECUTE: require a command to be executed. For example: rise temperature by 2 degrees.
  • DISCONNECT: Issued by Google when the user unlink it’s thermostat account from Google Assistant account.

Thermostats

The Google Smart Home ecosystem features a lot of Device Type:

Every device has its capabilities(traits in the Google jargon) The recommended trait for thermostats is:

There is no limit on the traits a device can support. For example our thermostat could support also Brightness trait.

TemperatureSetting

This trait defines the most used features of a typical thermostat: temperatures and modes. It’s composed by attibutes, states and commands. These are the ones supported by our thermostat:

Attributes:

  • availableThermostatModes:
    • off: turn off thermostat itself, disabling heating/cooling.
    • heat: set thermostat in heating mode (Winter mode)
    • cool: set thermostat in cooling mode (Summer mode)
    • on: turn on thermostat. Restore its last functioning mode.
    • auto: set thermostat in scheduled mode

States:

  • thermostatMode: current functioning mode
  • thermostatTemperatureSetpoint: single target temperature in °C
  • thermostatTemperatureAmbient: observed temperature in °C

Commands:

  • ThermostatTemperatureSetpoint
    • thermostatTemperatureSetpoint: single target temperature to set [float type]
  • ThermostatSetMode
    • thermostatMode: set a functioning mode (heat/cool/auto/on/off)
  • TemperatureRelative
    • thermostatTemperatureRelativeDegree: rise or lower temperature b a number of degrees (Turn down 5 degrees)

Fullfillment server

Now that we have clear which intents to support, we can start developing our fullfillment server. It basically is:

  • ASP.NET Core Web API server
  • A single REST API that accepts JSON input
  • AWS Lambda to host the Web API server
  • AWS API Gateway to expose the lambda on internet

The fullfillment server is where the biggest development effort is required, and it’s described in part 4.

End of part 3Part 4

Categories
Coding

Ok google 101 in c# – part 2

In the previous post we saw what a Google Smart Home Action is, and how to perform Account Linking.

Create a Google Smart Home Action

All Actions development happens in the Action Console. Just login with your Google Account and click New project:

Click on Smart Home

Define a Display name:

Then set the Account Linking URLs:

Since my OAuth server is following the Authorization Code flow, I will set up this Action accordingly:

Then fill the client ID, secret and URLs as defined in your OAuth server:

The OAuth server needs also to know the PROJECT_ID, that can be found in the Action Settings page:

Fullfillment and authentication

What is a Google Smart Home Action for a c# developer? Back to the bone is someone that pronounce some voice commands, and the corrisponding c# methods will be executed.

In the Google jargon would be: every Smart Home Intent will be fullfilled by a webhook. In other words the Google Assistant listen to the user’s words, and send the request to the fullfillment server that elaborates a response.

So we need a fullfillment server. It’s a web server that expose APIs that responds to user voice commands and do something on the IoT thermostat.

We will use a c# ASP.NET Web API to implement the fullfillment server. This server will be published in an AWS Lambda, but for now let’s focus on the Smart Home Intents.

Go to part #3.

Categories
Coding

Ok google 101 in c# – part 1

I’m working on an IoT thermostat project that needs integration with Alexa and Google Assistant.

The IoT architecture is quite standard:

IoT architecture

The IoT backend server offers a number of APIs to manage thermostats. It requires user authentication; once logged in a long-lived access token is issued and used in all subsequent API calls. It is a commercial server provided by a third-party.

Google Assistant / Smart Home

The technology behind “Ok google” is Google Assistant, a voice driven AI that interact with smart home devices. The same devices control can be achieved in the Google Home app.

The first step to connect the thermostat to Google Assistant is to create a smart home Action. [glossary]

A tipical Google voice Action requires these steps:

  • Define the grammars and keywords the users can pronounce, in every language you want to support
  • Handle all different paths of voice conversations
  • Trigger the right intent
  • Fullfill the intent (API calls, etc…)

Being in the smart home framework give us a big advantage: the conversation is already managed. You don’t need to define grammars, keywords, languages etc… The smart home framework will call directly our intents.

Account linking OAuth

When you say “Ok Google” you’re always logged in a Google account under the hood. It can be Google mini, our smartphone, or an Android TV; whatever device you’re using, it is always connected with some Google account. You can even setup Voice Match to automatically map your voice with your very personal Google account.

To use the thermostat you also need to be logged in the IoT backend server. So we need to link our personal Google accounts to our IoT account.

There is a (heavy) prerequisite: OAuth 2.

Minimal OAuth tutorial

We must provide Google an OAuth service. The quickest way is to use the so called implicit code flow, that is basically a single HTTPS endpoint:

GET https://myservice.example.com/auth

This endpoint accepts the following parameters:

  • client_id
  • redirect_uri
  • state
  • response_type=token

Then display a login/password GUI to let user authenticate in the IoT backend. If the user successfully login, the long-lived access token generated by the IoT backend is passed back to Google, redirecting the user’s browser to the redirect_uri.

OAuth authorization flow

To provide more security, and to be also Alexa-compatible, you can choose the authorization code flow.

This flow uses two endpoints:

  • Login page endpoint
  • Token exchange endpoint

If your IoT backend authentication system is already OAuth-compliant you can continue and configure the Google Action with your OAuth URLs.

If your authentication system is not OAuth-compliant you can develop your own OAuth server. Look at my other post about creating an OAuth server in Amazon AWS.

End of part 1Part 2

Categories
Coding

Creating an OAuth 2 server in AWS Lambda with C#

Google Smart Home and Alexa skills need to impersonate the user when calling private APIs, so an Account Linking is required.

Like the majority of the API economy, Google and Amazon support the OAuth 2 standard.

I will implement a OAuth server that supports the authorization code flow. This OAuth service will provide 2 endpoints:

  • Web login page
  • Token endpoint

It consists of a web login page that authenticates the user in my private API authentication system, and the provide back to Google/Amazon a grant code. With this grant code the Google/Amazon app can obtain from the token endpoint an access token and a refresh token. I will develop it in ASP.NET Core using a serverless AWS Lambda. For small number of calls (under 1 milion/month) it’s almost free. Most AWS services used in this tutorial are paid services, you have been warned!

This is the list of AWS services needed:

  • IAM: authentication service
  • CloudFormation
  • S3 bucket for CloudFormation files
  • APIGateway to get the Lambda public URL

Prerequisites

We need an AWS account and Visual Studio 2019.

Visual Studio ▶Extensions▶Manage extensions: install AWS Toolkit for Visual Studio 2019 (version now is 1.15.2.1).

The VS solution

Create a new AWS Serverless Application (.NET Core C#):

In the next dialog select the ASP.NET Core Web App blueprint:

We will find a full ASP.NET core project, based on .NET Core 2.1 and a few Razor pages.

The OAuth authorization code flow will redirect the user browser to our login page using such an URL: https://myservice.example.com/auth along with 4 parameters:

  • client_id
  • redirect_uri
  • state
  • response_type

So let’s add an auth login page that can receive these four parameters.

The login page

The login page is a razor page with a login/password form:

public async Task<IActionResult> OnPostAsync()
        {
                //TODO: check url params
                var login = await _ApiService.LoginAsync(Username, Password);
                if (login) {
                    //TODO: generate a code
                    string uri = RedirectUri + "?code=XXXXX&state=" + State;
                return Redirect(rup);
            }

The base html will be:

<form method="post">
    <input asp-for="Username" class="form-control" />
    <input asp-for="Password" class="form-control" type="password" />
    <input type="submit" value="Login" class="btn btn-primary" />
</form>

Summary: the login page is just a username/password form that authenticate against my private API backend, create an auth code, and redirect back to the caller, passing the auth code.

Token endpoint

The token endpoint is a Web Api controller:

  • Create an Api subfolder
  • Right click and select Add Items▶Api Controller Class
  • Call it ‘tokenController’

The token endpoint is responsible for two post action:

  • receive a grant auth code and return refresh+access tokens
  • receive a refresh token and return an access token
 [Route("api/v1/token")]
    [ApiController]
    public class tokenController : Controller
    {
        // POST api/v1/token
        [HttpPost]
        public IActionResult Post(string client_secret, string client_id, string grant_type, string code, string refresh_token) {
    if(authCodeRequest)
    {
        //generate refresh + access tokens
        return Content("{\"token_type\": \"Bearer\",\"access_token\": \"" + accessToken + "\",\"refresh_token\": \"" + refreshToken + "\",\"expires_in\": " + 3600 + "}");
    }
    else if (refreshTokenRequest)
    {
        return Content("{\"token_type\": \"Bearer\",\"access_token\": \"" + accessToken + "\",\"expires_in\": " + 3600 + "}");
    }

Publishing to AWS

Now that we have a working ASP.NET razor login page, and the token POST endpoint, we need to publish it in an AWS lambda, and it’s where the fun starts!

Visual Studio, with the ASW toolkit, provide a Publish to AWS Lambda… shortcut:

Publish to AWS Lambda…

Before publishing we need to understand what Publish to AWS Lambda means. In particular we need to know which AWS services will be created (also to understand what we will pay!).

AWS CloudFormation: think of it as a script that configure and starts all AWS services needed. It’s free.

Lambda function: the serverless function that will run the ASP.NET solution. Free for the first few millions calls.

S3 Bucket: the storage where the cloudformation script and the asp.net zipped solution will be stored. You pay for the disk usage.

API Gateway: the public endpoint that expose the ASP.NET site. Free for the first million calls monthly.

The Visual Studio project is already provided with a CloudFormation serverless.template configuration file that will create automatically all the AWS services required. This collection of services is called a stack.

Prerequisite: Install the AWS CLI to do the advanced administative work. You can test it in PowerShell, enter aws help:

IAM: create the administrative account in AWS to upload and run the lambda

To run the publish wizard Visual Studio needs an AWS user with the proper permissions. Go to AWS console, select IAM service and go to Users▶Add.

Visual Studio will use AWS SDK, so a programmatic access it’s enough:

Step 2: Set Permission. Select the Attach existing policies directly and select

  • AWSLambdaBasicExecutionRole
  • AmazonS3FullAccess
  • AWSCloudFormationFullAccess

Step 3: Tags. Skip this step safely.

Step 4: Review. The review page should look like this:

Click Create user. And be sure to download the CSV file!

This is your only chance to download the CSV file, make sure to backup it safely!

WARNING: The permissions set are not enough. The quickest way to configure the IAM user would be to give all permissions, but this could be a security issue. So we will select only the permissions we really need to create a ASP.NET stack.

Enter the IAM user and add a custom policy:

Add these permissions to the policy:

  • API Gateway
    • GET
    • DELETE
    • PATCH
    • POST
  • Lambda
    • GetFunction
    • GetFunctionConfiguration
    • CreateFunction
    • DeleteFunction
    • UpdateFunctionConfiguration
    • UpdateFunctionCode
    • AddPermission
    • RemovePermission
    • ListTags
    • TagResources
    • UntagResources
  • IAM
    • ListRolePolicies
    • ListRoles
    • ListRoleTags
    • ListUserTags
    • GetRole
    • CreateRole
    • DeleteRole
    • PassRole
    • AttachRolePolicy
    • DeleteRolePolicy
    • DetachRolePolicy
    • PutRolePolicy

Save the policy!

Let’s go back to Visual Studio

The CSV file is all we need to make Visual Studio publish the Lambda. Right click on the ASP.NET project and select Publish to AWS Lambda…:

Click on the ‘+’ icon to upload the CSV:

Enter a Profile name and select the CSV file:

Create a bucket, and give a CloudFormation stack name:

This is the final screen of the configured wizard:

When clicking Publish the wizard will start uploading the lambda and then creating the infrastructure with CloudFormation. At the end the status window will show the progresses:

? Congratulations! Your OAuth server is ready on-air!