Misty II Overview

Your Misty II robot has been designed and built by the Misty Robotics team. This document provides some quick, out-of-box setup steps and helpful details about using Misty II.

To avoid the risk of driving Misty off a high surface, we recommend either working with your robot on the floor, or temporarily elevating the treads so the robot cannot drive. To do this, you can place Misty on the the foam block stand that arrived in the box. While Misty is designed to automatically detect obstacles and ledges, placing her on this foam block will ensure that she doesn't accidentally roll off the table while you're coding.

Unpacking the Misty II

Misty II arrives packed safely inside her carrying case. The easiest and safest way to remove Misty from her case is to open the latches and position the case vertically, so the robot is fully upright. With the case open, grip the Misty II by its torso (never by the arms) and gently slide it out of or into the case.

Misty II arrives with her magnetic headpiece and backpack attachments connected. In addition to the robot, the carrying case includes the following items:

• charging station - Misty II Standard and Enhanced Editions only
• charging station AC power supply (thin barrel jack) - Misty II Standard and Enhanced Editions only
• Misty II AC power supply (thick barrel jack) - Misty II Basic only
• foam block stand for Misty II
• legal guide
• printed User Guide

Powering Up & Powering Down

We recommend powering up Misty on the floor, if possible, to avoid the risk of driving your robot off the edge of a table or other high surface. If you are experimenting with Misty on a table top, place her on the foam block stand that arrived in her carrying case to elevate her treads above the surface of the table.

Turning Misty On

1. Toggle the power switch on Misty's base (located between the treads beneath the rear time-of-flight sensor).
2. The Misty Robotics logo first appears on the screen. Then Misty's eyes appear, beginning in a closed state. The eyes gradually open as Misty boots up.
3. When the eyes appear fully open, Misty is done booting up. This should take a little more than a minute.

Restarting Misty

If you need to restart Misty:

1. Turn off the power switch on Misty's base.
2. Wait 10 seconds.
3. Turn on the power switch on Misty's base again. Misty has rebooted when her eyes are fully open.

Turning Misty Off

To turn off your robot, turn off the power switch on Misty's base.

Charging Misty II

You can charge Misty with her wireless charging station, or you can plug Misty's AC power supply (optional accessory) directly into the port on her base. When charging while powered on, Misty's chest LED pulses orange.

Wireless Charging

To use Misty's wireless charging station, follow these steps:

1. Connect the power supply for the wireless charging station to power. To prevent tripping hazards, wrap any extra length of the power cable around the nodes inside the rear compartment of the charging station.
2. Place the charging station on the floor.
3. Place Misty on the center of the charging station. Position her to be facing out, with her back against the rear compartment of the charging station, and make sure the arrows on Misty's base are lined up with the arrows on the charging station.

Wired Charging

Follow these steps to charge Misty via a wired connection:

1. Remove the rubber flap covering the power port underneath Misty's base.
2. Connect Misty's AC power supply (not the power supply for the wireless charging station) to her power port.
3. Connect the power supply to an outlet.

Connecting to Wi-Fi

• Misty offers several ways to get connected to your network. The most common method is to use the Misty App. We've provided and extensive guide on how to use get the app, and how to use it. After connecting Misty to your network, you'll find that you often use the Misty App to get your robot's IP address.

• In cases where using a mobile application won't work, or if you're having issues with Bluetooth, you can also use a USB drive to connect to the wifi network. To do so, follow these steps:

  1. Format the USB drive in FAT format.
  2. In the root of that file system, create a directory named 'misty' (without the quotes).
  3. In that directory, create a text file named wifi.txt.
  4. Edit that text file. On the first line, add the name of the SSID. On the second line, add the network key. The resulting file should look something like this:

     MyNetworkName
     MyNetworkKey
     

  5. Save the file and eject the USB drive. Insert the drive into Misty's USB port and give her 60 seconds to connect to the network.
  6. Remove the USB drive from Misty. Within the misty directory should be a file called 'onboard.txt'. This file will contain the robot's IP address.

• If you're connecting to an enterprise wifi network, you'll use a similar method to the USB drive method above:

  1. Format the USB drive in FAT format.
  2. In the root of that file system, create a directory named 'misty' (without the quotes).
  3. Into that directory, you add a JSON formatted text file named enterprisewifi.txt with the format:

     {
      "SSID": "THE-SSID",
      "Identity": "THE-USERNAME",
      "Password": "THE-PASSWORD",
      "EapMethod": "{NONE},{PEAP},{TLS},{TTLS},{PWD},{SIM},{AKA},{AKA_PRIME},{UNAUTH_TLS}",
      "Phase2Method": "{NONE},{PAP},{MSCHAP},{MSCHAPV2},{GTC},{SIM},{AKA},{AKA_PRIME}",
      "AltSubjectMatch": "THE-ALT-SUBJECT-MATCH",
      "AnonymousIdentity": "THE-ANONYMOUS-IDENTITY",
      "DomainSuffixMatch": "THE-DOMAIN-SUFFIX",
      "Realm": "THE-REALM",
      "KeyManagementMethods": [
          "{None},{WPA_PSK},{WPA_EAP},{IEEE8021X},{WPA2_PSK},{OSEN},{FT_PSK},{SAE},{OWE},{SUITE_B_192},{WPA_PSK_SHA256},{WPA_EAP_SHA256}",
          "{None},{WPA_PSK},{WPA_EAP},{IEEE8021X},{WPA2_PSK},{OSEN},{FT_PSK},{SAE},{OWE},{SUITE_B_192},{WPA_PSK_SHA256},{WPA_EAP_SHA256}",
          ...
          "{None},{WPA_PSK},{WPA_EAP},{IEEE8021X},{WPA2_PSK},{OSEN},{FT_PSK},{SAE},{OWE},{SUITE_B_192},{WPA_PSK_SHA256},{WPA_EAP_SHA256}"
      ]
     }
     

  4. The wifi file should be configured to match the requirements of the network. You'll likely need to work with your network administrator to determine the right options to use. The underlying system is Android, which supports these configuration options. Also, note that alues provided for EapMethod, Phase2Method, and KeyManagementMethods represent the valid options, and that KeyManagementMethods is an array.
  5. Save the file and eject the USB drive. Insert the drive into Misty's USB port and give her 60 seconds to connect to the network.
  6. Remove the USB drive from Misty. Within the misty directory should be a file called 'onboard.txt'. This file will contain the robot's IP address.

• Lastly, there are cases in which you may want to connect Misty to a wired network. Misty doesn't have an RJ45 ethernet port, so you'd need a USB to ethernet adapter (not supplied) to make the connection.

Coordinate System & Movement Ranges

As you develop skills and robot applications, it can be helpful to understand a few key details about Misty's coordinate system and the range of movement allowed for her head and arm motors.

• Misty uses a right-handed coordinate frame when orienting her body, head, and arms relative to her surroundings, when interpreting movement and locomotion commands, and when publishing sensor data.
• Most of Misty's commands and event types ingest and publish angular measurement data using degrees (instead of radians) as the default unit of measurement. As such, we recommend using degrees to calculate position and movement whenever you can.
• Misty's inertial measurement unit (IMU) orients its heading to 0/360 degrees each time Misty boots up. For Misty, an IMU yaw value of 0/360 degrees does not represent true north unless Misty is facing true north when the IMU orients its heading.
• Because the IMU is located in Misty's torso, readings from the IMU only change when Misty's body moves. They do not change relative to the position of Misty's head.
• A positional value of 0 degrees orients Misty's arms to point horizontally forward, directly in front of the robot. To prevent interference between the arms and the plastic lens protecting Misty's display, her arms cannot move higher than -29 degrees. Additionally, they cannot move lower than 90 degrees (pointing straight down).
• A positional value of 0 degrees for head movement in the roll, pitch, and yaw directions orients Misty's head to face straight forward, directly in front of the robot.
• The system returns audio localization data relative to the direction Misty's head is facing (instead of her torso). The heading of Misty's face is the 0/360 angle relative to incoming audio localization data.

Default Behaviors

Misty II exhibits certain behaviors that override commands she receives from skills and robot applications. Some of these default behaviors are cosmetic, and others help prevent Misty from executing commands that could cause harm to Misty, the people around her, or her environment. The sections below describe some of these behaviors in detail, so you can understand how to work with (or around) them in your skill development.

Chest LED Notifications

The following chest LED notifications are enabled by default. You can turn them off by sending a request to the SetNotificationSettings endpoint in Misty's HTTP API.

• Recording Audio - While Misty is recording audio or listening for the "Hey, Misty!" key phrase, her chest LED pulses blue.
• Charging - While Misty is powered on and charging, her chest LED pulses orange. When her battery is fully charged and she is on/connected to her charger, the LED turns solid orange.
• Face Training - When you are training Misty on a new face, her chest LED displays the following notifications:
  • When the face detection phase of the training process is complete, the LED turns green.
  • When training is complete, the LED blinks green three times.
  • If training fails, the LED blinks red three times.
  • If Misty sees more than one face, the LED blinks yellow three times.
  • If Misty doesn't see a face, the LED turns yellow.
• System Updates - While Misty is performing a system update, the LED blinks white.

Audio Notifications

The following audio notifications are enabled by default. You can turn them off by sending a request to the SetNotificationSettings endpoint in Misty's HTTP API.

• Wake Word - When Misty recognizes the "Hey, Misty!" key phrase, she plays the system audio file s_SystemWakeWord.wav. You can change the default wake word sound by sending a request to the SetNotificationSettings endpoint in Misty's HTTP API.

Tally Light Notifications

The blue tally light beneath Misty's flashlight automatically turns on during certain activities to notify users that the robot is collecting personally identifiable information (PII) such as video recordings, audio recordings, and pictures. You cannot override tally light notifications. By default, the tally light turns on while Misty is:

• recording a video
• AV streaming
• creating a SLAM sensor (OCC) recording
• recording audio (both during manual audio recordings and during speech capture recordings)
• recognizing faces
• taking pictures with her RGB or fisheye camera
• taking a depth image

The tally light does not turn on when Misty is:

• detecting faces (without face recognition)
• mapping or tracking
• listening for the wake word

Scruff Reflex

Misty ignores all commands she receives for as long as she detects someone touching the CapTouch_Scruff sensor in the handle on the back of her head. This behavior is designed is to prevent Misty from moving in ways that could damage her (or the human holding her) if she's picked up while running a skill. This behavior does not cancel any running skills. It only causes Misty to ignore commands those skills invoke. When the CapTouch_Scruff touch sensor is released, Misty resumes execution of any new commands she receives.

Hazards System

Misty's software includes a built-in hazards system that is intended to prevent your robot from executing commands that could cause her harm. This system uses data from Misty's sensors to prevent Misty from driving off of surfaces that could cause her to tip or fall, such as tables, desks, or stairs. It also stops Misty from continuing to drive when she senses an obstacle nearby, or when she detects that she has bumped into an object.

In addition to protecting your robot from harm, the hazards system sends an event message each time Misty enters or exits a hazards state. You can use these messages to programmatically alter Misty's course when she detects cliffs or obstacles while autonomously navigating her environment. See the Event Types section of these docs for details on using this data in your skills.

By default, Misty's bump and time-of-flight sensors put the robot into a hazard state in the following circumstances:

• when her edge time-of-flight sensors detect a drop distance of 0.06 meters (60 mm) or greater in the direction she is moving.
• when her range time-of-flight sensors detect an obstacle 0.215 meters (215 mm) or closer in the direction she is moving.
• when one of her bump sensors becomes activated, indicating she has collided with an object.

When Misty detects an obstacle or an edge, she ignores any commands that would move her in the direction of the sensors that are in a hazard state. For example, when Misty is driving forward, she stops driving if one or more of the following happens:

• if one of her front-facing bump sensors is pressed
• if the front edge time-of-flight sensors detect a high ledge
• if the front range sensors detect an obstacle nearby

In this situation, after she stops driving, Misty ignores any forward drive commands until the system indicates none of her front bump or time-of-flight sensors are in a hazard state. To get out of a hazard state, you can code Misty to reverse directions and choose a new path.

Misty's Max Speed

To enable the hazards system to work effectively, Misty's max speed is limited to ~450 mm/s. This is the highest speed at which Misty can safely detect most ledges and stop moving, without being carried over the edge by any built-up momentum. The hazards system will be enhanced in future updates to allow for increased performance and to increase Misty's default max speed.

Using Misty's RGB Camera

This section provides an overview of Misty's RGB camera and the services used for different camera functions. For detailed descriptions of the commands you call to use these functions, see the API reference documentation.

Misty's RGB camera is embedded in the center of her visor. Misty uses this camera to take pictures, record videos, and stream image data for computer vision (CV) activities like face training, face detection, and face recognition. You can also stream video from Misty's camera to an external media server, or directly to a streaming client on the same network as your robot.

Camera Service

To use the camera for taking pictures, recording videos, and for CV activities, Misty's camera service must be enabled. By default, the camera service is enabled when Misty boots up. You can enable and disable the camera service manually by using the EnableCameraService and DisableCameraService commands.

Misty cannot run commands that use the camera service, or stream messages from any event types that use the camera service, when the camera service is disabled. You can find the full list of these commands and event types in the documentation for the DisableCameraService command.

AV Streaming Service

To use the camera for AV streaming, Misty's AV streaming service must be enabled. By default, the AV streaming service is disabled when Misty boots up. You can enable and disable the camera service manually by using the EnableAvStreamingService and DisableAvStreamingService commands.

Picture and Video Resolution

When Misty powers on, she starts up a new camera session with default resolution settings of 1920 x 1080 for recording videos and 4160 x 3120 for taking pictures. When you take a picture or start recording video without specifying a resolution, the system defaults to using the resolution already set for that action in the current camera session. When you specify a different resolution than what is set for that action in the current camera session, the session resets to use the new resolution settings. This has the following implications:

• Misty cannot reset the camera session while actively recording video. If you try to take a picture at a new resolution while Misty is recording video, she takes a picture with the resolution settings for the current camera session (instead of the new resolution that you asked for).
• If Misty is already performing computer vision (CV) activities when the camera session resets, these activities automatically resume when the new camera session is ready.
• Recording videos at 3840 x 2160 changes the max resolution for taking pictures to 3840 x 2160. If you record video at 1920 x 1080 (or lower), then Misty can use the highest possible resolution for taking pictures. If you record a video at 3840 x 2160 when the resolution for taking pictures is set to the highest resolution, the system automatically lowers the resolution for taking pictures to 3840 x 2160.

A camera session remains active with the new resolution settings until a) Misty receives a command to take a picture or record a video with different resolution settings, b) Misty reboots, or c) you disable and re-enable the camera service.

Using Misty's Display

Misty's API enables you to show images, text, videos, and animated .gifs on the robot's display.

Misty uses a layering system to show different types of content. You can create as many layers as you like, and each layer can display one piece of content. The commands for displaying each type of content are:

• DisplayImage
• DisplayText
• DisplayVideo
• DisplayWebView

You can use the following commands to adjust the settings for each layer you create:

• SetImageDisplaySettings
• SetTextDisplaySettings
• SetVideoDisplaySettings
• SetWebviewDisplaySettings

Additionally, you can use the SetDisplaySettings command to revert the display to its default settings. This command clears all layers and resets the display to show Misty's default e_Content.jpg eyes.

If you do not specify a layer when you call a display command, the content displays on the default layer for that command. The default layers for each media type are:

• ErrorDisplayLayer
• DefaultImageLayer
• DefaultTextLayer
• DefaultVideoLayer
• DefaultWebViewLayer

Connecting to adb

You can use the Android Debug Bridge (adb) command line tool to communicate with the Android operating system running on Misty's 820 processor. This is useful when you want to access assets stored on the 820, view additional logs, or configure settings exposed through Misty's Android device. You can download adb for free from the Android developer documentation.

Once you have installed adb, you can use it to establish a connection to your Misty II robot. There are two different methods we recommend for establishing this connection. Which method to use depends on the changes you have applied to your robot's Android operating system.

• If you have not applied the Android patch for improved SLAM performance, you can connect to adb using Misty's Wi-Fi IP address.
• If you have applied the Android patch and you created an unblockwifiadb file in the \data\misty\ directory on the SDcard for Misty's 820 processor, you can connect to adb using Misty's Wi-Fi IP address.
• If you have applied the Android patch for improved SLAM performance and did not create an unblockwifiadb file in the \data\misty\ directory on the SDcard for Misty's 820 processor, you can connect to adb using the IP address for a USB-to-Ethernet adapter connected to the USB port on Misty's back.

adb Over Wi-Fi

Once you have installed adb on your computer, you can connect to adb using your robot's Wi-Fi IP address. This is the same IP address that appears in the Misty App. Follow these steps:

1. Open a new command prompt / terminal window.
   • On Windows operating systems, you can open a new command prompt by launching the Start menu and searching for "command prompt".
   • On Mac operating systems, you can open a new terminal window by pressing Command + Space, searching for "terminal", and pressing Enter.
2. Enter adb connect <Misty-Wi-Fi-ip-address>:5555

You can now use adb to communicate with your robot's Android device.

adb With a USB-to-Ethernet Adapter

You can connect to adb with the IP address for a USB-to-Ethernet adapter by following these steps:

1. Follow these instructions to get an IP address for connecting to Misty's 410 processor. This requires a USB-to-Ethernet adapter, which does not arrive with Misty and must be purchased separately.
2. Connect the USB end of the adapter to the port on your robot's back.
3. Open a new command prompt / terminal window.
4. Enter sudo ssh -L 127.0.0.1:5555:10.10.10.100:5555 administrator@<USB-ETH-IP>
   • Note: If you get an error that indicates ssh is using an obsolete cypher, your robot may be using an older version of Windows IoT. You can get around the issue by entering the following: sudo ssh -c aes256-cbc -L 127.0.0.1:5555:10.10.10.100:5555 administrator@<USB-ETH-IP>
5. Enter your robot's Windows administrator password to log in to the 410 via ssh. The Windows administrator password is printed on the sticker underneath your robot.
6. Keep this connection established and open a second command line / terminal window. Enter: adb connect 127.0.0.1:5555

You can now use adb as normal until the initial connection is lost.

Toggling adb Over Wi-Fi

If you have applied the Android patch for improved SLAM performance, then you can enable adb over Wi-Fi by creating an unblockwifiadb file in the \data\misty\ directory on the SDcard for Misty's 820 processor. If the unblockwifiadb file does not exist in the correct directory, then using adb with Misty's Wi-Fi IP address is disabled by default after you apply the patch.

To enable adb over Wi-Fi using a USB to Ethernet Adapter:

1. Open a new command prompt / terminal window.
2. Connect to adb.
   • If you have already applied the patch, then adb connections over Wi-Fi are disabled by default, and you must connect to adb using the IP address for Misty's USB-to-Ethernet adapter. Follow the steps in adb With a USB-to-Ethernet Adapter.
   • If you have not applied the patch, you can connect to adb using Misty's Wi-Fi IP address. In your prompt / terminal window, enter: adb connect <wifi-ip-address>:5555
3. Create a new adb shell: adb shell
4. Enter the following commands:
   1. su
   2. touch /data/misty/unblockwifiadb (Note that the name of the file is case-sensitive.)
   3. reboot

Connecting to adb over Wi-Fi is now enabled.

To disable adb over Wi-Fi using a USB to Ethernet Adapter:

1. Open a new command prompt / terminal window.
2. Connect to adb.
   • If you have already applied the patch, then adb connections over Wi-Fi are disabled by default, and you must connect to adb using the IP address for Misty's USB-to-Ethernet adapter. Follow the steps in adb With a USB-to-Ethernet Adapter.
   • If you have not applied the patch, you can connect to adb using Misty's Wi-Fi IP address. In your prompt / terminal window, enter: adb connect <wifi-ip-address>:5555
3. Create a new adb shell: adb shell
4. Enter the following commands:
   1. su
   2. rm /data/misty/unblockwifiadb
   3. reboot

Connecting to adb over Wi-Fi is now disabled by default. You can connect to adb (and recreate the file to enable adb over Wi-Fi) by connecting to adb with the IP address for a USB-to-Ethernet Adapter.

To enable or disable adb over Wi-Fi using a USB drive:

1. Format the USB drive in FAT format.
2. In the root of that file system, create a directory named 'misty' (without the quotes).
3. Into that directory, create an empty file named 'unblockwifiadb.txt'.
4. Eject that USB drive from the computer and insert it into Misty's USB port after she has completely booted.
5. After a moment, you should see a red error message on her screen, indicating that one of the processors is rebooting. After the error message is removed, adb over wifi should be enabled.
6. Removing the drive will disable adb over wifi again.

Configuring Misty's Wake Word & Voice Activity Detection Engine

Built in

You can create an audio configuration file on Misty's 820 processor to change the robot's default wake word from "Hey, Misty" to "Hey, Snapdragon". You can also use this file to set Misty to use the WebRTC voice activity detection (VAD) engine for speech capture activities, instead of her default VAD engine.

To change Misty's default wake word and VAD engine, create (or update) a configuration file with the name audio_config.json in the /sdcard/audio/ directory of Misty's 820 processor.

Follow these steps to create the configuration file and upload it to Misty. Use the keyphrase attribute to change Misty's default wake word, and use the vad_engine attribute to change the default VAD engine.

1. Create a file called audio_config.json on your computer with the following (case-sensitive) contents. You can omit key/value pairs for the attributes you don't want to change.

{
    "misty_settings": {
        "keyphrase": "HeySnapdragon",
        "vad_engine": "WebRTC"
    }
}

2. Open a new command prompt / terminal window. Navigate to the path where audio_config.json is saved: cd </path/to/audio_config.json>
3. With Misty turned on and fully booted, connect to adb.
4. Enter the following command in your command prompt / terminal window to upload the file to Misty: adb push audio_config.json /sdcard/audio

You can revert Misty to use her default wake word or VAD engine by updating the audio_config.json file to remove the relevant key/value pairs. Alternatively, you can follow these steps to remove the audio_config.json file entirely:

1. With Misty turned on and fully booted, connect to adb.
2. Open a new command prompt / terminal window and enter the following:
   1. adb shell
   2. rm sdcard/audio/audio_config.json

Azure Wake Word

Using similar steps as above, you can also update your Misty to use an Azure wake word file.

1. Create an Azure wake word file using your Azure subscription.
2. Name the file misty.table and place it in the /sdcard/audio/ directory of Misty's 820 processor.
3. Add or update the audio_config.json file to:

   {
    "misty_settings": {
        "keyphrase_engine": "MS",
        "keyphrase_engine_configuration": {
            "keyphrase": "/sdcard/audio/misty.table"
        }
    }
   }
   

Connecting to Misty's File System

You can access your robot's Windows file system by connecting to the robot's 410 processor over your local network connection.

1. Power Misty on and make sure she is connected to the same network as your computer.
2. Connect to your robot's file system.
   1. On a Mac:
      1. Open Finder
      2. Select Go → Connect to Server from the top menu
      3. Enter smb://<robot-ip-address>/c$
   2. On Windows 10:
      1. Open File Explorer
      2. Navigate to the path \\<robot-ip-address>\c$
3. When prompted, enter the username and password printed on the sticker on the bottom of your robot's base.