Sensor & Skill Data Types

The following are Misty’s available sensor and skill data types. You receive this data when you register for events using Misty's on-robot JavaScript API or when you subscribe to a WebSocket connection from an external device.

You can filter all data types to (a) return only a specified subset of the data and (b) check current values before the data is sent.

Note: All of Misty's sensor & skill data structures are subject to change.

TimeOfFlight

Misty has four time-of-flight sensors that provide raw proximity data (in meters) in a single stream. The TimeOfFlight WebSocket sends this data any time a time-of-flight sensor is triggered. It is possible for proximity data to be sent as frequently as every 70 milliseconds, though it can be significantly slower. It is not sent at timed intervals.

Sample time-of-flight sensor data:

TimeOfFlight{
    "EventName":"TimeOfFlight",
    "Message":{
        "Created":"2018-03-30T20:36:46.5816862Z",
        "DistanceInMeters":0.184,
        "Expiry":"2018-03-30T20:36:46.9316897Z",
        "SensorId":"CD727A0A",
        "SensorPosition":"Right"
    },
    "Type":"TimeOfFlight"
}

FaceRecognition (Beta)

You can subscribe to the FaceRecognition WebSocket to obtain data on both face detection and face recognition events.

The EventName value is the name you provide when you register the WebSocket connection.

If face recognition is running on the robot, and a previously trained face is recognized, the PersonName value is the name previously assigned to that face. The PersonName value is unknown_person if an untrained/unknown face is detected. The PersonName value is null if face recognition is not currently running.

TrackId is reserved data that may change in the future.

Sample FaceRecognition data for a face recognition event:

FaceRecognition{
    "EventName":"MyFaceRecognition",
    "Message":{
        "Bearing":-3,
        "Created":"2018-07-02T16:26:20.1718422Z",
        "Distance":71,
        "Elevation":3,
        "Expiry":"2018-07-02T16:26:20.9218446Z",
        "PersonName":"Barkley",
        "SensorId":null,
        "SensorName":null,
        "TrackId":0
    },
    "Type":"FaceRecognition"
}

LocomotionCommand

LocomotionCommand WebSocket data is sent every time the linear or angular velocity of the robot changes. It is not sent at timed intervals.

Sample locomotion data:

LocomotionCommand{
    "EventName":"LocomotionCommand",
    "Message":{
        "ActionId":0,
        "AngularVelocity":0,
        "Created":"2018-04-02T22:59:39.3350238Z",
        "LinearVelocity":0.30000000000000004,
        "UsePid":true,
        "UseTrapezoidalDrive":true,
        "ValueIndex":0
    },
    "Type":"LocomotionCommand"
}

HaltCommand

HaltCommand WebSocket data is sent every time the robot stops and contains the date and time of the event. It is not sent at timed intervals.

SelfState

The SelfState WebSocket provides a variety of data about Misty’s current internal state, including:

  • battery charge, voltage, and charging status
  • IP address
  • affect
  • position and orientation ("pose")
  • SLAM status
  • sensor messages

Note: There are a number of fields in the WebSocket data structure that are reserved for future use and which may be empty or contain null data:

  • Acceleration
  • BumpedState
  • CurrentGoal
  • MentalState
  • Personality
  • PhysiologicalBehavior

SelfState WebSocket messages are sent even if the data has not changed, as the data is sent via timed updates, instead of being triggered by events. The SelfState WebSocket can send data as frequently as every 100ms, though it is set by default to 250ms. To avoid having to handle excess data, you can change the message frequency for the WebSocket with the DebounceMs field, as shown in the lightSocket.js JavaScript helper.

Sample SelfState data:

SelfState {
    "eventName": "SelfState",
    "message": {
        "acceleration": null,
        "batteryChargePercent": 0,
        "batteryVoltage": 0,
        "bumpedState": {
            "disengagedSensorIds": [],
            "disengagedSensorNames": [],
            "disengagedSensors": [],
            "engagedSensorIds": [],
            "engagedSensorNames": [],
            "engagedSensors": []
        },
        "currentGoal": {
            "animation": null,
            "animationId": null,
            "directedMotion": null,
            "directedMotionBehavior": "SupersedeAll",
            "haltActionSequence": false,
            "haltAnimation": false
        },
        "isCharging": false,
        "localIPAddress": "10.0.1.160",
        "location": {
            "bearing": 2.1161846957231862,
            "bearingThreshold": {
                "lowerBound": 0,
                "upperBound": 0
            },
            "distance": 0.049783250606253104,
            "distanceThreshold": {
                "lowerBound": 0,
                "upperBound": 0
            },
            "elevation": -0.009038750542528028,
            "elevationThreshold": {
                "lowerBound": 0,
                "upperBound": 0
            },
            "pose": {
                "bearing": 2.1161846957231862,
                "created": "2018-09-17T21:01:35.7312016Z",
                "distance": 0.049783250606253104,
                "elevation": -0.009038750542528028,
                "frameId": "WorldOrigin",
                "framesProvider": {
                    "rootFrame": {
                        "created": "2018-09-17T18:21:22.8435331Z",
                        "id": "RobotBaseCenter",
                        "isStatic": true,
                        "linkFromParent": {
                            "isStatic": true,
                            "parentFrameId": "",
                            "transformFromParent": {
                                "bearing": 0,
                                "distance": 0,
                                "elevation": 0,
                                "pitch": 0,
                                "quaternion": {
                                    "isIdentity": true,
                                    "w": 1,
                                    "x": 0,
                                    "y": 0,
                                    "z": 0
                                },
                                "roll": 0,
                                "x": 0,
                                "y": 0,
                                "yaw": 0,
                                "z": 0
                            },
                            "transformToParent": {
                                "bearing": 3.141592653589793,
                                "distance": 0,
                                "elevation": 0,
                                "pitch": 0,
                                "quaternion": {
                                    "isIdentity": true,
                                    "w": 1,
                                    "x": 0,
                                    "y": 0,
                                    "z": 0
                                },
                                "roll": 0,
                                "x": 0,
                                "y": 0,
                                "yaw": 0,
                                "z": 0
                            }
                        }
                    }
                },
                "homogeneousCoordinates": {
                    "bearing": 2.1161846957231862,
                    "distance": 0.049783250606253104,
                    "elevation": -0.009038750542528028,
                    "pitch": -0.18708743155002594,
                    "quaternion": {
                        "isIdentity": false,
                        "w": 0.99558717,
                        "x": -0.008987884,
                        "y": -0.09339719,
                        "z": -0.0015491969
                    },
                    "roll": -0.017920719552386073,
                    "x": -0.025824014097452164,
                    "y": 0.04255925118923187,
                    "yaw": -0.001430802591800146,
                    "z": 0.00044997225631959736
                },
                "pitch": -0.18708743155002594,
                "roll": -0.017920719552386073,
                "x": -0.025824014097452164,
                "y": 0.04255925118923187,
                "yaw": -0.001430802591800146,
                "z": 0.00044997225631959736
            },
            "unitOfMeasure": "None"
        },
        "mentalState": {
            "affect": {
                "arousal": 0,
                "dominance": 0,
                "valence": 0
            },
            "created": "2018-09-17T21:01:35.7312016Z",
            "personality": {
                "agreeableness": 0,
                "conscientiousness": 0,
                "extraversion": 0,
                "neuroticism": 0,
                "openness": 0
            },
            "physiologicalBehavior": {
                "hunger": {
                    "isEating": false,
                    "level": 0
                },
                "sleepiness": {
                    "isSleeping": false,
                    "level": 0
                }
            }
        },
        "occupancyGridCell": {
            "x": 0,
            "y": 0
        },
        "occupancyGridCellMeters": 0,
        "orientation": {
            "pitch": -0.18708743155002594,
            "roll": -0.017920719552386073,
            "yaw": -0.001430802591800146
        },
        "position": {
            "x": -0.025824014097452164,
            "y": 0.04255925118923187,
            "z": -0.025824014097452164
        },
        "slamStatus": {
            "runMode": "Exploring",
            "sensorStatus": "Ready",
            "status": 132
        },
        "stringMessages": null,
        "touchedState": {
            "disengagedSensors": [],
            "engagedSensors": []
        }
    },
    "type": "SelfState"
}

WorldState

The WorldState WebSocket sends data about the environment Misty is perceiving, including:

  • the locations of perceived objects
  • the times they were perceived

WorldState WebSocket messages are sent even if the data has not changed, as the data is sent via timed updates, instead of being triggered by events. The WorldState WebSocket can send data as frequently as every 100ms, though it is set by default to 250ms. To avoid having to handle excess data, you can change the message frequency for the WebSocket with the DebounceMs field, as shown in the lightSocket.js JavaScript helper.

BatteryCharge

The BatteryCharge data stream provides information about the state of Misty's battery, including charge percentage, voltage, and charging status. The BatteryCharge data stream sends messages at timed intervals of five seconds.

Sample BatteryCharge data:

BatteryCharge {
    "eventName":"BatteryCharge",
    "message": {
        "batteryChargePercent": 0.51,
        "created": "2019-01-09T19:18:53.3921197Z",
        "currentVoltage": 6.949,
        "expiry": "2019-01-09T19:19:03.3921197Z",
        "isCharging": false,
        "sensorId": "charge",
        "sensorName": null
    },
    "type":"BatteryCharge"
}

ActuatorPosition

Available for Misty II only

The ActuatorPosition data stream provides information about the position of the actuators responsible for controlling the movement of Misty's head and arms. ActuatorPosition data is sent at timed intervals you define when you register for ActuatorPosition messages.

In the ActuatorPosition data object, the value of the sensorName property is the name of the actuator you are receiving information about (Actuator_HeadPitch, Actuator_HeadYaw, Actuator_HeadRoll, Actuator_LeftArm, or Actuator_RightArm). The value property holds a number indicating the position of the actuator (in radians).

Note: When you subscribe to the ActuatorPosition data stream, you should specify which actuator you want to receive messages about. For example, the following code from Misty's on-robot JavaScript API shows how to use a property comparison test to get data from the sensor for the actuator responsible for controlling the movement of Misty's right arm:

// Register for ActuatorPosition data for the actuator for Misty's right arm
misty.AddPropertyTest("ActuatorPosition", "sensorName", "==", "Actuator_RightArm", "string");
misty.RegisterEvent("ActuatorPosition", "ActuatorPosition", 1000, true);

// Callback function that triggers each time ActuatorPosition data is sent
function _ActuatorPosition(data) {
// Do something with the data
}

Sample ActuatorPosition data:

ActuatorPosition {
    "eventName": "ActuatorPosition",
    "message": {
        "actuatorId": "NBQEA0",
        "created": "2019-01-09T20:15:38.0870356Z",
        "expiry": "2019-01-09T20:15:39.0870356Z", 
        "sensorId": "ala",
        "sensorName": "Actuator_LeftArm",
        "value":-5.09
    },
    "type":"ActuatorPosition"
}

BumpSensor

Available for Misty II only

The BumpSensor data stream sends information each time one of the bump sensors on Misty's base is pressed or released. In the BumpSensor data object, the value of the sensorName property is the name of the bump sensor that triggered the event (Bump_FrontRight, Bump_FrontLeft, Bump_RearRight, or Bump_RearLeft). The value of the isContacted property is a boolean indicating whether the bump sensor was pressed (true) or released (false).The BumpSensor data object also provides “pose” information about Misty at the time of the event. For more about pose, see the mapping section of the API Explorer documentation.

For an example that shows how to register for and use data from BumpSensor events with Misty's on-robot JavaScript API, see the Bump Sensors skill tutorial.

Sample BumpSensor data:

BumpSensor {
    "eventName": "BumpSensor",
    "message": {
        "created": "2019-01-09T20:32:27.1036178Z",
        "expiry": "2019-01-09T20:32:27.2286178Z",
        "isContacted": true,
        "pose": {
            "bearing": -0.5585993432483588,
            "created": "2019-01-09T16:36:46.7089139Z",
            "distance": 0.14150971628586034,
            "elevation": 0,
            "frameId": "RobotBaseCenter",
            "framesProvider": {
                "rootFrame": {
                    "created": "2019-01-09T16:36:46.6151597Z",
                    "id": "RobotBaseCenter",
                    "isStatic": true,
                    "linkFromParent": {
                        "isStatic": true,
                        "parentFrameId": "",
                        "transformFromParent": {
                            "bearing": 0,
                            "distance": 0,
                            "elevation": 0,
                            "pitch": 0,
                            "quaternion": {
                                "isIdentity": true,
                                "w": 1,
                                "x": 0,
                                "y": 0,
                                "z": 0
                            },
                            "roll": 0,
                            "x": 0,
                            "y": 0,
                            "yaw": 0,
                            "z": 0
                        },
                        "transformToParent": {
                            "bearing": 3.141592653589793,
                            "distance": 0,
                            "elevation": 0,
                            "pitch": 0,
                            "quaternion": {
                                "isIdentity": true,
                                "w": 1,
                                "x": 0,
                                "y": 0,
                                "z": 0
                            },
                            "roll": 0,
                            "x": 0,
                            "y": 0,
                            "yaw": 0,
                            "z": 0
                        }
                    }
                }
            },
            "homogeneousCoordinates": {
                "bearing": -0.5585993432483588,
                "distance": 0.14150971628586034,
                "elevation": 0,
                "pitch": 0,
                "quaternion": {
                    "isIdentity": true,
                    "w": 1,
                    "x": 0,
                    "y": 0,
                    "z": 0
                },
                "roll": 0,
                "x": 0.11999999731779099,
                "y": -0.07500000298023224,
                "yaw": 0,
                "z": 0
            },
            "pitch": 0,
            "roll": 0,
            "x": 0.11999999731779099,
            "y": -0.07500000298023224,
            "yaw": 0,
            "z": 0
        },
        "sensorId": "bfr",
        "sensorName": "Bump_FrontRight"
    },
    "type":"BumpSensor"
}

DriveEncoders

Available for Misty II only

The DriveEncoders data stream provides information about the angular velocity (in radians per second) and rotation (in radians) for Misty's left and right encoders. DriveEncoders data is sent at timed intervals you define when you register for DriveEncoders messages.

Sample DriveEncoders sensor data:

DriveEncoders {
    "eventName": "DriveEncoders",
    "message": {
        "created": "2019-01-09T21:39:32.0681324Z",
        "expiry": "2019-01-09T21:39:33.0681324Z", 
        "leftEncoderAngularVelocity": 0,
        "leftEncoderRotations": 0,
        "rightEncoderAngularVelocity": 1.4223481644949679e-36,
        "rightEncoderRotations": 3.430562713439764e-34,
        "sensorId": "enc",
        "sensorName": null
    },
    "type":"DriveEncoders"
}

IMU

Available for Misty II only

The IMU data stream provides information from Misty's Inertial Measurement Unit (IMU) sensor. It includes information about:

  • the pitch, yaw, and roll orientation angles of the sensor (in radians)
  • the force (in meters per second squared) currently applied to the sensor along its pitch, yaw, and roll rotational axes
  • the force (in meters per second squared) currently applied to the sensor along its X, Y, and Z axes

By default, IMU data is sent at timed intervals of five seconds.

Sample IMU data:

IMU {
    "eventName": "IMU",
    "message": {
        "created": "2019-01-09T21:47:53.7607457Z",
        "expiry": "2019-01-09T21:47:54.1607457Z",
        "pitch":0.193,
        "pitchAcceleration": 0.057,
        "roll": 0.9,
        "rollAcceleration": -0.1,
        "sensorId": "imu",
        "sensorName": null,
        "xAcceleration": 0.75,
        "yAcceleration": 0.73,
        "yaw": 2.004,
        "yawAcceleration": 0.096,
        "zAcceleration": 20.04
    },
    "type":"IMU"
}

SerialMessage

Available for Misty II only

The SerialMessage data stream provides information sent to Misty by external hardware connected to the ports on her back. SerialMessage events trigger when Misty receives data sent through one of these ports.

To send SerialMessage data to Misty from an Arduino, use:

// ARDUINO

Serial.println("<your_data_to_Misty>")

Sending data to Misty as a JSON string can make it easier to parse the data in your skill code. This example shows how you can format data from a temperature and pressure sensor connected to an Arduino:

// ARDUINO

Serial.println("{\"temperature\":\""+String(<temp_value>)+"\",\"pressure\":\""+String(<pressure_value>)+"\"}");

Handle this data with Misty's on-robot JavaScript API by registering for SerialMessage events. Add SerialMessage as an additional return property, and parse the data in the _SerialMessage() callback that triggers when the data is ready.

// MISTY 

// Register for SerialMessage events and add SerialMessage as a return property
misty.AddReturnProperty("SerialMessage", "SerialMessage");
misty.RegisterEvent("SerialMessage", "SerialMessage", 0, true);

function _SerialMessage(data) {
    if(data !== undefined && data !== null) {
        // Parse SerialMessage data and assign it to a variable
        var obj = JSON.parse(data.AdditionalResults[0].Message);
        var temp = obj.temperature;
        var pressure = obj.pressure;
    }
}

For more about events and callbacks, see the Data Handling: Events & Callbacks section of On-Robot JavaScript API Architecture.

TouchSensor

Available for Misty II only

The TouchSensor data stream sends information each time one of the capacitive touch sensors on Misty's head is touched or released. In the TouchSensor data object, the value of the sensorName property is one of the following strings, indicating which sensor triggered the event:

  • CapTouch_Chin
  • CapTouch_ChinLeft
  • CapTouch_ChinRight
  • CapTouch_HeadLeft
  • CapTouch_HeadRight
  • CapTouch_HeadBack
  • CapTouch_HeadFront
  • CapTouch_HeadTop
  • CapTouch_Scruff

The value of the sensorPosition property is a string indicating the location of the sensor. This value corresponds with the name of the sensor that triggered the event (i.e. when the CapTouch_Chin sensor triggers the event, the value of sensorPosition is Chin). The value of isContacted is a boolean indicating whether the sensor was touched (true) or released (false). The TouchSensor data object also provides “pose” information about Misty at the time of the event. For more about pose, see the mapping section of the API Explorer documentation.

Sample TouchSensor data:

TouchSensor{  
    "eventName":"TouchSensor",
    "message":{  
       "created":"2019-01-09T22:55:20.5300385Z",
       "expiry":"2019-01-09T22:55:21.5300385Z",
       "isContacted":true,
       "pose":{  
          "bearing":1.0303768429594167,
          "created":"2019-01-09T20:11:03.2951639Z",
          "distance":0.09899494809418499,
          "elevation":-0.9409567798080669,
          "frameId":"RobotHead",
          "framesProvider":{  
             "rootFrame":{  
                "created":"2019-01-09T20:11:03.2013843Z",
                "id":"RobotBaseCenter",
                "isStatic":true,
                "linkFromParent":{  
                   "isStatic":true,
                   "parentFrameId":"",
                   "transformFromParent":{  
                      "bearing":0,
                      "distance":0,
                      "elevation":0,
                      "pitch":0,
                      "quaternion":{  
                         "isIdentity":true,
                         "w":1,
                         "x":0,
                         "y":0,
                         "z":0
                      },
                      "roll":0,
                      "x":0,
                      "y":0,
                      "yaw":0,
                      "z":0
                   },
                   "transformToParent":{  
                      "bearing":3.141592653589793,
                      "distance":0,
                      "elevation":0,
                      "pitch":0,
                      "quaternion":{  
                         "isIdentity":true,
                         "w":1,
                         "x":0,
                         "y":0,
                         "z":0
                      },
                      "roll":0,
                      "x":0,
                      "y":0,
                      "yaw":0,
                      "z":0
                   }
                }
             }
          },
          "homogeneousCoordinates":{  
             "bearing":1.0303768429594167,
             "distance":0.09899494809418499,
             "elevation":-0.9409567798080669,
             "pitch":-1.399999976158142,
             "quaternion":{  
                "isIdentity":false,
                "w":0.7648422,
                "x":0,
                "y":-0.64421767,
                "z":0
             },
             "roll":0,
             "x":0.029999999329447746,
             "y":0.05000000074505806,
             "yaw":0,
             "z":0.07999999821186066
          },
          "pitch":-1.399999976158142,
          "roll":0,
          "x":0.029999999329447746,
          "y":0.05000000074505806,
          "yaw":0,
          "z":0.07999999821186066
       },
       "sensorId":"cap",
       "sensorName":"CapTouch_HeadLeft",
       "sensorPosition":"HeadLeft"
    },
    "type":"TouchSensor"
 }

AudioPlayComplete

AudioPlayComplete WebSocket data is sent every time Misty finishes playing an audio file. It is not sent at timed intervals.

AudioPlayComplete {
    "eventName":"AudioPlayComplete",
    "message":{
        "created":"2019-04-08T20:54:36.7051135Z",
        "metaData":{
            "directory":"Idle",
            "duration":0,
            "name":"002-Ahhh.wav",
            "vad":[0,0,-0.5]
        }
    }
}