Audience and Interactions

Edge real-time events and statuses

Upon a detection, Broox Vision Node generates Events that can be emitted via different mediums: MQTT, OSC and/or HTTP.

Devices sharing the edge network of Broox Vision Node can listen and react to this events.

Different use cases can use different options:

For hard real-time interaction, OSC events are emitted over UDP (Optionally available over WebRTC for browser based engines).
For quick but not real time (example: media triggering), MQTT events are sent to the local MQTT broker and can be subscribed to over plain MQTT or MQTT over WebSocket.
For presence status monitoring, Vision Node can start periodical HTTP POST requests to any accessible host.

Stage, Actors and Boxes

The state of the scene being scanned by Broox Vision Node is modeled as a virtual Stage object.

This Stage contains a number of tracked Actors (people, usually).

Each Actor has several detected features:
- Face, full body are defined by rectangular boxes.
- Analytic features like age, perceived gender, attention%, etc.
- If enabled in larger pipelines, it might contain skeletal (pose) information.

Features

Depending on the pipeline selected, and enabled or disabled via Advanced Settings or API, several features can be sent associated to each actor.

IMPORTANT: Features might not be always present if the detection has not been possible (poor lighting, image quality, etc). This can present either as the feature key not being defined or its value being null. Always check for presence.

Available on audience-analyitics-[low, medium, high, full] pipelines:
- state: string, detection state. One of "entering", "entered", "lost", "exiting" or "exited".
- age: integer 0-99.
- gender: one of "M" of "F"
- dwell: float, seconds the actor has been in view.
- current_zone: string. If zones are defined in Advanced Settings, current zone the actor is in.
- last_zone: string. If zones are defined in Advanced Settings, last zone the actor was in.
Available on audience-analyitics-[medium, high, full] pipelines:
- attention: float. Percentage (0 to 1) of time the actor has been looking at the camera.
- attention_time: float. Seconds the actor has been looking at the camera (attention*dwell).
- is_view: boolean. If attention_time is more than a defined threshold (report.is_view_time_interval).
Available on audience-analyitics-[high, full] pipelines:
- dominant_emotion: string, most frequent emotion detected so far. One of calm, sad, happy, surprise, anger.

Skeleton

Pipelines skeleton-tracking and audience-analytics-skeleton-full can also provide pose info, both as joint coordinates and detected actions (raising hand, facing back, etc).

This information is generally only sent over OSC, as it's deemed for real time interactions.

Joints are indexed as pairs of (x, y) points normalized over the image space:

NECK = 0
NOSE = 1
CENTER_HIP = 2
L_SHOULDER = 3
L_ELBOW = 4
L_WRIST = 5
L_HIP = 6
L_KNEE = 7
L_ANKLE = 8
R_SHOULDER = 9
R_ELBOW = 10
R_WRIST = 11
R_HIP = 12
R_KNEE = 13
R_ANKLE = 14
R_EYE = 15
L_EYE = 16
R_EAR = 17
L_EAR = 18

MQTT Messages

MQTT messages are emitted by default via the /device/-deviceid endpoint on the MQTT broker, but a custom path can be configured via Advanced Settings to avoid reading unrelated messages.

Messages contain a JSON-encoded payload dictionary:

{ "cmd": "trigger",
  "class": "profile",
  "action": "in",
  "id": "mabotix",
  "age": 33,
  "gender": "M", ... }

cmd can be trigger (used for event triggering) or status (periodic status updates).

Triggers

Triggers are emited on the appearance or disappearance of an actor in the stage.

class for audience analytics will be profile.
id is a random string identifying the actor.
action is:
- in (actor appeared)
- out (actor disappeared/timed out)

Depending on the selected pipeline settings, extra data might be added to the event.

Note that in messages might not inmediately have profiling characteristics.

Status Updates

Periodic status updates are emitted via a cmd:status payload.

{ "cmd": "status",
  "sent-by": "stage",
  "camera": "HD Camera 1",
  "boxes": [{...}]
}

boxes will be a list of dictionaries with each known active actor. note: boxes might be renamed to actors in a future release

OSC Messages

When enabling OSC, boxes and features will be sent over OSC protocol.

By default to localhost:15000, this can be changed in the Advanced Settings.
Two kinds of messages are sent, each in separate OSC bundles:
- /snap frame# [actor id]: Every frame, containing a list of active actors*.
- /vac messages, several per present actor:
  - /vac actor "actor-id" actor-state frame# [boxes]*
    - actor-state, string, one of "entering", "entered", "lost", "exiting"
    - frame# is the same as /snap
    - boxes is a list of detected associated boxes, like "face", "person", "blob"...
  - /vac box "actor-id" box-type x y w h (one per each "box")
    - box-type is "face", "person" or "blob" depending on the detected object.
    - x, y are top-left coordinates of the box normalized over the frame, e.g (0.5, 0.5) will be in the middle of the frame)
    - w, h are width and height of the box, normalized over the frame.
  - /vac skel "actor-id" [(x,y) keypoints]+ ["pose"]*: Skeleton (pose) of the actor.
    - x, y are normalized coordinates of each joint over the whole image frame. If the joint is not detected, will be set to (-1, -1).
    - "pose" are detected pose actions like "left_hand_up", "right_hand_up", "both_hands_up", "points_left", "points_right", "both_hands_on_chest", "facing_back", etc.
- /vac prop "actor-id" feature [values]*: Feature properties, like age, dwell time, etc.

WebRTC endpoint

Due to limitations in browser based players, OSC data can also be accessed over WebRTC. This is implemented in the Broox.js Library.

POST Data

Vision Node can be configured to periodically POST the status of the "stage" as a JSON object, using the Advanced Settings.

The post URL should point to an HTTP/S service accepting HTTP POST requests with a Content-type: application/json.

The data will be POSTed as a raw JSON:

POST /stage HTTP/1.1
Host: my.server.url
Content-type: application/json
Content-length: 9999

[{"boxtype": "person", "id": "getef", "tags": [], 
  "dwell": 12, "attention": 100.0, "dominant_emotion": "neutral", 
  "face": true, "age": 63, "gender": "M", "features": ["handbag"]},
  ....
]

Configuration in `visionnode.ini`

All methods can be enabled or disabled using Vision Node Advanced Settings UI.

For provisioning purposes, they can also be enabled in the [pipeline] section(s) of visionnode.ini:

;; Pipeline settings and parameter overrides

[pipeline]

;; For Studio Analytics
enable_analytics = true

;; HTTP post
stage.enable_http = true
stage.send_status_post_url =  http://localhost:8085
; seconds between updates
stage.send_status_time_interval = 5.0 

;; OSC
osc.enabled = true
osc.target_host = 127.0.0.1
osc.target_port = 15000

;; MQTT
stage.enable_mqtt = true
stage.send_status_mqtt_channel = /vision