Broox Vision Node

Broox Vision node is an Intel OpenVino™ based vision engine.

It can read video from a camera (either webcam or network camera) and process it to extract interesting features, such as people, faces, approximate age and gender, etc. to use for targeting and/or demographic analysis.

Installing Broox Vision Node

Windows

Note: Windows N versions require installing the "Media Package" or Vision Node won't start

Download Broox Vision Node X.X.X.exe or the combo BrooxPlatform-X.X.exe installer
Double click on the installer and follow the screens.
Check if you want a desktop launcher icon.
Continue the installation clicking Next, Next. Wait for the installer to finish.
Close the installer.
You can now launch Broox Vison Node from the Start Menu (or desktop icon if you selected it).

Ubuntu Linux

Download broox-vision-node_X.X.X.amd64.deb
Ubuntu 20.04, open a terminal and cd to the download directory:

sudo apt install ./broox-vision-node_X.X.X.amd64.deb

To start up type:

/opt/broox/BrooxVisionNode/BrooxVisionNode

Note: On certain distributions it might be necessary to install libv4l and python-tk from the distribution repositories.

Configuration

Expand the Configuration section.
Select:
1. Controller: if the controller is running, it will show in the list.
2. Pipeline: Determines the processing done to the captured images. See "Pipeline Selection" below.
3. Video Input: Select the type of camera. See "Camera Setup" below.
4. The bottom checkboxes are used for interactive use and configuration:
  1. Display previews opens an OS window showing the capture results
  2. Configuration mode opens a webserver on http://localhost:5000 where you can configure the pipeline.
Press Start to run the Vision Node.

Video Input: Camera Set Up

With the Broox Vision Node you can use either a standard (USB Video Capture) camera/webcam/digitizer or connect to a RTSP/HTTP video stream.

In the Video Input Section you have:

Camera Type:
1. USB Webcam: Any USB Video Capture device.
2. Network camera: A URL pointing to a network stream.
3. Intel RealSense Camera: For 3D/Depth capture.
4. Custom Definition: To edit the .json file by hand
Camera Type Settings:
1. For USB Webcam, it lists the connected decives, supported resolutions and FPS.
2. For Network Camera offers an URL text field.
3. For Intel Realsense offers the same dropdown as a USB Webcam.
4. For Kinect Azure Camera offers the same dropdown as USB Webcam and a Body Tracking checkbox
5. For a Custom definition, a instruction to edit the camera.json file is displayed.
The Transform Section allows to flip, rotate and crop the input video:

Clipping, rotation, flip examples

Base Image
Rotated 90º

Horizontal Flip

Crop from x:50, y:40 with a width of 200 pixels and height of 200 pixels

Default Pipelines

Pipelines are default presets of different vision node processors. Vision nodes come with ready-to-use presets tailored for specific use cases. Below is the list of currently available pipelines:

Audience analytics low

Target use cases:

Measure analytics in digital signage displays
Trigger content based on audience

Edge Detections:

Detectors	Description	Default API output
Person	Tracks people in the scene	To set in .ini file
Age	Identifies a person’s age	To set in .ini file
Gender	Identifies a person’s gender	To set in .ini file
Dwell time	Identifies how long a person stays	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Dwell time	Calculates the average staying time
Age distribution	Provides the percentage distribution of age ranges
Gender distribution	Provides the percentage distribution of genders
Historics	Offers historical data for the above metrics by date range

Audience analytics mid

Target use cases:

Measure analytics in digital signage displays
Trigger content based on audience

Edge Detections:

Detectors	Description	Default API output
Person	Tracks people in the scene	To set in .ini file
Age	Identifies a person’s age	To set in .ini file
Gender	Identifies a person’s gender	To set in .ini file
Dwell time	Identifies how long a person stays	To set in .ini file
Attention Percentage	Captures attention time	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Viewers	Counts total views of the screen
Average attention time	Calculates the average attention time per person
Dwell time	Calculates the average staying time
Age distribution	Provides the percentage distribution of age ranges
Gender distribution	Provides the percentage distribution of genders
Historics	Offers historical data for the above metrics by date range

Audience analytics high

Target use cases:

Measure analytics in digital signage displays
Trigger content based on audience

Edge Detections:

Detectors	Description	Default API output
Person	Tracks people in the scene	To set in .ini file
Age	Identifies a person’s age	To set in .ini file
Gender	Identifies a person’s gender	To set in .ini file
Emotion	Identifies a person’s mood	To set in .ini file
Dwell time	Identifies how long a person stays	To set in .ini file
Attention Percentage	Captures attention time	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Viewers	Counts total views of the screen
Average attention time	Calculates the average attention time per person
Dwell time	Calculates the average staying time
Age distribution	Provides the percentage distribution of age ranges
Gender distribution	Provides the percentage distribution of genders
Emotion Distribution	Provides the percentage distribution of moods
Historics	Offers historical data for the above metrics by date range

Audience analytics full

Target use cases:

Measure analytics in digital signage displays
Trigger content based on audience

Edge Detections:

Detectors	Description	Default API output
Person	Tracks people in the scene	To set in .ini file
Age	Identifies a person’s age	To set in .ini file
Gender	Identifies a person’s gender	To set in .ini file
Emotion	Identifies a person’s mood	To set in .ini file
Dwell time	Identifies how long a person stays	To set in .ini file
Attention Percentage	Captures attention time	To set in .ini file
Pose estimation	Tracks a person’s skeletal joints	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Viewers	Counts total views of the screen
Average attention time	Calculates the average attention time per person
Dwell time	Calculates the average staying time
Age distribution	Provides the percentage distribution of age ranges
Gender distribution	Provides the percentage distribution of genders
Emotion Distribution	Provides the percentage distribution of moods
Historics	Offers historical data for the above metrics by date range

Skeleton Tracking

Target use cases:

Develop interactive content for digital signage displays

Edge Detections:

Detectors	Description	Default API output
Pose estimation	Tracks a person’s skeletal joints	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Dwell time	Calculates the average staying time
Historics	Offers historical data for the above metrics by date range

Skeleton Tracking + Audience analytics high

Target use cases:

Measure analytics in digital signage displays
Trigger content based on audience
Develop interactive content for digital signage displays

Edge Detections:

Detectors	Description	Default API output
Person	Tracks people in the scene	To set in .ini file
Age	Identifies a person’s age	To set in .ini file
Gender	Identifies a person’s gender	To set in .ini file
Emotion	Identifies a person’s mood	To set in .ini file
Dwell time	Identifies how long a person stays	To set in .ini file
Attention Percentage	Captures attention time	To set in .ini file
Pose estimation	Tracks a person’s skeletal joints	To set in .ini file

Cloud Analytics:

Key Metric	Description
Traffic	Counts total traffic
Viewers	Counts total views of the screen
Average attention time	Calculates the average attention time per person
Dwell time	Calculates the average staying time
Age distribution	Provides the percentage distribution of age ranges
Gender distribution	Provides the percentage distribution of genders
Emotion Distribution	Provides the percentage distribution of moods
Historics	Offers historical data for the above metrics by date range

Analytics message sending

Selecting Enable analytics message sending you enable the analysis data sending to Broox Studio

Preview Mode

Selecting Display previews of processors on a window you can start the vision node and check the results of the processing.

Advanced Settings

Introduction

AST (Advanced Settings Tool) is a robust and intuitive tool designed to provide users with advanced configuration options for the Vision Node (VN). It offers the flexibility to define areas of interest, fine-tune the camera calibration and configurating all parameters of the selected pipeline, ensuring that the Vision Node operates optimally for specific use cases.

Access

To access the Advanced Settings (AST), launch the Vision Node application on your system. Once the application is up and running, navigate to the bottom-right corner of the window, where you’ll find the 'Open Advanced Settings' button. Click on this button to open the settings interface and begin configuring the tool.

Camera Parameters Section

Introduction

This section provides essential details about the camera, such as its name, resolution, frames per second (FPS), rotation, and whether the image is flipped. Additionally, users can input necessary calibration values for the camera. This section ensures that both intrinsic and extrinsic camera parameters are properly configured, optimizing the camera’s performance.

How to Indicate Camera Preferences

To make the configuration process easier, two dropdown menus are provided: Manufacturer and Model. Once a manufacturer and model are selected, the Sensor, Focal Length, and Aspect Ratio fields will automatically populate. If the camera is not listed in the dropdown, select the "Custom" option under Manufacturer and manually input the camera's name (Model), sensor details, and focal length.

Calibration

Introduction

The Calibration section allows you to adjust both the extrinsic and intrinsic parameters of the camera. Here, users can configure the Sensor and Focal Length, while also marking a square on the image that aligns with real-world objects. Additionally, you can input the camera's distance, height, and the square's side length.

Calibrate the Camera

Start by activating the Calibration panel by toggling it on.

If the Manufacturer and Model fields are pre-filled, the Sensor and Focal Length will automatically be populated. If not, manually enter the required values.

To create a Marker Area, click the '+' button. A square will appear on the display, and you should position its vertices to match the real-world scene. For optimal accuracy, it’s recommended to physically outline a square in the environment and then replicate it on the display using the AST tool.

Once the Marker Area is aligned correctly, complete the remaining fields by entering the Distance (from the camera to the square), Camera Height, and Side Length (of the square).

Areas of Interest Section

Introduction

This section allows users to define areas of interest within the environment. A common application is setting up detection zones for specific regions, such as the entrance of a building, to track the number of people passing through.

Define Areas of Interest

To add a new detection area, click the "Add" button. The area will appear in editing mode, allowing you to adjust its shape and position. To enter editing mode, simply select the detection area from the list. You can rename the area by clicking the pencil icon or double-clicking the name field. If needed, you can delete the area by clicking the trash can icon.

Pipeline Configuration

Introduction

The Pipeline Configuration section enables the customization of processors for the selected pipeline. Each processor serves a unique function, and based on the requirements, users can enable or disable them to optimize the system’s processing capacity.

Pipeline Parameters

Traffic Processor: This processor is responsible for locating and identifying actors in the image, assigning them a unique identifier. This processor is integral to the system and cannot be disabled. The modifiable parameters for this processor are:
Threshold: Controls the confidence required for confirming a detection. Higher values mean greater confidence is needed for detection.
Precision: Determines the quality of the detection. A higher value ensures better accuracy. The Traffic processor visualizes detections by drawing a rectangle around the actor and displaying information like the actor's ID, current zone, last zone, and state.

Emotion Processor: This processor analyzes the actor’s facial expressions, categorizing them into neutral, happy, sad, angry, or surprised. The adjustable parameter is:
Precision: Determines the accuracy of the emotion detection. The emotion label will be displayed on the actor’s image.

Age & Gender Processor: This processor estimates the actor’s age and gender. The adjustable parameter is:
Precision: Impacts the accuracy of age and gender predictions. The labels "Age" and "Gender" will be displayed on the actor's image.

Attention Processor: This processor measures the actor’s attention level by calculating the tilt of the face in relation to the camera. The configurable parameters are:
Angle: Adjusts the attention point.
Precision: Controls the accuracy of the attention level. The "% Attention" label will appear on the actor’s image to indicate their attention level.

Pose Processor: The Pose processor detects the actor’s joints and creates a virtual skeleton to predict their posture. The adjustable parameter is:
Precision: Controls the accuracy of the posture prediction. A skeleton representation will be displayed on the actor’s image.

API Output Processor: This processor sends the collected data to external systems. Similar to the Traffic processor, it is essential and cannot be disabled. The API Output allows for the use of three communication protocols:
MQTT: Requires a communication channel
HTTP: Needs a POST endpoint and timeout configuration.
OSC: Requires an IP address and port. Multiple protocols can be enabled at once.

Save and Exit

Once all settings have been configured, click the "Save" button to apply your changes. If any mandatory fields are left incomplete, a notification will prompt you to fill in the missing information before proceeding.

In the event that the AST is unexpectedly closed, it retains the most recent configuration in memory. This provides you with the option to either continue with the latest modifications (Keep current Vision Node settings) or revert to the previously saved configuration (Restore last saved settings), ensuring a seamless experience and minimal disruption.

Usage Example

Connect the Camera

Start by connecting the camera to your computer.

Launch the Vision Node

Open the Vision Node application, select the 'Controller' running the pipeline, and choose the 'audience-analytics-low' pipeline.

Configure Video Input

In the "Video Input" section, select 'USB Webcam' as the Camera Type, and choose the camera you connected earlier. Ensure the two checkboxes in the bottom-left corner are checked, then click "Start."

Access AST

With the Vision Node running, click the 'Open Advanced Settings' button in the bottom-right corner to access the AST.

Configure Camera Parameters

Search for your camera in the Model list under the Manufacturer dropdown. If not listed, select "Custom", and manually input the camera's name, sensor details, and focal length.

Create Marker Area

Click the '+' button to create a marker area. Position the vertices in edit mode, and fill in the fields for Distance, Camera Height, and Side Length

Define Detection Zones

Add detection zones by selecting the areas of interest and configuring their size and placement.

Adjust Detection Presets

Enable or disable specific processors and tweak their settings according to your needs.

Detection example

This is an example of detection with the active Age & Gender processor and the sample zones we saw earlier.

Save Settings

After configuring all parameters, click "Save" to finalize your settings.