Skip to content

Format of Video Sources (Cameras)

Video Sources (a.k.a "cameras") are defined in the visionnode.ini file under the [camera] header.

[camera]
id=orbecc
type=usb-orbecc
name=Orbecc 3D Camera
resolution=640x480
...

Note: Multiple cameras can be defined in the .ini as [camera:0], [camera:1] sections after the base [camera] section, though multicamera support is still experimental.

A [camera] definition section can split into:

  1. Base Definition
  2. Frame properties and Transforms
  3. Camera specific properties

Base Definition

  • id: Short id for the camera.
  • name: Descriptive name for the camera (user facing).
  • type: Driver the camera uses.
    • opencv-nt uses OpenCV2 VideoCaptureOpenCV.
    • pyav uses PyAV (FFmpeg bindings) — recommended for RTSP/HTTP network cameras when OpenCV decoding shows undecoded macroblocks. See PyAV Driver.
    • picamera uses VideoCaptureRaspberry.
    • usb-realsense for Intel Realsense, uses VideoCaptureRealSense, requires pyrealsense2.
    • usb-orbbec for Orbbec Astra, uses VideoCaptureOrbbec, requires OpenNI2.
    • usb-k4a for Kinect 4, requires the Kinekt SDK (experimental).
  • calibration: Optional pointer to a .json file containing camera calibration parameters.

Capture Properties

  • resolution: In the form Width x Height, with no spaces: e.g. 640x480.
  • If the device supports capturing in this resolution, it will be set so. If not, frames will be rescaled to match.
  • fps: Frames per second, a decimal number, e.g. 15 or 24.5.
  • For higher framerates see fourcc setting in OpenCV Driver
  • rate_limit_fps: Rate limiting by discarding captured frames, useful if the device doesn't support a lower framerate.

Frame Transforms

  • rotation: One of none, 90, 180 or -90 degrees.
  • flip: One of none, horizontal, vertical or both axis.
  • clipping: A comma-separated coordinate values list of x,y,w,h defining a clipping frame (resulting frames will be w x h sized).

OpenCV/OpenCV-nt Driver

  • backend: String of the form CAP_XXXXX.
    • Default is CAP_ANY (0).
    • Others: CAP_FFMPEG, CAP_GSTREAMER, CAP_OPENNI2...
  • index: For webcams, an integer index for the camera.
  • fourcc: If the camera can offer video over MJPEG with a higher framerate, you can set to MJPG to select this framerate.
  • url: For network cameras, video files, etc:
    • http://... or rtsp://... URLs
    • A file path (e.g. /home/user/movie.mp4)
  • buffer: Number of frames for OpenCV to buffer (might cause lag).
  • rtsp_transport: For rtsp:// URLs, one of none, tcp, udp, http. Default tcp. TCP avoids the UDP packet loss that causes undecoded blocks at 1080p25; set to none to leave FFmpeg's default.
  • rtsp_buffer_size: FFmpeg socket buffer size in bytes (0 = leave default).
  • ffmpeg_threads: Force FFmpeg decoder thread count (0 = auto).
  • disable_hwaccel: Set to true to disable OpenCV's hardware-accelerated decode (D3D11VA/DXVA2/NVDEC on Windows, VAAPI on Linux). Default false.
  • use_wallclock_timestamps: If true, ignore the camera's RTP timestamps and use the local wallclock instead. Use when a camera sends bad timestamps that cause playback glitches. Default false.
  • fflags: Pass-through for FFmpeg's fflags option, e.g. +genpts+discardcorrupt+igndts.
  • allow_audio: If false (default), only the RTSP video stream is negotiated; audio is skipped entirely. Reduces UDP socket pressure on bursty I-frames. Set to true if you ever need the audio stream available.

PyAV Driver

Used when type=pyav. Decodes via FFmpeg bindings directly, exposing the knobs OpenCV's VideoCapture hides. Prefer this driver for RTSP cameras on Windows when the OpenCV pipeline produces glitched/undecoded regions, or when you need explicit hardware-accelerated decode control.

  • url: Same as the OpenCV driver — rtsp://, http://, or a file path.
  • rtsp_transport: One of none, tcp, udp, http. Default tcp. Applied only to rtsp:// URLs.
  • hwaccel: Hardware decoder type. Default auto.
    • auto → d3d11va on Windows, vaapi on Linux, videotoolbox on macOS.
    • Explicit values: d3d11va, dxva2, cuda, qsv, videotoolbox, vaapi, vdpau.
    • none for pure software decode.
  • frame_time: Same semantics as the OpenCV driver: send timestamps derived from frame number / fps instead of wall-clock capture time.

The following are advanced tuning parameters, rarely needed:

  • stimeout_us: RTSP socket timeout in microseconds. Default 5000000 (5 s). 0 leaves FFmpeg's default.
  • buffer_size: FFmpeg socket buffer size in bytes (0 = leave default).
  • max_delay_us: FFmpeg max_delay in microseconds — the RTP reorder window. Default 500000 (500 ms), matches the OpenCV driver's validated tuning. 0 reverts to FFmpeg's built-in default (~700 ms).
  • thread_count: Decoder thread count (0 = FFmpeg auto).
  • thread_type: auto, slice, or frame. frame maximises throughput; slice minimises latency.
  • low_delay: Set AV_CODEC_FLAG_LOW_DELAY on the decoder. Lower latency, may hurt quality with B-frames.
  • hwaccel_software_fallback: If HW decode init fails, fall back to software decode. Default true.
  • use_wallclock_timestamps: If true, ignore the camera's RTP timestamps and use the local wallclock instead. Use when a camera sends bad timestamps that cause playback glitches. Default false.
  • fflags: Pass-through for FFmpeg's fflags option, e.g. +genpts+discardcorrupt+igndts.
  • allow_audio: If false (default), only the RTSP video stream is negotiated; audio is skipped entirely. Reduces UDP socket pressure on bursty I-frames. Set to true if you ever need the audio stream available.
  • output_format: bgr24 (default) or rgb24.

Example for a 1080p25 RTSP camera on Windows:

[camera]
id=hikvision-lobby
type=pyav
url=rtsp://user:pass@192.168.1.20:554/Streaming/Channels/101
rtsp_transport=tcp
hwaccel=d3d11va
resolution=1920x1080
fps=25

Advanced Settings values

Values starting with underscore _ are advanced settings that are not commonly edited by hand.

Other settings are: - reported_screen_width_mm, reported_screen_height_mm: Default values for advanced settings to set the screen size. - reported_max_resolution_width_px, reported_max_resolution_height_px: Maximum resolutions for the camera.

Calibration Files

Calibration json files contain intrinsic and extrinsic calibration data for the given camera, as used by opencv and related toolkits.

{
  "sensor_size_mm": [ 3.6, 2.7 ],
  "intrinsic": {
    "width": 640,
    "height": 480,
    "matrix": [
      [ 779.5962524414062, 0.0, 511.0530090332031, 0.0 ],
      [ 0.0, 1008.8184204101562, 382.3992614746094, 0.0 ],
      [ 0.0, 0.0, 1.0, 0.0 ],
      [ 0.0, 0.0, 0.0, 1.0 ]
    ]
  },
  "extrinsic": {
    "matrix": [
      [ 0.9928808212280273, 0.09882735460996628, -0.06648891419172287, -0.19881358742713928 ],
      [ -0.06223934516310692, 0.9063953757286072, 0.41782012581825256, -0.9262909889221191 ],
      [ 0.10155729949474335, -0.41070738434791565, 0.9060935974121094, 36.0 ],
      [ 0.0, 0.0, 0.0, 1.0 ]
    ]
  }
}