Skip to content

video_filter: add sam2 feature

Brandon Lirequested to merge
PulseBeat_02/vlc:sam2 into master

This video filter adds SAM2 into VLC, allowing users to segment and crop out specific objects within a picture. This is very useful for extracting certain objects out of videos (such as on iPhone when you can hold and extract an object and save it as a sticker).

Notes:

  • Uses C++ since there aren't any SAM2 libraries for C.
  • The module inlines sam-cpp-macos, which is an open-source library licensed under MIT that removes the Python dependency in SAM2 and works directly with OpenCV and onnxruntime.
  • When the video is paused, the user can:
    • left-click to add a positive marker
    • right-click to add a negative marker
    • middle-click to remove an object.
  • For each click, an image mask of both the masked object and just the object itself is produced and exported to a directory if specified.
  • Download the SAM 2.1 tiny onnx model here if you want to test (from hugging face).

Filter Options:

  • sam-pre-model: SAM preprocessing path
  • sam-model: SAM model
  • sam-device: Device to make inferences on (cuda, cpu, etc)
  • sam-outline-thickness: How thick to make segmented object border
  • sam-outline-color: Outline color for segmented object border
  • sam-export-path: The folder to export both the visualization and mask images

SAM2 also has a tracking feature that can propagate the mask automatically and track a segmented object, but that's not added within the library. Perhaps for another module.

Merge request reports

VideoLAN code repository instance