AI operations¶
Editing operations that run a model. They live in videopython.ai rather than
videopython.editing so the core editing layer keeps no AI dependency
(why), but they
are ordinary registry entries: put them in a segment's operations list like any other.
They register only after import videopython.ai.
FaceTrackingCrop¶
The face_crop transform. Reframes around the tracked face, with framing rules (headroom
/ thirds / center) and a bounded camera speed. It constructs a
FaceSmoothingTracker internally.
from videopython.ai import FaceTrackingCrop
from videopython.editing import VideoEdit, SegmentConfig
# Horizontal source to vertical, following the subject
edit = VideoEdit(segments=[SegmentConfig(source="input.mp4", start=0, end=5, operations=[
FaceTrackingCrop(target_aspect=(9, 16)),
])])
edit.run_to_file("vertical.mp4")
# Headroom framing with a bounded camera speed
FaceTrackingCrop(framing_rule="headroom", max_speed=0.1)
FaceTrackingCrop
¶
Bases: Operation
Crops video to follow detected faces.
Useful for creating vertical (9:16) content from horizontal (16:9) video by tracking the speaker's face and keeping it framed.
The crop window has a fixed size -- the largest target_aspect box
that fits the frame (also the output size, so no resampling happens) --
and its position follows the smoothed face track. On the streaming path
the detection pass runs at plan-compile time over a bounded decode of
exactly the frames the filter will see, and the track compiles to a
per-frame crop position command file (ffmpeg sendcmd): zero
per-frame Python at render time.
Source code in src/videopython/ai/transforms.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 | |
to_ffmpeg_filter
¶
Compile the face track to a per-frame crop position command file.
Runs the detection pass at plan-compile time over a bounded decode of
the segment (through the same decode-stage filter prefix the render
will use, so the detector sees identical frames), then emits one
sendcmd interval per frame driving a fixed-size crop. Returns
None when the input frames are not reproducible at compile time
(decode_filters is None -- the op sits behind per-frame Python
effects) or the source is unknown.
Source code in src/videopython/ai/transforms.py
ObjectDetectionOverlay¶
The object_detection_overlay effect. Detects objects with a D-FINE COCO model and
composites colour-coded boxes with class labels. The detector
(ObjectDetector) is constructed internally; the
drawing is done by the AI-free renderer.
from videopython.ai import ObjectDetectionOverlay
from videopython.editing import VideoEdit, SegmentConfig
# Defaults: per-class colours, confidence shown, detection every 2nd frame
edit = VideoEdit(segments=[SegmentConfig(source="street.mp4", start=0, end=5, operations=[
ObjectDetectionOverlay(),
])])
edit.run_to_file("annotated.mp4")
ObjectDetectionOverlay(class_filter=["person", "car"], detection_interval=1, model_size="s")
In a JSON plan (it is LLM-exposed):
{"op": "object_detection_overlay", "class_filter": ["person", "car", "dog"],
"confidence_threshold": 0.4, "detection_interval": 2,
"window": {"start": 0, "stop": 5}}
Cost¶
Memory stays bounded on long clips — it streams — but compute does not: a D-FINE forward pass runs per sampled frame. To cap it:
| Knob | Effect |
|---|---|
window |
Restricts the overlay, and therefore detection, to a time range |
detection_interval |
Detect every Nth frame, hold boxes in between (default 2). Higher is faster; fast motion shows more lag |
class_filter |
Fewer classes to draw |
model_size |
"n" (nano, default, fastest) → "s" → "m" (most accurate) |
ObjectDetectionOverlay
¶
Bases: Effect
Detect objects per frame and overlay labelled bounding boxes.
Runs a D-FINE COCO detector and composites tidy, colour-coded boxes with class labels (and optional confidence) onto every frame in the window.
Detection runs on a detection_interval cadence in the streaming path and
boxes are held between detections, so the cost is compute-bound, not
memory-bound: "streamable" here means bounded memory, not bounded
compute. On long clips, cap cost with window (limit the time range),
a larger detection_interval, a class_filter, and/or the smaller
model_size. Only streaming_init and process_frame are
overridden; the streaming engine drives that contract for bounded-memory
execution.
Source code in src/videopython/ai/effects.py
Renderer¶
Pure and AI-free, reusable with any list of
DetectedObject. Colours are deterministic per class,
so a class keeps its colour across frames and across runs.
from videopython.base import DetectionStyle, class_color, draw_detections
frame = draw_detections(frame, detections, DetectionStyle(show_confidence=False))
draw_detections
¶
draw_detections(
frame: ndarray,
detections: list[DetectedObject],
style: DetectionStyle = DetectionStyle(),
) -> np.ndarray
Return a copy of frame with detections drawn as labelled boxes.
Shape-preserving: the result is the same (H, W, 3) uint8 array. An
empty detections list (or one filtered out by min_confidence) is a
no-op that returns frame unchanged. Boxes are clamped to the frame, so
off-frame coordinates clip cleanly instead of raising. Label chips flip
inside the box when they would overflow the top edge and clamp horizontally
so they never leave the frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
ndarray
|
Source frame as |
required |
detections
|
list[DetectedObject]
|
Objects to draw; each uses its normalized |
required |
style
|
DetectionStyle
|
Visual styling (colours, stroke width, label options). |
DetectionStyle()
|
Returns:
| Type | Description |
|---|---|
ndarray
|
A new |
Source code in src/videopython/base/draw_detections.py
DetectionStyle
dataclass
¶
Styling for :func:draw_detections.
Lengths expressed as a fraction of the frame's longer side are resolution-independent: the same style reads consistently at 1080p and 4k.
Source code in src/videopython/base/draw_detections.py
box_color
class-attribute
instance-attribute
¶
Fixed (R, G, B) for every box, or None for per-class colours.
line_thickness
class-attribute
instance-attribute
¶
Box stroke width as a fraction of max(height, width) (~3px at 1080p).
show_confidence
class-attribute
instance-attribute
¶
Append the confidence as a whole-number percent to each label.
label_font_size
class-attribute
instance-attribute
¶
Label text height as a fraction of max(height, width) (~24px at 1080p).
label_text_color
class-attribute
instance-attribute
¶
Colour of the label text drawn on the chip.
label_bg_alpha
class-attribute
instance-attribute
¶
Opacity (0-255) of the label chip background.
min_confidence
class-attribute
instance-attribute
¶
Detections below this confidence are skipped.
font
class-attribute
instance-attribute
¶
Bundled font name or path; None uses the default font.
class_color
¶
Deterministic RGB colour for a class label.
Common COCO classes get a reserved Material hue; everything else maps
md5(label) -> HSV hue at fixed saturation/value. md5 (not the
salted built-in hash) is used so colours are stable across processes
and test runs.