Finding the dull parts of a screen recording

A long screen recording has plenty of stretches where nothing happens on screen. Reelify is an experiment in cutting them out automatically, on my own machine. It scores each sampled frame by how much it differs from the one before, cuts the stretches where nothing changes, and can speed up what’s left to fit a target length. Most of its history is from one day, 25 April 2026, so there isn’t much code, and most of the interesting decisions are in a few small files.

Scoring a frame

src/reelify/analyser.py asks FFmpeg for small greyscale frames rather than decoding them in Python:

cmd = [
    "ffmpeg", "-i", str(video_path),
    "-vf", f"fps={sample_fps},scale={_THUMB_WIDTH}:{thumb_height}",
    "-f", "rawvideo", "-pix_fmt", "gray",
    "-",
]

sample_fps is a quarter of the recording’s frame rate, and _THUMB_WIDTH is 320, so a 1080p recording at 30 frames a second becomes 7.5 frames a second at 320 by 180. Each frame arrives on standard output as 57,600 bytes, one per pixel, with no header. Python reads exactly that many bytes at a time and compares each thumbnail with the one before:

gray = np.frombuffer(raw, dtype=np.uint8).reshape((thumb_height, _THUMB_WIDTH))
if prev_gray is None:
    scores.append(0.0)
else:
    mad = np.mean(np.abs(gray.astype(np.float32) - prev_gray.astype(np.float32)))
    scores.append(mad / 255.0)

The score is the mean absolute difference, scaled to between 0 and 1. Zero means the two thumbnails are identical, and 1 would mean every pixel went from black to white or back.

An earlier version read the video through OpenCV at full size, reading one frame in four and calling grab() to skip the rest. FFmpeg still decodes every frame, but now it drops three in four and shrinks the rest before Python sees them, so Python handles 57,600 bytes per sample instead of a full-size colour frame.

What the numbers look like

To see what a score means, I made a test clip: frames drawn with NumPy and encoded with FFmpeg, 30 seconds at 1920 by 1080 and 30 frames a second. It shows a white page with rows of dark blocks, like lines of text. Nothing moves for the first 10 seconds. Then a 16 by 24 pixel pointer moves across the page for 5 seconds, the page scrolls 6 pixels a frame for 5 seconds, and nothing moves for the last 10.

The pointer scored about 0.0002. The scroll scored about 0.43. The still parts scored 0, apart from two blips of about 0.001 near 8.33 and 25 seconds, the two points in the still parts where the encoder had started a new keyframe, every 250 frames.

The default threshold in reelify process is 0.02, which the project’s architecture notes say mirrors auto-editor’s motion threshold. At that level, a pointer moving on its own counts as idle and gets cut, and a scroll is more than twenty times over. The cli preset lowers the threshold to 0.005, which still leaves the pointer out. Going much lower starts catching the encoder: at 0.0001, each keyframe blip became a short active stretch of its own.

Padding the cuts

classify() in src/reelify/classifier.py turns scores into chunks. Each score stands for four frames of the original, so its label is copied to all four. Then every active frame is padded by 0.3 seconds on each side, 9 frames at 30 frames a second, so a cut doesn’t clip the start or end of something happening.

The earlier version marked a window around each active frame with a nested loop. The current one sweeps the frames twice, forwards and then backwards, carrying a countdown that restarts at every active frame:

streak = 0
for f in range(result.total_frames):
    if labels[f]:
        streak = margin_frames + 1
    if streak > 0:
        padded[f] = True
        streak -= 1

The backward pass is the same loop run from the end. Between them, every frame within the margin of an active frame is marked, and each frame is visited twice whatever the margin. A test classifies 54,000 frames, 30 minutes at 30 frames a second, and fails if that takes a second or more. Finally, runs of frames with the same label become chunks.

On the test clip, the active chunk ran from 14.77 to 20.43 seconds: the scroll with its padding. Everything else was cut, and the real process command, run with --no-dedup (more on that below), turned 30 seconds into 5.67.

Fitting a target length

build_speed_map() in src/reelify/speed_map.py gives idle chunks a speed of 0, which means they’re dropped, and active chunks a speed of 1. If the result is still longer than the target, every kept segment gets the same faster speed:

if total_output > max_duration_secs:
    total_active_secs = sum(
        segment_duration_secs(seg, result.fps) for seg in segments if seg.speed > 0.0
    )
    required_speed = total_active_secs / max_duration_secs
    scaled_speed = min(required_speed, MAX_SPEED)

MAX_SPEED is 8, so a target can be missed. One of the tests gives it 100 seconds of activity and a 5-second target, and expects 8 times speed, which comes out at 12.5 seconds. On the test clip, --max-duration 2 asks for 5.67 seconds to fit in 2, a speed of about 2.83. Without --max-duration, the command uses a target of 86,400 seconds, a whole day, so nothing is sped up and only the cuts apply. That’s how it’s meant to work: the speed only changes to fit a target, and there’s no separate speed for quiet stretches yet.

Cutting out the segments

The encoder cuts each kept segment out with FFmpeg, several at once, and joins the pieces with FFmpeg’s concat demuxer. The start and length of each segment go before the input:

half_frame = 0.5 / fps
start_time = max(0.0, start_frame / fps - half_frame)
duration = (end_frame - start_frame) / fps - half_frame

cmd = [
    "ffmpeg",
    "-y",
    "-ss", f"{start_time:.6f}",
    "-t", f"{duration:.6f}",
    "-i", str(input_path),
]

Before the input, -ss and -t are measured on the recording’s own timeline. FFmpeg decodes from the keyframe before the start and throws frames away until it gets there, and because every segment is re-encoded anyway, the cut lands on the exact frame. Both ends sit halfway between two frames, so a timestamp that rounds the wrong way can’t add a frame or lose one. Then the video filter retimes the segment:

vf = (
    f"setpts=(PTS-STARTPTS)/{speed},"
    f"fps={fps}:round=up,"
    "scale=trunc(iw/2)*2:trunc(ih/2)*2"
)

setpts restarts the segment’s clock at zero and divides it by the speed. fps keeps the recording’s frame rate, so at 3 times speed the segment keeps its first frame and every third one after it. The scale rounds an odd width or height down to an even number, which libx264 needs. Sound is retimed with a chain of atempo filters of no more than 2 times each, so 8 times becomes atempo=2.0,atempo=2.0,atempo=2.0000.

A change in April moved -ss after the input for an accurate seek, but there it’s an output option, measured after setpts has divided the timestamps, so sped-up segments came from the wrong part of the video or came out empty until I moved both options back. On the test clip with --no-dedup, --max-duration 2 now gives 2 seconds of video, roughly every third frame from 14.77 to 20.43 seconds. The encoder’s older speed test encodes a whole 3-second clip at 2 times and checks that the file is bigger than 1KB, which can’t show the mistake. The new tests encode a clip whose frames show their own number as eight black or white stripes, one per bit, and check exactly which frames come out: frames 20 to 39 at 2 times must give 20, 22, 24 and so on up to 38.

The dedup step

One step runs before all of this. By default, reelify process drops repeated frames first, and the rest of the pipeline works on the shorter video that leaves. It uses FFmpeg’s mpdecimate filter, which compares each frame with the last one it kept, in blocks of 8 by 8 pixels, and drops the frame if no block has changed by more than a threshold:

hi = round(64 * 255 * (1.0 - similarity))
lo = round(hi * 5 / 12)
decimate = f"mpdecimate=hi={hi}:lo={lo}"

hi is a sum of differences over the 64 pixels in a block, so with the default --dedup-similarity of 0.90, a block counts as changed when its pixels have moved more than a tenth of the way between black and white, on average. lo is a lower threshold that a frame can also cross with enough blocks at once. A moving pointer or a typed character clears hi easily, and the encoder’s noise in a still picture stays well under it. On the test clip, the step kept 302 of the 900 frames: the first, the 300 where the pointer moves or the page scrolls, and the first still frame after the scroll. The default command then turned 30 seconds into 5.3, a little less than with --no-dedup, because no still frames are left after the scroll to pad it with.

The first version of this step used FFmpeg’s scene score instead, which measures how likely a frame is to start a new scene rather than how much it changed, so it stayed near zero through the scroll, kept a single frame of my clip and stopped the run with “No active segments to encode”.

Using it

You need Python 3.10 or later and FFmpeg.

pip install -e .
reelify process recording.webm
reelify process recording.webm --max-duration 60

Each writes recording_summary.mp4 next to the input, and the second speeds it up, if it needs to, to fit in a minute, as long as eight times is fast enough. reelify analyse recording.webm writes the segments, scores and keyframe captions as JSON instead, with the captions from a vision model running locally in LM Studio. The code is on GitHub.