Architecture Deep Dive: How FFmpeg Powers FrameFuseVid

FrameFuseVid combines three technologies to deliver a seamless desktop video-merging experience: Electron for the native shell, React for the UI, and FFmpeg for video encoding. In this post we walk through the architecture, explain the IPC bridge that keeps the renderer sandboxed, and show how FFmpeg filter graphs are generated for each layout.

The Three-Layer Stack

Each layer has a clear responsibility and minimal coupling to the others:

• Renderer — React renders the 4-step UI (file selection, layout preview, processing, completion). It has no access to Node.js APIs or the filesystem.
• Preload Bridge — preload.js uses Electron's contextBridge to expose a whitelist of IPC channels as window.electronAPI. This is the only communication path between the UI and the backend.
• Main Process — handles all privileged operations: file dialogs, folder scanning, FFprobe metadata extraction, and spawning FFmpeg processes.

The IPC Bridge

Electron's security model strongly recommends context isolation and no direct Node.js access in the renderer. FrameFuseVid follows this pattern:

const { contextBridge, ipcRenderer } = require('electron');

contextBridge.exposeInMainWorld('electronAPI', {
  // Dialog operations
  openFiles:     () => ipcRenderer.invoke('dialog:openFiles'),
  openFolder:    () => ipcRenderer.invoke('dialog:openFolder'),
  saveFile:      () => ipcRenderer.invoke('dialog:saveFile'),

  // FFmpeg operations
  combine:       (opts) => ipcRenderer.invoke('ffmpeg:combine', opts),
  cancelProcess: () => ipcRenderer.invoke('ffmpeg:cancel'),

  // Event listeners (one-way from main to renderer)
  onFFmpegProgress: (cb) =>
    ipcRenderer.on('ffmpeg:progress', (_e, data) => cb(data)),

  // ... more channels
});

The renderer calls window.electronAPI.combine(options), which invokes ipcRenderer.invoke('ffmpeg:combine', options). The main process handles this with ipcMain.handle('ffmpeg:combine', ...) and spawns FFmpeg.

How FFmpeg Filter Graphs Work

FFmpeg's -filter_complex flag accepts a text-based graph description that chains video/audio filters together. FrameFuseVid generates these dynamically based on the user's layout choice.

Picture-in-Picture

The PIP layout overlays one video on another. The user's drag position and size slider are converted to pixel coordinates:

The scale factor comes from the overlay size slider (10%–50%). The x and y values are computed from the user's drag position as percentages of the main video dimensions, then converted to absolute pixels.

Side-by-Side

Both inputs are scaled to half the output width, aspect-ratio preserved, and padded to fill their half:

# Scale left input to 960px wide, pad to fill 960x1080
[0:v]scale=960:1080:force_original_aspect_ratio=decrease,
  pad=960:1080:(ow-iw)/2:(oh-ih)/2[left];

# Same for the right input
[1:v]scale=960:1080:force_original_aspect_ratio=decrease,
  pad=960:1080:(ow-iw)/2:(oh-ih)/2[right];

# Stack them horizontally
[left][right]hstack[out]

Sequential (Concatenation)

Videos are concatenated using FFmpeg's concat demuxer. If subtitles are enabled, the subtitle filter is applied before output:

# Burn VTT captions into the video
-vf subtitles='captions.vtt'

Audio Merge

The simplest layout — maps the video stream from one input and the audio stream from another:

ffmpeg -i video.mp4 -i audio.m4a \
  -map 0:v -map 1:a \
  -c:v copy -c:a aac -b:a 192k \
  output.mp4

The video stream is copied without re-encoding (-c:v copy), so this operation is fast.

Quality Presets Under the Hood

The three presets map to FFmpeg's -preset and -crf flags:

CRF (Constant Rate Factor) controls quality on a logarithmic scale where lower numbers mean higher quality. The difference between CRF 23 and 20 is visually noticeable on detailed content like slides and code demos.

Progress Tracking

FFmpeg reports progress to stderr. The fluent-ffmpeg library parses this and emits progress events with percentage, timemark, and FPS. The main process forwards these to the renderer via IPC:

File Detection Strategy

When a user scans a folder, the main process walks the directory and matches each filename against Zoom's conventions:

function detectFileType(filename) {
  const lower = filename.toLowerCase();

  if (/shared_screen|screenshare|screen_share/.test(lower))
    return 'screen';

  if (/speaker|active_speaker|_as_|_avo_/.test(lower))
    return 'speaker';

  if (/gallery|_gv_|_gvo_/.test(lower))
    return 'gallery';

  if (/audio_only|\.m4a$|\.mp3$/.test(lower))
    return 'audio';

  if (/\.vtt$|\.srt$/.test(lower))
    return 'transcript';

  return 'unknown';
}

Users can override the detected type via a dropdown in the file list. This keeps the auto-detection non-destructive while still handling edge cases.

Build and Distribution

The CI pipeline in .github/workflows/build.yml runs on a matrix of Ubuntu, Windows, and macOS runners. Each builds the app using electron-builder and uploads platform-specific artifacts:

• macOS — .dmg via electron-builder's dmg target
• Windows — .exe NSIS installer
• Linux — .AppImage

On version tag pushes (v*), a release job downloads all three artifacts and creates a GitHub Release with the binaries attached.

What's Next

The current architecture is intentionally simple — a single React component for the UI and a single main process file for the backend. As the feature set grows (batch processing, custom templates, CLI mode), we plan to modularize both layers while keeping the same security model intact.

Interested in the code? Browse the repository on GitHub or check the documentation for the full IPC API reference.