Semantic Media Search Engine

Terabytes of media, no way to find anything

The client has accumulated years of video and image content: product demonstrations, training recordings, marketing assets, field documentation. Terabytes of files organized by date and folder name.

When someone needs a specific shot (a particular product in use, a specific moment from a training video, a photo from a job site years ago), the search process is manual. Browse folders. Scan filenames. Open files one by one. Hope you remember roughly when it was created.

Manual tagging was considered and rejected. Too expensive. Too slow. And it would never cover the backlog.

Search that understands meaning, not just filenames

We built a private, internal search engine that indexes the media library and enables natural-language queries. Instead of searching for “IMG_2847.jpg,” users search for what they are looking for: “equipment demonstration outdoors” or “safety training with protective gear.”

For video content, the search returns results with timestamp precision. Not just “this video is relevant,” but “this moment at 2:47 in this video matches your query.” Users can jump directly to the matching segment.

How it works

Read-only indexing. The fundamental design principle: original media files are never modified, moved, or copied during indexing. They remain in their original location on a read-only mount. All organization is virtual. Tags, collections, and search results reference the originals without touching them.

Embedding pipeline. Each media file is processed to extract machine-understandable signals:

• Visual embeddings capture what appears in images and video frames
• Transcripts capture spoken content in video
• OCR captures text visible in frames
• Metadata captures dates, dimensions, duration, and file properties

These signals are stored in a vector database, enabling similarity search across the entire library.

Video segmentation. Videos are broken into segments, each with its own embedding. A search query matches against segments, not whole videos. This is what enables timestamp-precise results.

Hybrid search. Queries combine semantic similarity (meaning-based) with metadata filtering (date range, file type, folder, duration). A search for “field operations 2024” uses both the semantic understanding of “field operations” and the metadata filter of the year.

Built for safety

Every processing job is idempotent. If indexing is interrupted, it can resume without reprocessing already-indexed files. Failed jobs do not corrupt the index. The system is designed for a NAS environment where the media library is the most valuable asset and cannot be put at risk.

Performance targets

The system was designed against specific, measurable quality bars:

• Recall@20 of 0.80 or higher. The relevant result appears in the top 20 results at least 80% of the time
• Top-3 Timestamp Hit Rate of 0.70 or higher. The matching video moment is in the first 3 results 70% of the time
• Filter correctness of 0.99 or higher. Metadata filtering works reliably
• Search latency at p95 under 1.5 seconds. Fast enough for interactive use

Results

Natural-language search across the full library. Users describe what they are looking for in plain English instead of guessing filenames.

Video timestamp precision. Search results link directly to the matching moment in a video. No more scrubbing through hour-long recordings.

Zero risk to originals. Read-only mount means the media library is never modified by the search system.

Scalable architecture. Dockerized microservices on the client’s existing NAS infrastructure. No cloud dependency for the media itself.

Engagement model: built and deployed. We built the search engine and deployed it to the client’s NAS. The system runs entirely on their infrastructure. They own it, they operate it, and their media never leaves their network.