I Built an AI Tool That Removes Silences, Retakes, and Filler From Videos Automatically

Rough cut editing is the most tedious part of making videos. You scrub through hours of footage, find every silence, cut every “um,” delete every stumbled take — and by the time you’re done, you’ve spent more time on the rough cut than on the actual creative edit. I got tired of it. So I built a tool that does it for me.

The Problem: Rough Cut Editing Eats Your Time

If you shoot talking-head videos, tutorials, or any kind of creator content, you know the drill. You record yourself explaining something, stumble over a sentence, re-record it, pause to think, say “um” twelve times, and then hand that raw mess to your editing timeline. Before any real creative decisions get made, you have to dig through all of that and cut the dead weight.

For a 10-minute final video, a rough cut can easily take 45 minutes to an hour. That’s not editing — that’s cleaning. And it’s the kind of repetitive mechanical work that should be automated.

Watch the Full Story on YouTube

Before we get into the technical breakdown, here’s the video where I walk through the entire build journey — from a simple browser tool to a full AI-powered Mac app.

The video covers every stage of how this tool evolved, the AI stack powering it, and what the editing workflow looks like now. If you’re a creator or someone interested in building tools with free AI, it’s worth watching the full thing.

How It Started: A Simple Browser-Based Video Converter

The Rough-Cut Cleaner didn’t start as an AI editing tool. It started much smaller — as a browser-based utility that converted video file formats. Nothing fancy. Just a practical tool I built because I needed it, using freely available web APIs and a bit of JavaScript.

But building that first tool taught me something important: you can solve real production problems with surprisingly simple code, especially when you’re building for your own exact workflow. That realization opened the door to the next idea.

Stage 2: Adding Silence Removal

The next obvious problem was silences. Every raw recording has them — the pauses between sentences, the moments where you stop to think, the gap before a re-take. I added a silence detection layer to the tool: analyze the audio waveform, find segments below a volume threshold, mark them for removal.

This alone made a huge difference. Even without any transcription or AI understanding, just removing the dead air trimmed meaningful time off the rough cut. But it was still blunt — it couldn’t tell the difference between a thoughtful pause and a useless gap, and it had no idea what was being said.

Stage 3: Whisper — Local AI That Understands What You’re Saying

This is where things got genuinely interesting. I integrated OpenAI’s Whisper — a locally-installed, open-source speech recognition model that runs entirely on your machine. No API costs, no cloud dependency, no data leaving your computer.

Whisper transcribes your footage with high accuracy, and that transcription unlocks a completely different level of editing intelligence. Now the tool can see your re-takes. When you stumble over a sentence and repeat it, that repetition shows up in the transcript as near-duplicate segments. The tool identifies them and marks the earlier, flawed versions for removal.

It can also flag filler words — “um,” “uh,” “you know,” “like” — and identify the exact timestamps where they appear. You review the suggestions, approve or skip, and the cut list builds itself.

Stage 4: Gemini — A Second Layer of AI Intelligence

Whisper tells you what was said. But understanding the context of what should be cut — identifying where a thought was abandoned and restarted, where an explanation went in the wrong direction and was corrected, where an entire segment is a duplicate of something said earlier in the video — that requires something smarter.

I added Google’s Gemini (free tier) as a second AI layer. Gemini reads the full transcript from Whisper and reasons about it — finding patterns that pure speech recognition can’t catch. It surfaces additional cut suggestions based on meaning, not just audio waveforms or keyword matches. A sentence that’s technically well-spoken but is clearly a false start before a better version gets flagged. Redundant explanations get flagged.

This is the “council of models” approach in action: one AI handles perception (transcription), another handles reasoning (editorial suggestions). Together they cover ground that neither could cover alone.

What Rough-Cut Cleaner Does Right Now

The app is a native Mac application. Here’s what it handles in a single session:

• Silence removal — Detects and marks audio gaps below your chosen threshold. Three tightness levels: Loose, Normal, Very Tight.
• Retake detection — Whisper transcription identifies where you re-recorded a line and marks the earlier attempt for removal.
• Filler word removal — “Um,” “uh,” and other vocal filler gets flagged with precise timestamps.
• Duplicate take detection — Identifies segments where you repeated yourself — intentionally or not.
• AI editorial suggestions — Gemini reads the full transcript and surfaces contextual cuts the other layers miss.
• Text-based editor — Review the transcript, select segments to cut or keep, approve or reject individual suggestions.
• FCPXML export — The approved cut list exports as a Final Cut Pro XML file you import directly into your existing timeline.

The tool processes multiple clips in one session. In the current build, it handled 6 minutes and 40 seconds of raw footage across two clips and surfaced 121 suggestions ready for review.

It Doesn’t Replace My Workflow — It Improves It

This is the part I want to be clear about. Rough-Cut Cleaner doesn’t ask you to abandon your editing software or learn a new workflow. The entire output is a single FCPXML file.

My workflow is unchanged: I still edit in Final Cut Pro. I still make all the creative decisions there — color, music, graphics, pacing. The only thing that changed is that I no longer spend an hour on the mechanical part before any of that starts. The rough cut comes in pre-cleaned, and I start from a timeline that’s already had the dead weight removed.

Import the FCPXML, review what the AI suggested, keep what’s right, undo what isn’t, and move on. The creative work starts immediately.

The Result: From 1 Hour to 10 Minutes

That’s the number that surprised me even after building the tool. A rough cut that used to take around an hour — scrubbing through footage, finding every silence, cutting every stumble, cleaning every re-take — now takes about 10 minutes with Rough-Cut Cleaner. Ten minutes to review the AI suggestions, approve the cuts, and export the XML.

For someone publishing multiple videos a week, that’s hours back every single week. Time that goes into actual creative work, scripting the next video, or just not burning out on the editing process.

How to Build Your Own Tools With Free AI

The broader point of this project — and the one I keep coming back to — is that the tools to build serious AI-powered software are free and available right now. Everything that powers Rough-Cut Cleaner is either open-source or available on a free tier.

Start With a Real Problem You Have

Every stage of this tool’s development started with a specific pain point in my own workflow. Format conversion, then silence removal, then retake detection. The problem defined the solution, not the other way around. If you start by asking “what can I build with AI?”, you’ll go in circles. Start with “what is the most annoying thing I do repeatedly?”

Use Claude or ChatGPT for the Code

Neither Whisper integration, audio analysis, nor the Gemini API calls required me to be a software engineer. I used Claude AI and ChatGPT (both free tiers) to write the code for each component. I described what I needed in plain English, reviewed the output, tested it, and iterated. The AI wrote the code; I made the product decisions.

Stack AI Models for Different Tasks

No single model is best at everything. Whisper is exceptional at transcription but doesn’t reason about content. Gemini reasons about content but doesn’t do audio analysis. Stacking them — using each for what it’s actually good at — is what makes the tool genuinely useful rather than a toy demo.

Build Incrementally, Ship Early

The browser-based format converter was a complete, useful tool on its own. So was the silence remover. Each stage was functional before the next one was added. This approach keeps you from getting lost in an endless build before you have anything working — and it means you’re using and testing the tool at every stage.

Side note on workflow: when you’re moving between an AI tool and your video editor, you end up copying and pasting constantly — prompts, scripts, captions, timestamps. I built Clipboard Empire for this exact problem on Mac. It keeps your full clipboard history locally on your device, never syncs anywhere, and has LLM support built in. One-time purchase, no subscription.

Common Questions and Problems

Whisper transcription failing — “Python whisper module is not installed”

This is the most common setup issue. Whisper runs locally and requires Python and the Whisper package installed on your machine. To fix it: install Python 3 if you haven’t already, then run pip install openai-whisper in your Terminal. Once installed, click “Retry transcription” in the app. If you’re not comfortable with Terminal setup, the app also offers a “Continue silence-only cleanup” option which works without Whisper — you get silence removal but not retake or filler detection.

Can I use this without Final Cut Pro?

The current export format is FCPXML, which is Final Cut Pro’s native XML format. FCPXML support is also built into DaVinci Resolve and some other professional editors. Adobe Premiere does not natively support FCPXML, though there are converter tools. Future export formats are on the roadmap.

How accurate is the AI at finding cuts?

The silence detection is highly reliable — it’s based on audio levels, which are objective. The retake and filler detection via Whisper depends on transcription accuracy, which is very good for clear speech in English and other well-supported languages. The Gemini suggestions are more interpretive — some will be exactly right, some you’ll want to skip. The workflow is designed as review-and-approve, not auto-apply, so you stay in control of every cut.

Where to Follow the Build

Frequently Asked Questions

Is Rough-Cut Cleaner free?

Pricing is TBD — it’s still in active development. The AI components it uses (Whisper, Gemini) are free, so the intention is to keep costs minimal. Follow the project at techtippr.com/get-alerts to get notified when pricing is announced.

Does it work on Windows?

The current build is Mac-only. A Windows version is a future possibility but not in scope for the initial release. The app uses Mac-native frameworks for the current version.

What video formats does it support?

MOV and MP4 are fully supported. The tool was built and tested with 1920×1080 footage. Support for other formats and resolutions is part of ongoing development.

Does the video stay on my computer?

Yes. Whisper runs fully locally — your footage never leaves your machine for transcription. Gemini does send the text transcript (not the video) to Google’s API for editorial analysis. No video data is transmitted externally.

How do I get early access?

Head to techtippr.com/get-alerts to sign up for launch notifications via email, Telegram, and social. Early access details will be shared there first.

Get Launch Alerts ↗
Watch the Build Video

Conclusion

The Rough-Cut Cleaner is still being built — but it already does what it set out to do. It takes the most mechanical, time-consuming part of video production and hands it to AI. Not to replace the creative work, but to get out of the way of it faster.

The bigger lesson, though, is the build process itself. Every component of this app was built using free tools — free AI for the code, free AI for the transcription, free AI for the editorial reasoning. The gap between “I have a problem” and “I have a working tool” has never been smaller. You don’t need a team or a budget. You need a clear problem and the patience to build toward it one stage at a time.

Follow the build, try the tool when it launches, and if you’re building something similar yourself — drop a comment. This is an interesting space and I want to know what other people are working on.

Recommended for you

Prompt Manager

Never lose your best prompts

Our Product

Learn More