Silence removal is the highest-leverage cleanup there is (on a long recording it can take a third of the runtime out) and it is also the easiest to overdo. The failure is distinctive: everyone has watched a video where the speech is technically continuous and the person sounds like they are being chased.
The difference between a tightened edit and a rushed one is not the threshold. It is whether the pauses that survived were chosen.
Silence is not automatically dead air
A gap in speech can be doing any number of jobs. It can be a breath. It can be the beat before a punchline, which stops being funny without it. It can be the space after a difficult statement where a viewer catches up, or the pause while someone decides how honest to be, often the most interesting moment in an interview.
None of that is visible in a waveform. A waveform shows you where the amplitude dropped; it cannot show you why. That is the whole reason a review step exists, and the reason a duration threshold on its own is never sufficient.
Start conservative and tighten
Choose a minimum gap longer than you think you need: a full second or more on conversational material. You will get fewer proposed cuts, and the ones you get will be unambiguous: the stretch where you were finding a file, the silence where someone left to get a glass of water.
Run that pass, watch the result, and only then decide whether to tighten. Working in this direction means every pass improves the edit. Working the other way (starting aggressive and restoring what you lost) means fighting the tool, and it is much harder to notice something missing than to notice something present.
Keep room around the words
This is the single setting that decides whether the result sounds natural. Speech does not begin at full volume; it ramps, and the first consonant of a word is often quieter than the vowel that follows. Cut exactly at the detected speech boundary and you clip the front of words: "probably" becomes "robably" in a way most people hear as wrong without being able to say why.
Leave a margin before speech starts and after it ends. Um Out calls this padding, and it defaults to keeping a small amount rather than cutting flush, because flush is almost never what you want. Widen it on breathy or softly-spoken material.
Judge by purpose, not by number
When you review the proposed cuts, the question is not "is this gap longer than my threshold", because the tool already answered that. The question is "what was happening here". A two-second pause while someone thinks is content. A two-second pause while someone repositions a microphone is not. They are identical in a waveform and obvious on playback.
This is why reviewing with audio is worth the extra minutes. Scanning a list of timecodes and lengths, you will approve things you would have rejected if you had heard them.
Listen across the joins
A cut that looks clean can still sound wrong. Play across each join rather than stopping at it. Listen for breaths that got truncated halfway, for room tone that changes abruptly between takes, and for two sentences that now run together with no space at all, which reads as urgency whether or not you wanted it.
On camera, also watch. A cut that works on audio can produce a visible jump in posture or hand position. This is the reason to do silence removal before any visual polish: those jumps are much cheaper to hide when you have not yet built anything on top of them.
Know when to stop
There is a point where further tightening stops making the video better and starts making it relentless. It usually arrives sooner than the tooling suggests, because the tooling is optimising a measurable thing and you are optimising something else.
A useful check: watch two minutes of the tightened edit and notice whether you feel like you can breathe. If you do not, the viewer will not either.
Common mistakes
Cutting flush to the speech boundary
Clipped consonants. The most common cause of a tightened edit sounding cheap, and the easiest to fix: increase the padding.
Setting one threshold for mixed material
A scripted monologue and a relaxed conversation want different numbers. Recordings that contain both want two passes, not one average.
Reviewing without listening
Timecodes and durations cannot tell you whether a pause was doing something. Playback can, in about a second per candidate.
Removing every breath
Breath is part of how speech sounds like a person. Take out the loud ones, keep the rest.
Questions
What is a good minimum gap to start with?
Longer than feels necessary. On conversational material, starting around a second and tightening from there gives you unambiguous candidates first. The right final number depends on the speaker and the format, which is why it is a setting rather than a default we claim is correct.
Why does my edit sound rushed after silence removal?
Almost always padding set too tight, so cuts land flush against the speech. Widen the margin kept either side of each phrase and re-run. The second most common cause is removing the structural pauses (the ones between topics) along with the incidental ones.
Does it cut the video as well as the audio?
Yes. Cuts apply to linked picture and sound together, so they cannot drift out of sync. That is a deliberate constraint rather than a convenience.
Can I remove silences from just part of a sequence?
Yes. The analysis runs on the range and track you point it at, so you can clean one section without touching the rest.