uniq

What it does

uniq filters adjacent matching lines from input, either collapsing them down to one occurrence, counting them, or showing only the duplicated or only the unique ones. The critical detail is "adjacent" — uniq compares each line only to the one immediately before it, so it does not detect duplicates scattered throughout an unsorted file. That's why uniq is almost always used right after sort.

Beginner examples

• uniq file.txt — collapse adjacent duplicate lines (file must already be sorted for full effect)
• sort file.txt | uniq — the standard way to get a fully deduplicated list
• sort file.txt | uniq -c — count how many times each line occurs
• sort file.txt | uniq -d — show only lines that repeat

sort visitors.txt | uniq -c

Advanced examples

• Rank lines by frequency, most common first: sort access.log | uniq -c | sort -rn | head
• Find usernames that appear only once in a login log (rare/first-time users): sort logins.txt | uniq -u
• Deduplicate case-insensitively for messy user-entered data: sort emails.txt | uniq -i
• Compare lines ignoring the first field (e.g. a timestamp column): sort log.txt | uniq -f 1
• Count unique IP addresses hitting a server, sorted by frequency: awk '{print $1}' access.log | sort | uniq -c | sort -rn

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

Common mistakes

• Running uniq on an unsorted file and assuming all duplicates were removed — only adjacent duplicates are collapsed, so scattered repeats slip through untouched.
• Forgetting -c when you actually need frequency counts, then trying to reconstruct counts manually from plain uniq output.
• Confusing -d (show only duplicated lines) with -u (show only lines with no duplicates) — they're opposites, not variations of the same filter.
• Piping uniq before sort instead of after, e.g. uniq file.txt | sort, which sorts already-incorrect output instead of fixing the underlying adjacency problem.

Tips

• Always sort first: sort file | uniq is the standard idiom; uniq alone assumes pre-sorted input.
• Use uniq -c combined with a secondary sort -rn to build a quick frequency-ranked report from raw text.
• sort -u does an equivalent single-pass dedup without a count and without a separate uniq step, when you don't need counts.

Best practices

• Treat "sort then uniq" as one atomic operation in scripts and pipelines; never rely on uniq alone for deduplication unless you've explicitly guaranteed the input is sorted upstream.
• Use uniq -c for frequency analysis rather than writing a manual counting loop — it's faster and less error-prone.
• When comparing lines that share a variable prefix (like a timestamp), use -f/-s to skip the fields/characters that shouldn't factor into the comparison, instead of pre-processing with cut first.

Real-world use cases

• Finding the most frequent IP addresses hitting a web server from access logs: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head.
• Deduplicating a mailing list before a send: sort -u subscribers.txt > subscribers_clean.txt.
• Auditing a file of error codes to see which ones are truly one-off versus recurring: sort errors.txt | uniq -c | sort -n.

Common interview questions

• Why does uniq sometimes fail to remove duplicates? uniq only compares adjacent lines; if the input isn't sorted, duplicate lines that aren't next to each other won't be collapsed.
• How do you count occurrences of each unique line in a file? sort file | uniq -c — sort groups identical lines together, then uniq -c prefixes each with its count.
• What's the difference between uniq -d and uniq -u? -d prints only lines that have duplicates (appeared more than once); -u prints only lines that are truly unique (appeared exactly once).