uniq Cheatsheet
Quick reference for filtering and counting duplicate lines with uniq in Linux
The `uniq` command filters or reports repeated adjacent lines from a file or standard input. This cheatsheet covers counting duplicates, showing only unique or repeated lines, case-insensitive matching, and the common `sort | uniq` pipelines.
Basic Syntax
Core uniq command forms.
| Command | Description |
|---|---|
uniq file.txt | Remove adjacent duplicate lines |
uniq input.txt output.txt | Write filtered result to a file |
sort file.txt | uniq | Sort first so all duplicates are adjacent |
cat file.txt | uniq | Read from standard input |
uniq -z file.txt | Use NUL as the line delimiter |
Count and Filter
Count occurrences or isolate duplicate and unique lines.
| Command | Description |
|---|---|
uniq -c file.txt | Prefix each line with its occurrence count |
uniq -d file.txt | Print only duplicated lines, one per group |
uniq -D file.txt | Print every line of each duplicate group |
uniq -u file.txt | Print lines without an adjacent duplicate |
sort file.txt | uniq -c | sort -rn | Frequency count, most common first |
Comparison Control
Change which part of each line is compared.
| Command | Description |
|---|---|
uniq -i file.txt | Ignore case when comparing |
uniq -f1 file.txt | Skip the first field before comparing |
uniq -s5 file.txt | Skip the first 5 characters |
uniq -w10 file.txt | Compare only the first 10 characters |
uniq --group file.txt | Show all lines, groups split by a blank line |
Common Pipelines
Practical combinations with other commands.
| Command | Description |
|---|---|
sort access.log | uniq -c | sort -rn | head | Most frequent log lines |
cut -d, -f1 data.csv | sort | uniq | Unique values from a CSV column |
awk '{print $1}' access.log | sort | uniq -c | Count requests per IP address |
sort emails.txt | uniq -d | Addresses that appear more than once |
sort words.txt | uniq -u | Words that appear exactly once |
Troubleshooting
Quick checks for common uniq issues.
| Issue | Check |
|---|---|
| Duplicates not removed | Lines must be adjacent; pipe through sort first |
| Case treated as different | Add -i to ignore case |
| Trailing spaces split groups | Normalize whitespace first (e.g. sed 's/ *$//') |
| Fields skipped incorrectly | -f skips whitespace-separated fields, not delimiter columns |
| Count column hard to rank | Pipe uniq -c into sort -rn to order by frequency |
Related Guides
Use these guides for full duplicate-handling and text-processing workflows.
| Guide | Description |
|---|---|
| uniq Command in Linux | Full uniq guide with examples |
| sort Command in Linux | Sort lines so duplicates become adjacent |
| sort Cheatsheet | Quick reference for sorting options |
| wc Command in Linux | Count lines, words, and bytes |
| cut Command in Linux | Extract fields from text |
| awk Command in Linux | Field-based text processing |