Sort lines and remove duplicate lines on a Mac.

Download Caxton for Free · 7-day free trial of Caxton Pro, no credit card required · 7 MB · macOS 13.0+

How do I sort the lines of a text file in Terminal?

sort is the tool, and it ships with macOS. It reads the file, orders the lines, and prints the result, so send it to a new file rather than overwriting the one you are reading:

sort names.txt > names-sorted.txt

By default the order is character by character, so uppercase letters sort before lowercase and "10" sorts before "9". Two options change that. -f folds case so that "apple" and "Apple" sort together, and -n sorts by numeric value, which Apple's manual page describes as "Sort fields numerically by arithmetic value." -r reverses the order.

sort -f names.txt
sort -n sizes.txt
sort -rn sizes.txt

sort is also the right tool for a file larger than memory. Apple's manual page says that when the input does not fit in its buffer, "the temporary disk files are used to perform the sorting," with -T choosing the directory for them, so a multi-gigabyte file sorts, slowly, without anything else to install.

How do I remove duplicate lines in Terminal?

There are three commands, and they differ in one thing: the order of the result.

sort -u sorts and keeps one copy of each line. The manual page calls it "Unique keys. Suppress all lines that have a key that is equal to an already processed one." The result is sorted, which is often fine and sometimes not.

sort -u hosts.txt > hosts-unique.txt

uniq removes repeated lines, but only when they sit next to each other. Its manual page is explicit: it works by "comparing adjacent lines," and "Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first." That is why uniq is nearly always written after sort:

sort hosts.txt | uniq

Two of its options answer other questions. uniq -c counts, printing each line with the number of times it appeared, and uniq -d prints only the lines that were repeated:

sort hosts.txt | uniq -c | sort -rn
sort hosts.txt | uniq -d

awk '!seen[$0]++' removes duplicates and keeps the original order. It prints a line the first time it sees it and never again, because the counter for that line is zero only once:

awk '!seen[$0]++' hosts.txt > hosts-unique.txt

Run on a file of b, a, b, c, a, it prints b, a, c. It holds every distinct line in memory, so it fits a file of any length whose distinct lines fit in memory. All three commands compare lines exactly: a trailing space or a different capitalization makes two lines different, and a sort with -f does not make the dedupe ignore case.

How do I sort lines and remove duplicates in Caxton?

Caxton, a native Mac editor for text and data files, puts the same operations in the Text menu, where they act on the selected lines or, with nothing selected, on the whole document. Each one lands as a single undo step, and the status bar reports what changed. Here is the walk through a file of hostnames.

1. Open the file and select the lines, or nothing

Drag the file onto Caxton or use File ▸ Open. With no selection a Text command works on every line; select a range to work on only those. Opening, reading and searching are Caxton Free.

2. Sort

Text ▸ Lines holds five sorts. Sort Lines orders A to Z, ignoring case, and lines that compare equal keep their order, so a sort followed by a dedupe gives the same result every time. Sort Lines Descending is Z to A. Sort Lines Numerically sorts by the number each line starts with, comparing values rather than digits, so 9 comes before 10, and lines that do not start with a number go last in their original order. Sort Lines by Length puts the shortest line first, and Reverse Lines puts the last line first. Changing the file is Caxton Pro, which every download includes as a 7-day free trial with no credit card required.

3. Delete the duplicates

Text ▸ Lines ▸ Delete Duplicate Lines keeps the first copy of each line and removes the later ones, so the file keeps its original order, the same result as the awk route above. Lines must match exactly, case included. The opposite question has its own command: Keep Only Duplicate Lines keeps every occurrence of the lines that appear more than once and drops the rest, which is the quick way to see what was repeated. Both are Caxton Pro.

4. Check, then undo or save

The status bar reports the count, and ⌘Z reverses the whole operation as one step. ⌘S writes the file back. To see which lines repeat without changing anything, use the free route first: Edit ▸ Find ▸ Filter Lines (⇧⌘L) with a hostname shows every line that contains it, with the count in the status bar.

5. The size limit, and what to do past it

Text menu commands work on up to 256 MB at a time. On a larger document the command says so and asks for a smaller selection, so select a region of the file and run it there, or sort the file once with sort in Terminal and open the result. A file that is really a table has a better route, described next.

Duplicate lines or duplicate rows?

A line is a duplicate when every character matches. A row in a CSV is usually a duplicate when a few columns match, such as the same email address with a different signup date, and the line-based tools cannot see that. For a table, Caxton opens the file in its grid and offers CSV ▸ Sort Rows… by one or more columns and CSV ▸ Delete Duplicate Rows… by a key you choose, both streamed, with the duplicate groups reported before anything is removed. Remove duplicate rows from a large CSV walks through it, and sort a large CSV covers the ordering.

Which approach fits which job?

Frequently asked questions

Sources

• Caxton manual: Editing, the Text menu's sorting and duplicates commands
• Caxton manual: Menus, the Text ▸ Lines submenu
• Caxton manual: the CSV workbench, Sort Rows and Delete Duplicate Rows
• Apple's manual pages on macOS: man sort, man uniq and man awk, read with the shipped tools in Terminal