Order from Chaos with wordlist_sort in C++26

Working with wordlists over the years usually means dealing with raw, unformatted text files. When aggregating lists from password audits, domain recon, or data dumps, the input files often contain duplicates, HTML tags, irregular whitespace, and hashes.

The traditional approach relies on standard shell pipelines:

cat list1.txt list2.txt | tr '[:upper:]' '[:lower:]' | grep -v '^[0-9]' | sort -u > clean.txt

This works for smaller inputs. With multi-gigabyte files and tens of millions of entries, shell pipelines spend significant time on process boundaries, intermediate copies, and disk I/O.

On July 18, 2024, I started wordlist_sort to handle transformations, filtering, and deduplication in a single tool directly in memory.

What it does

The program takes an output destination and one or more input files, applies the selected transformations, and writes the sorted result.

For example, to convert entries to lowercase, remove leading whitespace, filter words longer than 16 characters, and deduplicate the result:

./build/word_sorter --maxlen 16 --lower --detab --deduplicate output.txt list1.txt list2.txt list3.txt

Available filter options include:

• Hex hash stripping with --hash-remove
  Removes hexadecimal strings with 32 or more characters.

• HTML stripping with --dewebify
  Removes HTML tags from scraped web text.

• Email splitting with --email-sort
  Splits email addresses into user and domain tokens.

• Character frequency filtering with --dup-sense
  Filters words where a single character exceeds a defined percentage threshold.

C++26 implementation details

The project serves as an implementation test for modern C++26 features without external dependencies:

• Standard I/O with std::print
  Formatted terminal output directly via the standard library without stream overhead.

• Error handling with std::expected
  Explicit return types for error propagation instead of exceptions.

• Memory views with std::span and std::ranges
  Direct in-place transformations over non-owning contiguous memory buffers.

• Parallel file processing
  Input files are read into memory in bulk and processed concurrently via std::async tasks before merging into the main container using move operations.

Challenges during development

Maintaining a clean build with the C++26 standard involved several specific adjustments:

1. Compiler warnings on new standard features
   The build enforces -Wall -Wextra -Wpedantic -Wconversion -Wshadow -Werror. Combining strict conversion checks with new standard library algorithms required explicit type casts and careful index handling for buffer offsets.

2. Error handling across threads
   Replacing exceptions with std::expected and std::optional required refactoring worker task return paths so that errors propagate as structured values.

3. Replacing CLI11 with a zero-dependency C++26 parser
   Early versions used CLI11 for command-line parsing. While functional, it introduced external package management overhead, relied on exceptions for control flow, and performed heap allocations for storing string options. Modern C++26 provides the tools to write a compact parser without third-party code: std::span and std::string_view enable zero-copy argument inspection, std::from_chars parses integers without throwing exceptions, and std::expected handles input validation explicitly. Replacing the library reduced compile times, cut binary size, and left CMake with zero external dependencies.

On completion, the binary outputs word counts and elapsed runtime in milliseconds.

Project repository

The source code is licensed under the MIT license and available on GitHub as wordlist_sort.