Skip to content

CSV dedupe

Remove duplicate rows from a CSV/TSV file and download the cleaned document.

CSVdedupedataclean

Upload

Files (0–1)
One delimited file, up to 8 MB, or paste it below.
Paste a delimited document to deduplicate instead.

Options

Deduplicate on columns Comma-separated header names to match on. Without a header, use 1-based column numbers (1,3). Empty means exact-row matching. Matching is on VALUES; the output keeps first occurrences verbatim.
Questions people ask
What counts as a duplicate?
By default, a row whose EVERY field matches an earlier row exactly. Name columns in "deduplicate on columns" to match only those values (for example email, name), which is handy when a free-form notes column makes whole-row matching useless.
Which occurrence is kept, and in what order?
The first occurrence, always, and every kept row keeps its original position. The output is the input with the duplicates deleted, never reordered.
Do you normalise my values?
No, not unless you ask. Case-insensitive and whitespace-trimmed comparisons are opt-ins that affect only the matching; the downloaded file keeps each first occurrence's original values byte for byte (with standard RFC 4180 re-encoding).
What about case differences and stray spaces?
They count as different values until you enable the corresponding option. With case-insensitive comparison, "Ada", "ada" and "ADA" are one row (the exact "Ada" is kept). Unicode folds correctly, so accents and scripts compare by the same rule.
How are inconsistent row widths handled?
The run refuses them by default and names every affected line with the expected versus actual column counts, because a dedupe over half-known records would be a guess. "Permissive" keeps ragged rows (whole-row comparison; nothing is ever lost).
What about malformed CSV?
Unterminated quotes and similar breakage stop the run with a diagnostic naming the line where the malformed row begins; a NUL-byte or non-UTF-8 document is likewise refused with a named reason. Only complete, consistent documents are rewritten.
Can I get JSON instead of CSV?
Yes. The output-format option emits the surviving rows as JSON records keyed by your header, ready for the JSON tools. Values stay strings.
How big can the file be?
Up to 8 MB in, and the cleaned file is at most that (rows are only removed). It stays downloadable for about 10 minutes, then it is deleted automatically.