How to remove duplicates from a list in Python
To remove duplicates and keep the original order, wrap the list in dict.fromkeys(): list(dict.fromkeys(items)). If order doesn't matter, set(items) is the fastest way.
A duplicate is any value that already appeared earlier in the list. The trick is picking a method that matches what you care about: keeping the order, raw speed, or handling unhashable items like dicts. Each example runs on this page — hit Run, then edit the code and run it again.
1dict.fromkeys() (keeps order)Recommended
dict.fromkeys() builds a dict using the list items as keys. Dict keys are unique and, since Python 3.7, keep insertion order — so wrapping it back in list() gives you the unique items in their original order, in one line.
Output
This is the go-to method when order matters: no imports, one line, and the first occurrence of each item wins.
2set() (fastest, order not kept)
Passing the list to set() drops duplicates instantly — it's the fastest option. The catch: a set has no order, so the result can come back in any arrangement. Here we sorted() it so the output is stable and predictable.
Output
Reach for set() when you only need the unique values and don't care about order. Printing the raw set (without sorted()) can even vary between runs, so sort it if you need a repeatable result.
3A loop (order + custom logic)
When you need order and custom rules — case-insensitive matching, deduping on one field — a loop with a seen set is the clearest tool: skip anything you've already recorded, keep the rest.
Output
More code than dict.fromkeys(), but the loop body is where you drop in your own "have I seen this?" rule.
4Which should you use?
| Method | Keeps order | Speed | Best for |
|---|---|---|---|
| dict.fromkeys() | Yes | Fast | Most cases |
| set() | No | Fastest | Order-agnostic |
| seen-set loop | Yes | Good | Custom rules |
5Common variation: dedupe a list of dicts
set() and dict.fromkeys() both require hashable items, so they raise TypeError on a list of dicts. Fall back to the loop — comparing with not in works on any type:
Output
Frequently asked questions
Does removing duplicates keep the original order?
It depends on the method. dict.fromkeys() and a seen-set loop both keep the first-seen order. set() does not — a set is unordered, so sort it if you need a predictable result.
Which method is the fastest?
Converting to a set() is the fastest because membership checks are O(1) and it does all the work in C. dict.fromkeys() is nearly as fast and keeps order, which is why it is usually the better default.
How do I remove duplicates from a list of dicts?
Dicts are unhashable, so set() and dict.fromkeys() fail. Use a loop that appends each item only if it is not in a running list — see the variation above. For large lists, dedupe on a hashable key (like a tuple of fields) with a seen set instead.
How do I keep only the items that appear exactly once?
That is different from deduping. Use collections.Counter(items) and keep the keys whose count is 1: [x for x, n in Counter(items).items() if n == 1].