How to remove duplicates from a slice in Go

1A map seen-set, in one passRecommended

This is the answer nine times out of ten. The map is the set: map[T]struct{} stores keys and nothing else, because the empty struct occupies zero bytes. Append each value the first time you meet it and you get first-seen order for free — the property Sort + Compact throws away. Since Go 1.18 you can write it once as a generic helper and use it for every comparable element type.

package main

import "fmt"

// Dedupe returns s with duplicates removed, keeping the FIRST occurrence of
// each value and the original order. It works for any comparable element type.
func Dedupe[T comparable](s []T) []T {
	seen := make(map[T]struct{}, len(s))
	out := make([]T, 0, len(s))
	for _, v := range s {
		if _, ok := seen[v]; ok {
			continue // already emitted
		}
		seen[v] = struct{}{}
		out = append(out, v)
	}
	return out
}

func main() {
	langs := []string{"go", "rust", "go", "zig", "rust", "go"}

	unique := Dedupe(langs)
	fmt.Println(unique)
	fmt.Println(len(langs), "->", len(unique))

	// The same function, a different element type
	fmt.Println(Dedupe([]int{4, 7, 4, 1, 7, 7, 9}))

	// The input is untouched -- Dedupe builds a new slice
	fmt.Println(langs)
}

Prints [go rust zig], 6 -> 3, [4 7 1 9], then the unchanged [go rust go zig rust go]. Both make calls are sized up front, which matters: without the hints the map rehashes and the slice regrows several times over a long input. map[T]bool works too and reads a touch shorter (if seen[v] { continue }), at one byte per entry — struct{} is the idiom precisely because a set has no values worth storing. The constraint is comparable, so element types that can't be compared with == — slices, maps, functions — won't compile; key on something derived instead, as in section 4.

2slices.Sort + slices.Compact

slices.Compact (Go 1.21+) is the closest thing the standard library has to a dedupe, and the trap is right in the doc comment: it removes consecutive duplicates only. Hand it unsorted data and it collapses runs and leaves everything else alone. Sort first and every equal pair becomes adjacent, so the pair does the whole job — no map, no second slice, at the cost of O(n log n) and the original order. See sort a slice for the sorting half.

package main

import (
	"fmt"
	"slices"
)

func main() {
	nums := []int{5, 2, 5, 9, 2, 2}

	// Compact only collapses ADJACENT duplicates
	asIs := slices.Compact(slices.Clone(nums))
	fmt.Println(asIs)

	// Sort first and every duplicate becomes adjacent
	sorted := slices.Clone(nums)
	slices.Sort(sorted)
	fmt.Println(sorted)
	fmt.Println(slices.Compact(sorted))

	// Clone kept the original safe
	fmt.Println(nums)
}

The unsorted call prints [5 2 5 9 2] — only the trailing 2 2 collapsed, so both 5s and two of the three 2s are still there. Sorted, it reads [2 2 2 5 5 9] and compacts to [2 5 9]. Two rules come with this form: Compact returns a shorter slice header and rewrites the backing array in place, so you must assign the result back, and the caller's slice is modified unless you work on slices.Clone(s) — which is why the last line still prints [5 2 5 9 2 2]. Since Go 1.22 Compact zeroes the slots it vacates, so a slice of pointers doesn't leak.

3Custom equality and case-insensitive dedupe

slices.CompactFunc(s, eq) is Compact with your own equality test, so strings.EqualFold gives you a case-insensitive dedupe in one call. It is still adjacent-only, which means the sort in front of it has to agree with it: sort by the same normalised key, or values that eq considers equal will never meet. SortStableFunc rather than SortFunc keeps the input order inside each group, so the spelling that survives is the first one you wrote.

package main

import (
	"fmt"
	"slices"
	"strings"
)

func main() {
	tags := []string{"Go", "go", "GO", "Rust", "rust"}

	// CompactFunc compares neighbours with your own equality test
	fmt.Println(slices.CompactFunc(slices.Clone(tags), strings.EqualFold))

	// Still adjacent-only: sort with the SAME notion of equality first.
	// SortStableFunc keeps the first-seen spelling of each group.
	mixed := []string{"Rust", "go", "GO", "rust", "Go"}
	slices.SortStableFunc(mixed, func(a, b string) int {
		return strings.Compare(strings.ToLower(a), strings.ToLower(b))
	})
	fmt.Println(mixed)
	fmt.Println(slices.CompactFunc(mixed, strings.EqualFold))
}

The already-grouped input compacts straight to [Go Rust]. The shuffled one sorts to [go GO Go Rust rust] and compacts to [go Rust] — note the survivors are go and Rust, the first member of each group in the stable order, not the first element of the original slice. If you want case-insensitive dedupe and the original order, skip CompactFunc and go back to section 1 with a normalised key: seen[strings.ToLower(v)] while you append the untouched v.

4Deduping structs by a key field

"Unique users" almost never means "unique whole rows" — it means unique ID. Key the seen-set on the field instead of the element and the rest of the loop is unchanged. Go will also let you key on the whole struct, because a struct is comparable when all of its fields are, which is the version to use when two rows differing in any field are genuinely different.

package main

import "fmt"

type User struct {
	ID    int
	Email string
}

// dedupeByID keeps the first user seen for each ID.
func dedupeByID(users []User) []User {
	seen := make(map[int]struct{}, len(users))
	out := make([]User, 0, len(users))
	for _, u := range users {
		if _, ok := seen[u.ID]; ok {
			continue
		}
		seen[u.ID] = struct{}{}
		out = append(out, u)
	}
	return out
}

func main() {
	users := []User{
		{1, "[email protected]"},
		{2, "[email protected]"},
		{1, "[email protected]"},
		{3, "[email protected]"},
		{2, "[email protected]"},
	}

	for _, u := range dedupeByID(users) {
		fmt.Println(u.ID, u.Email)
	}

	// A struct with only comparable fields can be a map key itself,
	// which dedupes on the WHOLE value instead of one field.
	whole := make(map[User]struct{}, len(users))
	for _, u := range users {
		whole[u] = struct{}{}
	}
	fmt.Println("distinct rows:", len(whole))
}

Prints 1 [email protected], 2 [email protected], 3 [email protected], then distinct rows: 4. The two numbers tell the story: keying on ID gives 3 users — the second Ann row was dropped even though its email differed, because first-seen wins — while keying on the whole value gives 4, since only the duplicated Bob row is identical in every field. Whole-struct keys stop compiling the moment a field is a slice, map, or function; for those, key on a derived string (or a hash) you build yourself. And if you'd rather not allocate a map at all, slices.SortFunc by ID followed by slices.CompactFunc(users, func(a, b User) bool { return a.ID == b.ID }) does the same job in O(n log n).

5In-place dedupe that reuses the backing array

keep := s[:0] is a slice of length 0 over the same array, so appending the survivors writes them over the front of the input as you go. It never allocates for the result — a real win in a hot path — and the write index can never overtake the read index, so the pass is safe. What you give up is the input: by the end, s holds the deduped prefix followed by leftovers. Same trick as the filter in remove an element from a slice.

package main

import "fmt"

func main() {
	ids := []int{4, 7, 4, 1, 7, 7, 9}

	seen := make(map[int]struct{}, len(ids))
	keep := ids[:0] // same backing array, length 0
	for _, id := range ids {
		if _, ok := seen[id]; ok {
			continue
		}
		seen[id] = struct{}{}
		keep = append(keep, id)
	}
	fmt.Println(keep, len(keep), cap(keep))

	// The input was overwritten in place -- and the tail still holds junk
	fmt.Println(ids[:cap(ids)])

	// Zero the leftovers so they cannot pin memory
	clear(ids[len(keep):cap(ids)])
	fmt.Println(ids[:cap(ids)])
}

Prints [4 7 1 9] 4 7 — four elements over the original capacity of seven, proof that nothing was allocated for the result. The full array then reads [4 7 1 9 7 7 9]: the survivors up front, and stale copies of already-emitted values in the tail. For int that is only untidy, but for a []*User or [][]byte those leftovers are live references the garbage collector can't free, which is what clear fixes — after it the array reads [4 7 1 9 0 0 0]. Note the map is still an allocation; this trick removes the second slice, not the set.

6Which should you use?