With a map, we enforce uniqueness of elements. We can use a map to remove duplicates while preserving element order. If order does not matter, we can ignore it.
Ints, retains order. Here we introduce a removeDuplicates method for ints that preserves the ordering of ints in a slice. It returns a new slice with just the duplicates removed.
Result The slice returned by removeDuplicates has all duplicates removed, but everything else about the original slice is left the same.
package main
import "fmt"
func removeDuplicates(elements []int) []int {
// Use map to record duplicates as we find them.
encountered := map[int]bool{}
result := []int{}
for v := range elements {
if encountered[elements[v]] == true {
// Do not add duplicate.
} else {
// Record this element as an encountered element.
encountered[elements[v]] = true
// Append to result slice.
result = append(result, elements[v])
}
}
// Return the new slice.
return result
}
func main() {
elements := []int{100, 200, 300, 100, 200, 400, 0}
fmt.Println(elements)
// Test our method.
result := removeDuplicates(elements)
fmt.Println(result)
}[100 200 300 100 200 400 0]
[100 200 300 400 0]
Strings, ignore ordering. This is an alternative approach. Here we remove duplicate strings in a slice. But we ignore the order of the elements—the resulting slice can be in any order.
Note We add all elements from the string slice to a string map. The value (bool) is not important here.
Finally We loop over the map and add all keys to a resulting slice. The map may store its keys in any order. This affects the result.
package main
import "fmt"
func removeDuplicatesUnordered(elements []string) []string {
encountered := map[string]bool{}
// Create a map of all unique elements.
for v:= range elements {
encountered[elements[v]] = true
}
// Place all keys from the map into a slice.
result := []string{}
for key, _ := range encountered {
result = append(result, key)
}
return result
}
func main() {
elements := []string{"cat", "dog", "cat", "bird"}
fmt.Println(elements)
// Remove string duplicates, ignoring order.
result := removeDuplicatesUnordered(elements)
fmt.Println(result)
}[cat dog cat bird]
[cat dog bird]
Nested loops. We do not need a map to detect and remove duplicate elements in a slice. On long slices, a nested loop will perform poorly. But on short slices, this is a fast approach.
Warning A nested loop has a major algorithmic inefficiency. If a slice becomes long, this Go program will not perform well.
package main
import "fmt"
func removeDuplicates(elements []int) []int {
result := []int{}
for i := 0; i < len(elements); i++ {
// Scan slice for a previous element of the same value.
exists := false
for v := 0; v < i; v++ {
if elements[v] == elements[i] {
exists = true
break
}
}
// If no previous element exists, append this one.
if !exists {
result = append(result, elements[i])
}
}
return result
}
func main() {
elements := []int{10, 20, 30, 10, 10, 20, 40}
fmt.Println(elements)
// Test the nested loop method.
result := removeDuplicates(elements)
fmt.Println(result)
}[10 20 30 10 10 20 40]
[10 20 30 40]
Has duplicates. Often the best optimization in a program is to avoid unneeded work. If a program often has no duplicates, we can first check for duplicates.
And If no duplicates are found, we avoid creating a new slice. This will optimize programs that rarely need the "dedupe" operation.
Often, a program that needs duplicates removed is poorly designed. We can store elements in a map instead of a slice. This will eliminate duplicates as we go along.
A review. Duplicates can removed in a slice. We can ensure that ordering is retained, or we can discard element orders. A map will keep performance good on large slices.
Dot Net Perls is a collection of tested code examples. Pages are continually updated to stay current, with code correctness a top priority.
Sam Allen is passionate about computer languages. In the past, his work has been recommended by Apple and Microsoft and he has studied computers at a selective university in the United States.
This page was last updated on Feb 23, 2023 (edit).
