Remove
duplicatesA Go slice can contain different values, and sometimes may have duplicate ones. We remove these elements with custom methods.
With a map, we enforce uniqueness of elements. We can use a map to remove duplicates while preserving element order. If order does not matter, we can ignore it.
Here we introduce a removeDuplicates
method for ints that preserves the ordering of ints in a slice. It returns a new slice with just the duplicates removed.
removeDuplicates
. This records ints as we encounter them in the slice.for
-loop. We record each int
in the map. We then add all unencountered ints to the result slice.removeDuplicates
has all duplicates removed, but everything else about the original slice is left the same.package main import "fmt" func removeDuplicates(elements []int) []int { // Use map to record duplicates as we find them. encountered := map[int]bool{} result := []int{} for v := range elements { if encountered[elements[v]] == true { // Do not add duplicate. } else { // Record this element as an encountered element. encountered[elements[v]] = true // Append to result slice. result = append(result, elements[v]) } } // Return the new slice. return result } func main() { elements := []int{100, 200, 300, 100, 200, 400, 0} fmt.Println(elements) // Test our method. result := removeDuplicates(elements) fmt.Println(result) }[100 200 300 100 200 400 0] [100 200 300 400 0]
This is an alternative approach. Here we remove duplicate strings in a slice. But we ignore the order of the elements—the resulting slice can be in any order.
string
slice to a string
map. The value (bool
) is not important here.package main import "fmt" func removeDuplicatesUnordered(elements []string) []string { encountered := map[string]bool{} // Create a map of all unique elements. for v:= range elements { encountered[elements[v]] = true } // Place all keys from the map into a slice. result := []string{} for key, _ := range encountered { result = append(result, key) } return result } func main() { elements := []string{"cat", "dog", "cat", "bird"} fmt.Println(elements) // Remove string duplicates, ignoring order. result := removeDuplicatesUnordered(elements) fmt.Println(result) }[cat dog cat bird] [cat dog bird]
We do not need a map to detect and remove duplicate elements in a slice. On long slices, a nested loop will perform poorly. But on short
slices, this is a fast approach.
package main import "fmt" func removeDuplicates(elements []int) []int { result := []int{} for i := 0; i < len(elements); i++ { // Scan slice for a previous element of the same value. exists := false for v := 0; v < i; v++ { if elements[v] == elements[i] { exists = true break } } // If no previous element exists, append this one. if !exists { result = append(result, elements[i]) } } return result } func main() { elements := []int{10, 20, 30, 10, 10, 20, 40} fmt.Println(elements) // Test the nested loop method. result := removeDuplicates(elements) fmt.Println(result) }[10 20 30 10 10 20 40] [10 20 30 40]
Often the best optimization in a program is to avoid unneeded work. If a program often has no duplicates, we can first check for duplicates.
Often, a program that needs duplicates removed is poorly designed. We can store elements in a map instead of a slice. This will eliminate duplicates as we go along.
Duplicates can removed in a slice. We can ensure that ordering is retained, or we can discard element orders. A map will keep performance good on large slices.