Remove duplicates

A list contains strings: chair, table, table. It has a duplicate string. With Scala 3.3 we have many ways to remove duplicates from our list.

With distinct, a method on the List type, we eliminate duplicates and retain a list's order. Other approaches are possible. We can convert the list to a set: this also dedupes a list.

Example

Let us begin with this example. We create a constant, immutable List of furniture strings. The list has two instances of the string "table."

object Program {
    def main(args: Array[String]): Unit = {
        // Create a list of strings.
        val furniture = List("chair", "bed", "table", "table", "couch")
        // Get distinct strings.
        // ... This removes duplicates but retains order.
        val result = furniture.distinct

        // Print results.
        println(furniture)
        println(result)
    }
}
List(chair, bed, table, table, couch)
List(chair, bed, table, couch)

ToSet, toList

Here we use toSet and toList to strip duplicate Ints. We create a list that has six Ints, and two duplicate Ints. We then remove those duplicates.

object Program {
    def main(args: Array[String]): Unit = {
        // Create a list of Ints.
        val ids = List(10, 10, 1, 2, 3, 3)
        println(ids)

        // Convert list to set.
        // ... Duplicate elements are removed at this step.
        val set = ids.toSet
        println(set)

        // Convert set to list.
        val ids2 = set.toList
        println(ids2)
    }
}
List(10, 10, 1, 2, 3, 3)
Set(10, 1, 2, 3)
List(10, 1, 2, 3)

Map, distinct

With the map method we can transform all elements into a standard form. This may result in duplicates. With distinct we can remove the dupes.

object Program {
    def main(args: Array[String]): Unit = {
        val codes = List("abC", "Abc", "ABC", "xyz", "XyZ")
        println(codes)

        // Convert all strings to uppercase.
        // ... Then get distinct strings.
        val result = codes.map(_.toUpperCase()).distinct
        println(result)
    }
}
List(abC, Abc, ABC, xyz, XyZ)
List(ABC, XYZ)

Sets, maps

Some collections enforce unique elements. These never need to have duplicates removed—duplicates never occur. Sets and maps cannot store duplicates.

Removing duplicates from a list is an essential task in programming. It helps us understand basic manipulations of lists. And it is useful in real programs.