F# - List Remove Duplicates (Seq.distinct)

Remove duplicates. A list contains numbers: 10, 20, 10, 30. There are two tens. But we only want distinct values in this list. With F# we can eliminate dupes.

With Seq functions, we can apply common operations to lists. The distinct function receives a list. It eliminates the second occurrence of values in the list.

An example. Let us begin—we have a list of popular drinks. But there are two "coffee" strings in the list. We then call Seq.distinct and pass the list to it.

Result The Seq.distinct function returns a seq instance. The second "coffee" was removed, and no other changes were made.

Seq

Finally We use Seq.toList and Seq.toArray to convert our sequence to other types. These may be more useful in other parts of the program.

// This list has a duplicate string.
let drinks = ["coffee"; "beer"; "tea"; "coffee"; "milk"]

// Use Seq.distinct to remove duplicates.
// ... The order is retained.
let result = Seq.distinct drinks
printfn "%A" result

// We can convert from the Seq to a list or array.
let resultList = Seq.toList result
let resultArray = Seq.toArray result
printfn "%A" resultList.Length
printfn "%A" resultArray.Length
seq ["coffee"; "beer"; "tea"; "milk"]
4
4

DistinctBy. Sometimes duplicate-removing (dedupe) can be augmented with transformations. For example we can trim whitespace from strings before comparing them.

Here We use distinctBy and pass it a lambda expression. The lambda returns a trimmed string by calling Trim.

fun

Result The seq has no duplicates. The leading and trailing whitespace is ignored by distinctBy.

// Has some duplicates if we ignore whitespace.
let codes = [" ABC "; "DEF"; "ABC"; "XYZ"; " XYZ"]

// Use Seq.distinctBy to trim strings before removing duplicates.
let result = Seq.distinctBy (fun (a : string) -> a.Trim()) codes
printfn "%A" result
seq [" ABC "; "DEF"; "XYZ"]

Count by value. This function counts elements by value—if a sequence has two "Red" values it returns 2. It uses "where" to create an intermediate sequence with only matching elements.

And With the pipeline operator, it returns the length of the sequence, which is the count of matching elements in the sequence.

Tip This does not remove duplicates. But it tells us whether duplicates are present for a certain value.

// This count function accepts two arguments: value and list.
// ... It computes a Seq where the value equals the argument.
//     It returns the length of that sequence.
let count value list =
    Seq.where (fun y -> y = value) list
    |> Seq.length

// Has two red values.
let colors = ["Blue"; "Red"; "Red"; "Orange"]

// Count blue values.
let total1 = count "Blue" colors
printfn "Blue = %A" total1

// Count red values.
let total2 = count "Red" colors
printfn "Red = %A" total2Blue = 1
Red = 2

Int array. This program uses a pipelined expression to remove duplicate elements from an Int array. The result is a new Int array.

Array

Detail We use Seq.ofArray to treat the array as a sequence. We then call distinct, and then toArray to return an array.

let numbers = [|10; 10; 20; 30; 40; 40|]

// Use ofArray to use an array as a sequence.
// ... Use distinct to eliminate duplicates.
//     Convert back into an Int array.
let result =
    Seq.ofArray numbers
    |> Seq.distinct
    |> Seq.toArray

printfn "%A" result
[|10; 20; 30; 40|]

A review. With sequences we use declarative logic to process data. And we can combine operations into more powerful, sequence-based ones—this is powerful and elegant.