F# Remove Duplicates From List (Seq.distinct)

Invoke Seq.distinct to remove duplicates from a list. Convert from the seq back into a list or array.

Remove duplicates. A list contains numbers: 10, 20, 10, 30. There are two tens. But we only want distinct values in this list. With F# we can eliminate dupes.

With Seq functions, we can apply common operations to lists. The distinct function receives a list. It eliminates the second occurrence of values in the list.

An example. Let us begin. We have a list of popular drinks. But there are two "coffee" strings in the list. We then call Seq.distinct and pass the list to it.

Result: The Seq.distinct function returns a seq instance. The second "coffee" was removed, and no other changes were made.

Seq

Finally: We use Seq.toList and Seq.toArray to convert our sequence to other types. These may be more useful in other parts of the program.

F# program that uses distinct, removes duplicates // This list has a duplicate string. let drinks = ["coffee"; "beer"; "tea"; "coffee"; "milk"] // Use Seq.distinct to remove duplicates. // ... The order is retained. let result = Seq.distinct drinks printfn "%A" result // We can convert from the Seq to a list or array. let resultList = Seq.toList result let resultArray = Seq.toArray result printfn "%A" resultList.Length printfn "%A" resultArray.Length Output seq ["coffee"; "beer"; "tea"; "milk"] 4 4

DistinctBy. Sometimes duplicate-removing (dedupe) can be augmented with transformations. For example we can trim whitespace from strings before comparing them.

Here: We use distinctBy and pass it a lambda expression. The lambda returns a trimmed string by calling Trim.

Fun

Result: The seq has no duplicates. The leading and trailing whitespace is ignored by distinctBy.

F# program that uses distinctBy, ignores whitespace // Has some duplicates if we ignore whitespace. let codes = [" ABC "; "DEF"; "ABC"; "XYZ"; " XYZ"] // Use Seq.distinctBy to trim strings before removing duplicates. let result = Seq.distinctBy (fun (a : string) -> a.Trim()) codes printfn "%A" result Output seq [" ABC "; "DEF"; "XYZ"]

Count by value. This function counts elements by value—if a sequence has two "Red" values it returns 2. It uses "where" to create an intermediate sequence with only matching elements.

And: With the pipeline operator, it returns the length of the sequence, which is the count of matching elements in the sequence.

Tip: This does not remove duplicates. But it tells us whether duplicates are present for a certain value.

F# program that counts instances in list // This count function accepts two arguments: value and list. // ... It computes a Seq where the value equals the argument. // It returns the length of that sequence. let count value list = Seq.where (fun y -> y = value) list |> Seq.length // Has two red values. let colors = ["Blue"; "Red"; "Red"; "Orange"] // Count blue values. let total1 = count "Blue" colors printfn "Blue = %A" total1 // Count red values. let total2 = count "Red" colors printfn "Red = %A" total2 Output Blue = 1 Red = 2

Int array. This program uses a pipelined expression to remove duplicate elements from an Int array. The result is a new Int array.Array

Description: We use Seq.ofArray to treat the array as a sequence. We then call distinct, and then toArray to return an array.

F# program that removes duplicates from Int array let numbers = [|10; 10; 20; 30; 40; 40|] // Use ofArray to use an array as a sequence. // ... Use distinct to eliminate duplicates. // Convert back into an Int array. let result = Seq.ofArray numbers |> Seq.distinct |> Seq.toArray printfn "%A" result Output [|10; 20; 30; 40|]

A review. With sequences in F#, we use prebuilt logic to process data. And we can combine operations into more powerful, sequence-based ones. This is powerful. It is elegant.