Go Read All Files, WaitGroup Example

Read All Files, WaitGroup ExampleRead all files in a directory on threads with the go keyword and the WaitGroup type.

This page was last reviewed on Apr 7, 2023.

Files, WaitGroup. In Go programs we often want to open multiple files at once, on separate threads (goroutines). This can be done safely, provided we use a WaitGroup.

In some operating systems like macOS, a common error is reached when too many files are open. We must limit the number of simultaneous threads.

Example. The design of this code involves calling a func ReadFileSafely on multiple threads with the "go" keyword. We use a global chan to limit the number of threads.

Info To start we specify a directory containing the files we wish to open. Any number of files may be present in the folder.

Next We read in all the files in the directory. We call Add() on the WaitGroup to ensure ReadFileSafely() is part of the group.

File

Tip In ReadFileSafely, we use defer and call Done() to indicate we are done with the func.

recover, panic, defer

Tip 2 When we enter ReadFileSafely, we acquire a token, and when we leave, we release the token. This limits the number of open files.

package main import ( "fmt" "os" "io/ioutil" "sync" ) // For limiting threads. var tokens = make(chan struct{}, 10) func ReadFileSafely(fileNameHere string, folder string, wg *sync.WaitGroup) { // For wait group (wait until all threads done). defer wg.Done() // Acquire token. tokens <- struct{}{} // Get full path. fullPath := folder + fileNameHere // Copy file. inputFile, err := os.Open(fullPath) if err != nil { panic(err) } data, err := ioutil.ReadAll(inputFile) if err != nil { panic(err) } // Close file. inputFile.Close() // Debug info. fmt.Println("FILE:", fullPath, len(data)) // Release token. <-tokens } func main() { // Read in all files in this folder. folder := "/Users/sam/test/" // For waiting on threads. var wg sync.WaitGroup // Get files in stage. dirRead, _ := os.Open(folder) dirFiles, _ := dirRead.Readdir(0) for dirIndex := range dirFiles { fileHere := dirFiles[dirIndex] fileNameHere := fileHere.Name() // Increment the WaitGroup counter. wg.Add(1) // Thread. go ReadFileSafely(fileNameHere, folder, &wg) } // Wait on threads to end. wg.Wait() }

FILE: /Users/sam/test/top.svg 308 FILE: /Users/sam/test/search.svg 306

Wait. We call Wait() on the WaitGroup to ensure all threads have finished before the program exits. This ensures every file is read in correctly.

Important We must be careful to call Close() on a file after we call Open() on it. This can reduce "too many files open" errors.

A summary. With the "go" keyword, each call to read in a file can be placed on a separate thread. This can vastly improve performance.

Dot Net Perls is a collection of tested code examples. Pages are continually updated to stay current, with code correctness a top priority.

Sam Allen is passionate about computer languages. In the past, his work has been recommended by Apple and Microsoft and he has studied computers at a selective university in the United States.

This page was last updated on Apr 7, 2023 (edit).

Home

Changes