Files, WaitGroup. In Go programs we often want to open multiple files at once, on separate threads (goroutines). This can be done safely, provided we use a WaitGroup.
In some operating systems like macOS, a common error is reached when too many files are open. We must limit the number of simultaneous threads.
Example. The design of this code involves calling a func ReadFileSafely on multiple threads with the "go" keyword. We use a global chan to limit the number of threads.
Info To start we specify a directory containing the files we wish to open. Any number of files may be present in the folder.
Next We read in all the files in the directory. We call Add() on the WaitGroup to ensure ReadFileSafely() is part of the group.
Tip 2 When we enter ReadFileSafely, we acquire a token, and when we leave, we release the token. This limits the number of open files.
package main
import (
"fmt""os""io/ioutil""sync"
)
// For limiting threads.
var tokens = make(chan struct{}, 10)
func ReadFileSafely(fileNameHere string, folder string, wg *sync.WaitGroup) {
// For wait group (wait until all threads done).
defer wg.Done()
// Acquire token.
tokens <- struct{}{}
// Get full path.
fullPath := folder + fileNameHere
// Copy file.
inputFile, err := os.Open(fullPath)
if err != nil {
panic(err)
}
data, err := ioutil.ReadAll(inputFile)
if err != nil {
panic(err)
}
// Close file.
inputFile.Close()
// Debug info.
fmt.Println("FILE:", fullPath, len(data))
// Release token.
<-tokens
}
func main() {
// Read in all files in this folder.
folder := "/Users/sam/test/"// For waiting on threads.
var wg sync.WaitGroup
// Get files in stage.
dirRead, _ := os.Open(folder)
dirFiles, _ := dirRead.Readdir(0)
for dirIndex := range dirFiles {
fileHere := dirFiles[dirIndex]
fileNameHere := fileHere.Name()
// Increment the WaitGroup counter.
wg.Add(1)
// Thread.
go ReadFileSafely(fileNameHere, folder, &wg)
}
// Wait on threads to end.
wg.Wait()
}FILE: /Users/sam/test/top.svg 308
FILE: /Users/sam/test/search.svg 306
Wait. We call Wait() on the WaitGroup to ensure all threads have finished before the program exits. This ensures every file is read in correctly.
Important We must be careful to call Close() on a file after we call Open() on it. This can reduce "too many files open" errors.
A summary. With the "go" keyword, each call to read in a file can be placed on a separate thread. This can vastly improve performance.
Dot Net Perls is a collection of pages with code examples, which are updated to stay current. Programming is an art, and it can be learned from examples.
Donate to this site to help offset the costs of running the server. Sites like this will cease to exist if there is no financial support for them.
Sam Allen is passionate about computer languages, and he maintains 100% of the material available on this website. He hopes it makes the world a nicer place.
This page was last updated on Jan 11, 2024 (edit).