Concurrent URL Checker

Overview

Checking a few hundred URLs one at a time, waiting for each HTTP response before starting the next, would take minutes even though the actual work — waiting on a network round trip — barely touches the CPU at all. This is precisely the situation Go's concurrency primitives were designed for: a goroutine is a function that runs concurrently with the rest of the program for the cost of a few kilobytes of stack, and a channel is a typed pipe that goroutines use to send values to each other safely, without touching a shared variable directly.

By the end of this tutorial you will have a command-line tool that checks a list of URLs concurrently using a fixed-size worker pool: a handful of goroutines pull URLs off a jobs channel, check each one's HTTP status with http.Get, and push a Result onto a results channel, while sync.WaitGroup tracks exactly when every worker has finished so the program knows it is safe to stop reading results and print a summary. This worker-pool-plus-WaitGroup shape is one of the most common concurrency patterns in real Go code.

Prerequisites

Project Structure

The whole program lives in a single file, main.go. Two unbuffered-by-design channels connect three pieces: a jobs channel carrying URL strings out to the workers, a fixed pool of worker goroutines each running worker(), and a results channel carrying Result values back to main(). main() is the only goroutine that ever prints to the console — the workers never call fmt.Println directly — because interleaving prints from multiple goroutines can produce garbled output; sending structured Result values over a channel and printing them from one place avoids that entirely.

A sync.WaitGroup tracks exactly how many workers are still running. Its Wait() call is used inside its own goroutine to close the results channel the moment every worker has returned — closing it from main() directly would risk closing the channel while a worker is still trying to send on it, which panics.

Step 1: Define the Result Type

Result deliberately carries both a StatusCode and an Err field rather than only one or the other, because a URL check can fail in two structurally different ways: the server responds with a non-2xx status (a valid HTTP exchange happened, just with bad news), or the request never completes at all — DNS failure, connection refused, timeout — which net/http reports as a Go error, not a status code.

package main

// Result captures the outcome of checking one URL. Exactly one of
// StatusCode/Err is meaningful: a successful HTTP round trip fills in
// StatusCode, a request that never got a response fills in Err instead.
type Result struct {
	URL        string
	StatusCode int
	Err        error
}

Step 2: Write a Single URL Check

checkURL() builds its own http.Client with an explicit Timeout, rather than using the package-level http.Get() directly, because the zero-value default HTTP client has no timeout at all — a single unresponsive server could hang that request (and therefore that worker) forever. defer resp.Body.Close() is required any time an HTTP response is read successfully; skipping it leaks the underlying network connection.

import (
	"net/http"
	"time"
)

// checkURL performs a single GET request against url and reports its
// outcome as a Result. It never panics on a failed request — a timeout,
// DNS failure, or connection refusal all come back as Result.Err.
func checkURL(url string) Result {
	client := http.Client{
		Timeout: 5 * time.Second, // Without an explicit timeout, a hung server could block this goroutine forever
	}

	resp, err := client.Get(url)
	if err != nil {
		return Result{URL: url, Err: err} // Request never completed; StatusCode stays at its zero value, 0
	}
	defer resp.Body.Close() // Always close a successful response body to release the underlying connection

	return Result{URL: url, StatusCode: resp.StatusCode}
}

Step 3: Build a Worker Pool With Channels

A worker() is a plain function running the whole time as a goroutine: it uses for url := range jobs to keep pulling URLs off the jobs channel until that channel is closed and drained, at which point the range loop ends on its own and the goroutine returns — no explicit "stop" signal is needed. Starting a fixed number of these (rather than one goroutine per URL) caps how many requests are in flight at once, which is what keeps a list of hundreds of URLs from opening hundreds of sockets simultaneously.

// worker reads URLs from jobs until it is closed, checks each one, and
// sends the Result to results. Multiple workers run this function
// concurrently, each pulling from the same jobs channel.
func worker(jobs <-chan string, results chan<- string, wg *sync.WaitGroup) {
	// Note: jobs and results below use directional channel types (<-chan
	// and chan<-) in the real version in Step 4 — kept generic here to
	// isolate the receive/check/send loop on its own.
}

That sketch is refined in the next step once sync.WaitGroup is introduced, since a worker also needs to signal when it is done — which is a coordination concern, not a channel concern, and belongs together with the WaitGroup machinery.

Step 4: Coordinate Workers With sync.WaitGroup

Every worker's parameter types use directional channels — <-chan string for a channel this function only ever receives from, chan<- Result for one it only ever sends to — which is not required by the compiler but is idiomatic Go: it documents each channel's intended direction and makes it a compile error to accidentally send on jobs or receive from results inside worker(). defer wg.Done() at the top guarantees the WaitGroup is decremented exactly once no matter how the function returns.

import "sync"

// worker reads URLs from jobs until jobs is closed and drained, checks
// each one with checkURL, and sends the Result to results. wg.Done() is
// deferred so it fires exactly once, whenever this goroutine returns.
func worker(jobs <-chan string, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done() // Signals "this worker is finished" no matter how the function exits
	for url := range jobs {     // Keeps receiving until jobs is closed AND every buffered value has been drained
		results <- checkURL(url) // Blocks here if results is unbuffered and nothing is currently receiving
	}
}

Step 5: Wire Up main() and Print Results

main() starts a fixed number of workers, calling wg.Add(1) once per worker before starting it — Add must happen before the corresponding go worker(...) line, not inside the goroutine, or main() could reach wg.Wait() before the count is even incremented. A separate goroutine calls wg.Wait() and then close(results); closing results only after every worker has returned is what lets the final for range results loop in main() end naturally instead of blocking forever.

import "fmt"

const numWorkers = 10 // Fixed pool size: caps how many HTTP requests are in flight at once

func main() {
	urls := []string{
		"https://example.com",
		"https://golang.org",
		"https://github.com",
		"https://this-domain-should-not-exist-12345.com", // Deliberately unreachable, to exercise the error path
	}

	jobs := make(chan string, len(urls))     // Buffered so main() can queue every URL without blocking on a receiver
	results := make(chan Result, len(urls))  // Buffered the same way, so workers never block sending a result
	var wg sync.WaitGroup

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)                       // Must run before "go worker(...)" below, not inside it
		go worker(jobs, results, &wg)   // &wg: workers share one WaitGroup, not a copy each
	}

	for _, url := range urls {
		jobs <- url // Fan out: queue every URL for whichever worker picks it up next
	}
	close(jobs) // No more URLs coming; lets each worker's "for range jobs" loop end once the channel drains

	go func() {
		wg.Wait()      // Blocks until every worker has called Done()
		close(results) // Safe to close now: nothing will ever send on results again
	}()

	successCount := 0
	for result := range results { // Fan in: keeps receiving until results is closed and drained
		if result.Err != nil {
			fmt.Printf("FAIL  %-50s %v\n", result.URL, result.Err)
		} else {
			fmt.Printf("%d   %-50s\n", result.StatusCode, result.URL)
			if result.StatusCode >= 200 && result.StatusCode < 300 {
				successCount++
			}
		}
	}
	fmt.Printf("\n%d/%d URLs returned a successful status.\n", successCount, len(urls))
}

Complete Code

Here is the full program assembled in one file, ready to save as main.go and run with go run main.go.

package main

import (
	"fmt"
	"net/http"
	"sync"
	"time"
)

const numWorkers = 10

type Result struct {
	URL        string
	StatusCode int
	Err        error
}

func checkURL(url string) Result {
	client := http.Client{
		Timeout: 5 * time.Second,
	}

	resp, err := client.Get(url)
	if err != nil {
		return Result{URL: url, Err: err}
	}
	defer resp.Body.Close()

	return Result{URL: url, StatusCode: resp.StatusCode}
}

func worker(jobs <-chan string, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()
	for url := range jobs {
		results <- checkURL(url)
	}
}

func main() {
	urls := []string{
		"https://example.com",
		"https://golang.org",
		"https://github.com",
		"https://this-domain-should-not-exist-12345.com",
	}

	jobs := make(chan string, len(urls))
	results := make(chan Result, len(urls))
	var wg sync.WaitGroup

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go worker(jobs, results, &wg)
	}

	for _, url := range urls {
		jobs <- url
	}
	close(jobs)

	go func() {
		wg.Wait()
		close(results)
	}()

	successCount := 0
	for result := range results {
		if result.Err != nil {
			fmt.Printf("FAIL  %-50s %v\n", result.URL, result.Err)
		} else {
			fmt.Printf("%d   %-50s\n", result.StatusCode, result.URL)
			if result.StatusCode >= 200 && result.StatusCode < 300 {
				successCount++
			}
		}
	}
	fmt.Printf("\n%d/%d URLs returned a successful status.\n", successCount, len(urls))
}

Sample Run

Extend This Project

Summary

You built a concurrent URL checker using the worker-pool pattern that underlies a huge share of real-world Go concurrency: a bounded number of goroutines pulling work from a jobs channel, sending results back over a results channel, and a sync.WaitGroup coordinating exactly when it is safe to close that results channel. Fanning work out to a pool and fanning results back in through channels — rather than sharing a mutable slice directly between goroutines — is the same shape you will reach for any time you need many independent, blocking operations to run at once in Go.