Concurrent URL Checker

Overview

Checking a few hundred URLs one at a time, waiting for each HTTP response before starting the next, would take minutes even though the actual work — waiting on a network round trip — barely touches the CPU at all. This is precisely the situation Go's concurrency primitives were designed for: a `goroutine` is a function that runs concurrently with the rest of the program for the cost of a few kilobytes of stack, and a `channel` is a typed pipe that goroutines use to send values to each other safely, without touching a shared variable directly.

By the end of this tutorial you will have a command-line tool that checks a list of URLs concurrently using a fixed-size worker pool: a handful of goroutines pull URLs off a jobs channel, check each one's HTTP status with `http.Get`, and push a `Result` onto a results channel, while `sync.WaitGroup` tracks exactly when every worker has finished so the program knows it is safe to stop reading results and print a summary. This worker-pool-plus-WaitGroup shape is one of the most common concurrency patterns in real Go code.

Prerequisites

Project Structure

The whole program lives in a single file, `main.go`. Two unbuffered-by-design channels connect three pieces: a `jobs` channel carrying URL strings out to the workers, a fixed pool of worker goroutines each running `worker()`, and a `results` channel carrying `Result` values back to `main()`. `main()` is the only goroutine that ever prints to the console — the workers never call `fmt.Println` directly — because interleaving prints from multiple goroutines can produce garbled output; sending structured `Result` values over a channel and printing them from one place avoids that entirely.

A `sync.WaitGroup` tracks exactly how many workers are still running. Its `Wait()` call is used inside its own goroutine to close the `results` channel the moment every worker has returned — closing it from `main()` directly would risk closing the channel while a worker is still trying to send on it, which panics.

Step 1: Define the Result Type

`Result` deliberately carries both a `StatusCode` and an `Err` field rather than only one or the other, because a URL check can fail in two structurally different ways: the server responds with a non-2xx status (a valid HTTP exchange happened, just with bad news), or the request never completes at all — DNS failure, connection refused, timeout — which `net/http` reports as a Go `error`, not a status code.

package main

// Result captures the outcome of checking one URL. Exactly one of
// StatusCode/Err is meaningful: a successful HTTP round trip fills in
// StatusCode, a request that never got a response fills in Err instead.
type Result struct {
	URL        string
	StatusCode int
	Err        error
}

Step 2: Write a Single URL Check

`checkURL()` builds its own `http.Client` with an explicit `Timeout`, rather than using the package-level `http.Get()` directly, because the zero-value default HTTP client has no timeout at all — a single unresponsive server could hang that request (and therefore that worker) forever. `defer resp.Body.Close()` is required any time an HTTP response is read successfully; skipping it leaks the underlying network connection.

import (
	"net/http"
	"time"
)

// checkURL performs a single GET request against url and reports its
// outcome as a Result. It never panics on a failed request — a timeout,
// DNS failure, or connection refusal all come back as Result.Err.
func checkURL(url string) Result {
	client := http.Client{
		Timeout: 5 * time.Second, // Without an explicit timeout, a hung server could block this goroutine forever
	}

	resp, err := client.Get(url)
	if err != nil {
		return Result{URL: url, Err: err} // Request never completed; StatusCode stays at its zero value, 0
	}
	defer resp.Body.Close() // Always close a successful response body to release the underlying connection

	return Result{URL: url, StatusCode: resp.StatusCode}
}

Step 3: Build a Worker Pool With Channels

A `worker()` is a plain function running the whole time as a goroutine: it uses `for url := range jobs` to keep pulling URLs off the `jobs` channel until that channel is closed and drained, at which point the `range` loop ends on its own and the goroutine returns — no explicit "stop" signal is needed. Starting a fixed number of these (rather than one goroutine per URL) caps how many requests are in flight at once, which is what keeps a list of hundreds of URLs from opening hundreds of sockets simultaneously.

// worker reads URLs from jobs until it is closed, checks each one, and
// sends the Result to results. Multiple workers run this function
// concurrently, each pulling from the same jobs channel.
func worker(jobs <-chan string, results chan<- string, wg *sync.WaitGroup) {
	// Note: jobs and results below use directional channel types (<-chan
	// and chan<-) in the real version in Step 4 — kept generic here to
	// isolate the receive/check/send loop on its own.
}

That sketch is refined in the next step once `sync.WaitGroup` is introduced, since a worker also needs to signal when it is done — which is a coordination concern, not a channel concern, and belongs together with the `WaitGroup` machinery.

Step 4: Coordinate Workers With sync.WaitGroup

Every worker's parameter types use directional channels — `<-chan string` for a channel this function only ever receives from, `chan<- Result` for one it only ever sends to — which is not required by the compiler but is idiomatic Go: it documents each channel's intended direction and makes it a compile error to accidentally send on `jobs` or receive from `results` inside `worker()`. `defer wg.Done()` at the top guarantees the `WaitGroup` is decremented exactly once no matter how the function returns.

import "sync"

// worker reads URLs from jobs until jobs is closed and drained, checks
// each one with checkURL, and sends the Result to results. wg.Done() is
// deferred so it fires exactly once, whenever this goroutine returns.
func worker(jobs <-chan string, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done() // Signals "this worker is finished" no matter how the function exits
	for url := range jobs {     // Keeps receiving until jobs is closed AND every buffered value has been drained
		results <- checkURL(url) // Blocks here if results is unbuffered and nothing is currently receiving
	}
}

Step 5: Wire Up main() and Print Results

`main()` starts a fixed number of workers, calling `wg.Add(1)` once per worker before starting it — `Add` must happen before the corresponding `go worker(...)` line, not inside the goroutine, or `main()` could reach `wg.Wait()` before the count is even incremented. A separate goroutine calls `wg.Wait()` and then `close(results)`; closing `results` only after every worker has returned is what lets the final `for range results` loop in `main()` end naturally instead of blocking forever.

import "fmt"

const numWorkers = 10 // Fixed pool size: caps how many HTTP requests are in flight at once

func main() {
	urls := []string{
		"https://example.com",
		"https://golang.org",
		"https://github.com",
		"https://this-domain-should-not-exist-12345.com", // Deliberately unreachable, to exercise the error path
	}

	jobs := make(chan string, len(urls))     // Buffered so main() can queue every URL without blocking on a receiver
	results := make(chan Result, len(urls))  // Buffered the same way, so workers never block sending a result
	var wg sync.WaitGroup

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)                       // Must run before "go worker(...)" below, not inside it
		go worker(jobs, results, &wg)   // &wg: workers share one WaitGroup, not a copy each
	}

	for _, url := range urls {
		jobs <- url // Fan out: queue every URL for whichever worker picks it up next
	}
	close(jobs) // No more URLs coming; lets each worker's "for range jobs" loop end once the channel drains

	go func() {
		wg.Wait()      // Blocks until every worker has called Done()
		close(results) // Safe to close now: nothing will ever send on results again
	}()

	successCount := 0
	for result := range results { // Fan in: keeps receiving until results is closed and drained
		if result.Err != nil {
			fmt.Printf("FAIL  %-50s %v\n", result.URL, result.Err)
		} else {
			fmt.Printf("%d   %-50s\n", result.StatusCode, result.URL)
			if result.StatusCode >= 200 && result.StatusCode < 300 {
				successCount++
			}
		}
	}
	fmt.Printf("\n%d/%d URLs returned a successful status.\n", successCount, len(urls))
}

Complete Code

Here is the full program assembled in one file, ready to save as `main.go` and run with `go run main.go`.

package main

import (
	"fmt"
	"net/http"
	"sync"
	"time"
)

const numWorkers = 10

type Result struct {
	URL        string
	StatusCode int
	Err        error
}

func checkURL(url string) Result {
	client := http.Client{
		Timeout: 5 * time.Second,
	}

	resp, err := client.Get(url)
	if err != nil {
		return Result{URL: url, Err: err}
	}
	defer resp.Body.Close()

	return Result{URL: url, StatusCode: resp.StatusCode}
}

func worker(jobs <-chan string, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()
	for url := range jobs {
		results <- checkURL(url)
	}
}

func main() {
	urls := []string{
		"https://example.com",
		"https://golang.org",
		"https://github.com",
		"https://this-domain-should-not-exist-12345.com",
	}

	jobs := make(chan string, len(urls))
	results := make(chan Result, len(urls))
	var wg sync.WaitGroup

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go worker(jobs, results, &wg)
	}

	for _, url := range urls {
		jobs <- url
	}
	close(jobs)

	go func() {
		wg.Wait()
		close(results)
	}()

	successCount := 0
	for result := range results {
		if result.Err != nil {
			fmt.Printf("FAIL  %-50s %v\n", result.URL, result.Err)
		} else {
			fmt.Printf("%d   %-50s\n", result.StatusCode, result.URL)
			if result.StatusCode >= 200 && result.StatusCode < 300 {
				successCount++
			}
		}
	}
	fmt.Printf("\n%d/%d URLs returned a successful status.\n", successCount, len(urls))
}

Sample Run

Extend This Project

Summary

You built a concurrent URL checker using the worker-pool pattern that underlies a huge share of real-world Go concurrency: a bounded number of goroutines pulling work from a `jobs` channel, sending results back over a `results` channel, and a `sync.WaitGroup` coordinating exactly when it is safe to close that results channel. Fanning work out to a pool and fanning results back in through channels — rather than sharing a mutable slice directly between goroutines — is the same shape you will reach for any time you need many independent, blocking operations to run at once in Go.