To handle the map, we defined two kinds of type parameters. First, the values can be of any type: V any. However, in Go, the map keys can’t be of any type. For example, we cannot use slices:
varmmap[[]byte]int
-
This code leads to a compilation error: invalid map key type []byte. Therefore, instead of accepting any key type, we are obliged to restrict type arguments so that the key type meets specific requirements. Here, being comparable (we can use == or !=). Hence, we defined K as comparable instead of a``ny.
+
This code leads to a compilation error: invalid map key type []byte. Therefore, instead of accepting any key type, we are obliged to restrict type arguments so that the key type meets specific requirements. Here, being comparable (we can use == or !=). Hence, we defined K as comparable instead of any.
Restricting type arguments to match specific requirements is called a constraint. A constraint is an interface type that can contain:
When returning an interface, be cautious about returning not a nil pointer but an explicit nil value. Otherwise, unintended consequences may result because the caller will receive a non-nil value.
+
When returning an interface, be cautious about not returning a nil pointer but an explicit nil value. Otherwise, unintended consequences may occur and the caller will receive a non-nil value.
@@ -3800,6 +3806,10 @@
Propagating an inappropriate co
However, if the response is written before or during the Kafka publication, the message shouldn’t be published.
In the latter case, calling publish will return an error because we returned the HTTP response quickly.
+
+Note
+
From Go 1.21, there is a way to create a new context without cancel. context.WithoutCancel returns a copy of parent that is not canceled when parent is canceled.
+
In summary, propagating a context should be done cautiously.
Starting a goroutine without knowing when to stop it (#62)
@@ -3867,7 +3877,7 @@
Expecti
Understanding that select with multiple channels chooses the case randomly if multiple options are possible prevents making wrong assumptions that can lead to subtle concurrency bugs.
One common mistake made by Go developers while working with channels is to make wrong assumptions about how select behaves with multiple channels.
-
For example, let's consider the following case (disconnectCh is a buffered channel):
+
For example, let's consider the following case (disconnectCh is a unbuffered channel):
インターフェイスは、オブジェクトの動作を指定する方法を提供します。複数のオブジェクトが実装できる共通項を抽出するために、インターフェイスは使用されます。Go言語のインターフェイスが大きく異なるのは、暗黙的に満たされることです。オブジェクト X がインターフェイス Y を実装していることを示す implements のような明示的なキーワードはありません。
diff --git a/site/search/search_index.json b/site/search/search_index.json
index cd0ce9d9..acc4a6da 100644
--- a/site/search/search_index.json
+++ b/site/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Common Go Mistakes","text":"
This page is a summary of all the mistakes in the 100 Go Mistakes book. Meanwhile, it's also a page open to the community. If you believe that a mistake should be added, please create a community mistake issue.
Warning
You're currently viewing a new version that I'm enriching with significantly more content. Yet, this version is still under development; please be gentle if you find an issue, and feel free to create a PR.
"},{"location":"#code-and-project-organization","title":"Code and Project Organization","text":""},{"location":"#unintended-variable-shadowing-1","title":"Unintended variable shadowing (#1)","text":"TL;DR
Avoiding shadowed variables can help prevent mistakes like referencing the wrong variable or confusing readers.
Variable shadowing occurs when a variable name is redeclared in an inner block, but this practice is prone to mistakes. Imposing a rule to forbid shadowed variables depends on personal taste. For example, sometimes it can be convenient to reuse an existing variable name like err for errors. Yet, in general, we should remain cautious because we now know that we can face a scenario where the code compiles, but the variable that receives the value is not the one expected.
Avoiding nested levels and keeping the happy path aligned on the left makes building a mental code model easier.
In general, the more nested levels a function requires, the more complex it is to read and understand. Let\u2019s see some different applications of this rule to optimize our code for readability:
When an if block returns, we should omit the else block in all cases. For example, we shouldn\u2019t write:
if foo() {\n// ...\nreturn true\n} else {\n// ...\n}\n
Instead, we omit the else block like this:
if foo() {\n// ...\nreturn true\n}\n// ...\n
We can also follow this logic with a non-happy path:
if s != \"\" {\n// ...\n} else {\nreturn errors.New(\"empty string\")\n}\n
Here, an empty s represents the non-happy path. Hence, we should flip the condition like so:
if s == \"\" {\nreturn errors.New(\"empty string\")\n}\n// ...\n
Writing readable code is an important challenge for every developer. Striving to reduce the number of nested blocks, aligning the happy path on the left, and returning as early as possible are concrete means to improve our code\u2019s readability.
When initializing variables, remember that init functions have limited error handling and make state handling and testing more complex. In most cases, initializations should be handled as specific functions.
An init function is a function used to initialize the state of an application. It takes no arguments and returns no result (a func() function). When a package is initialized, all the constant and variable declarations in the package are evaluated. Then, the init functions are executed.
Init functions can lead to some issues:
They can limit error management.
They can complicate how to implement tests (for example, an external dependency must be set up, which may not be necessary for the scope of unit tests).
If the initialization requires us to set a state, that has to be done through global variables.
We should be cautious with init functions. They can be helpful in some situations, however, such as defining static configuration. Otherwise, and in most cases, we should handle initializations through ad hoc functions.
Source code
"},{"location":"#overusing-getters-and-setters-4","title":"Overusing getters and setters (#4)","text":"TL;DR
Forcing the use of getters and setters isn\u2019t idiomatic in Go. Being pragmatic and finding the right balance between efficiency and blindly following certain idioms should be the way to go.
Data encapsulation refers to hiding the values or state of an object. Getters and setters are means to enable encapsulation by providing exported methods on top of unexported object fields.
In Go, there is no automatic support for getters and setters as we see in some languages. It is also considered neither mandatory nor idiomatic to use getters and setters to access struct fields. We shouldn\u2019t overwhelm our code with getters and setters on structs if they don\u2019t bring any value. We should be pragmatic and strive to find the right balance between efficiency and following idioms that are sometimes considered indisputable in other programming paradigms.
Remember that Go is a unique language designed for many characteristics, including simplicity. However, if we find a need for getters and setters or, as mentioned, foresee a future need while guaranteeing forward compatibility, there\u2019s nothing wrong with using them.
Abstractions should be discovered, not created. To prevent unnecessary complexity, create an interface when you need it and not when you foresee needing it, or if you can at least prove the abstraction to be a valid one.
An interface provides a way to specify the behavior of an object. We use interfaces to create common abstractions that multiple objects can implement. What makes Go interfaces so different is that they are satisfied implicitly. There is no explicit keyword like implements to mark that an object X implements interface Y.
In general, we can define three main use cases where interfaces are generally considered as bringing value: factoring out a common behavior, creating some decoupling, and restricting a type to a certain behavior. Yet, this list isn't exhaustive and will also depend on the context we face.
In many occasions, interfaces are made to create abstractions. And the main caveat when programming meets abstractions is remembering that abstractions should be discovered, not created. What does this mean? It means we shouldn\u2019t start creating abstractions in our code if there is no immediate reason to do so. We shouldn\u2019t design with interfaces but wait for a concrete need. Said differently, we should create an interface when we need it, not when we foresee that we could need it. What\u2019s the main problem if we overuse interfaces? The answer is that they make the code flow more complex. Adding a useless level of indirection doesn\u2019t bring any value; it creates a worthless abstraction making the code more difficult to read, understand, and reason about. If we don\u2019t have a strong reason for adding an interface and it\u2019s unclear how an interface makes a code better, we should challenge this interface\u2019s purpose. Why not call the implementation directly?
We should be cautious when creating abstractions in our code (abstractions should be discovered, not created). It\u2019s common for us, software developers, to overengineer our code by trying to guess what the perfect level of abstraction is, based on what we think we might need later. This process should be avoided because, in most cases, it pollutes our code with unnecessary abstractions, making it more complex to read.
Rob Pike
Don\u2019t design with interfaces, discover them.
Let\u2019s not try to solve a problem abstractly but solve what has to be solved now. Last, but not least, if it\u2019s unclear how an interface makes the code better, we should probably consider removing it to make our code simpler.
Source code
"},{"location":"#interface-on-the-producer-side-6","title":"Interface on the producer side (#6)","text":"TL;DR
Keeping interfaces on the client side avoids unnecessary abstractions.
Interfaces are satisfied implicitly in Go, which tends to be a gamechanger compared to languages with an explicit implementation. In most cases, the approach to follow is similar to what we described in the previous section: abstractions should be discovered, not created. This means that it\u2019s not up to the producer to force a given abstraction for all the clients. Instead, it\u2019s up to the client to decide whether it needs some form of abstraction and then determine the best abstraction level for its needs.
An interface should live on the consumer side in most cases. However, in particular contexts (for example, when we know\u2014not foresee\u2014that an abstraction will be helpful for consumers), we may want to have it on the producer side. If we do, we should strive to keep it as minimal as possible, increasing its reusability potential and making it more easily composable.
To prevent being restricted in terms of flexibility, a function shouldn\u2019t return interfaces but concrete implementations in most cases. Conversely, a function should accept interfaces whenever possible.
In most cases, we shouldn\u2019t return interfaces but concrete implementations. Otherwise, it can make our design more complex due to package dependencies and can restrict flexibility because all the clients would have to rely on the same abstraction. Again, the conclusion is similar to the previous sections: if we know (not foresee) that an abstraction will be helpful for clients, we can consider returning an interface. Otherwise, we shouldn\u2019t force abstractions; they should be discovered by clients. If a client needs to abstract an implementation for whatever reason, it can still do that on the client\u2019s side.
Only use any if you need to accept or return any possible type, such as json.Marshal. Otherwise, any doesn\u2019t provide meaningful information and can lead to compile-time issues by allowing a caller to call methods with any data type.
The any type can be helpful if there is a genuine need for accepting or returning any possible type (for instance, when it comes to marshaling or formatting). In general, we should avoid overgeneralizing the code we write at all costs. Perhaps a little bit of duplicated code might occasionally be better if it improves other aspects such as code expressiveness.
Source code
"},{"location":"#being-confused-about-when-to-use-generics-9","title":"Being confused about when to use generics (#9)","text":"TL;DR
Relying on generics and type parameters can prevent writing boilerplate code to factor out elements or behaviors. However, do not use type parameters prematurely, but only when you see a concrete need for them. Otherwise, they introduce unnecessary abstractions and complexity.
Read the full section here.
Source code
"},{"location":"#not-being-aware-of-the-possible-problems-with-type-embedding-10","title":"Not being aware of the possible problems with type embedding (#10)","text":"TL;DR
Using type embedding can also help avoid boilerplate code; however, ensure that doing so doesn\u2019t lead to visibility issues where some fields should have remained hidden.
When creating a struct, Go offers the option to embed types. But this can sometimes lead to unexpected behaviors if we don\u2019t understand all the implications of type embedding. Throughout this section, we look at how to embed types, what these bring, and the possible issues.
In Go, a struct field is called embedded if it\u2019s declared without a name. For example,
type Foo struct {\nBar // Embedded field\n}\ntype Bar struct {\nBaz int\n}\n
In the Foo struct, the Bar type is declared without an associated name; hence, it\u2019s an embedded field.
We use embedding to promote the fields and methods of an embedded type. Because Bar contains a Baz field, this field is promoted to Foo. Therefore, Baz becomes available from Foo.
What can we say about type embedding? First, let\u2019s note that it\u2019s rarely a necessity, and it means that whatever the use case, we can probably solve it as well without type embedding. Type embedding is mainly used for convenience: in most cases, to promote behaviors.
If we decide to use type embedding, we need to keep two main constraints in mind:
It shouldn\u2019t be used solely as some syntactic sugar to simplify accessing a field (such as Foo.Baz() instead of Foo.Bar.Baz()). If this is the only rationale, let\u2019s not embed the inner type and use a field instead.
It shouldn\u2019t promote data (fields) or a behavior (methods) we want to hide from the outside: for example, if it allows clients to access a locking behavior that should remain private to the struct.
Using type embedding consciously by keeping these constraints in mind can help avoid boilerplate code with additional forwarding methods. However, let\u2019s make sure we don\u2019t do it solely for cosmetics and not promote elements that should remain hidden.
Source code
"},{"location":"#not-using-the-functional-options-pattern-11","title":"Not using the functional options pattern (#11)","text":"TL;DR
To handle options conveniently and in an API-friendly manner, use the functional options pattern.
Although there are different implementations with minor variations, the main idea is as follows:
An unexported struct holds the configuration: options.
Each option is a function that returns the same type: type Option func(options *options) error. For example, WithPort accepts an int argument that represents the port and returns an Option type that represents how to update the options struct.
type options struct {\nport *int\n}\ntype Option func(options *options) error\nfunc WithPort(port int) Option {\nreturn func(options *options) error {\nif port < 0 {\nreturn errors.New(\"port should be positive\")\n}\noptions.port = &port\nreturn nil\n}\n}\nfunc NewServer(addr string, opts ...Option) ( *http.Server, error) { <1>\nvar options options <2>\nfor _, opt := range opts { <3>\nerr := opt(&options) <4>\nif err != nil {\nreturn nil, err\n}\n}\n// At this stage, the options struct is built and contains the config\n// Therefore, we can implement our logic related to port configuration\nvar port int\nif options.port == nil {\nport = defaultHTTPPort\n} else {\nif *options.port == 0 {\nport = randomPort()\n} else {\nport = *options.port\n}\n}\n// ...\n}\n
The functional options pattern provides a handy and API-friendly way to handle options. Although the builder pattern can be a valid option, it has some minor downsides (having to pass a config struct that can be empty or a less handy way to handle error management) that tend to make the functional options pattern the idiomatic way to deal with these kind of problems in Go.
Source code
"},{"location":"#project-misorganization-project-structure-and-package-organization-12","title":"Project misorganization (project structure and package organization) (#12)","text":"
Regarding the overall organization, there are different schools of thought. For example, should we organize our application by context or by layer? It depends on our preferences. We may favor grouping code per context (such as the customer context, the contract context, etc.), or we may favor following hexagonal architecture principles and group per technical layer. If the decision we make fits our use case, it cannot be a wrong decision, as long as we remain consistent with it.
Regarding packages, there are multiple best practices that we should follow. First, we should avoid premature packaging because it might cause us to overcomplicate a project. Sometimes, it\u2019s better to use a simple organization and have our project evolve when we understand what it contains rather than forcing ourselves to make the perfect structure up front. Granularity is another essential thing to consider. We should avoid having dozens of nano packages containing only one or two files. If we do, it\u2019s because we have probably missed some logical connections across these packages, making our project harder for readers to understand. Conversely, we should also avoid huge packages that dilute the meaning of a package name.
Package naming should also be considered with care. As we all know (as developers), naming is hard. To help clients understand a Go project, we should name our packages after what they provide, not what they contain. Also, naming should be meaningful. Therefore, a package name should be short, concise, expressive, and, by convention, a single lowercase word.
Regarding what to export, the rule is pretty straightforward. We should minimize what should be exported as much as possible to reduce the coupling between packages and keep unnecessary exported elements hidden. If we are unsure whether to export an element or not, we should default to not exporting it. Later, if we discover that we need to export it, we can adjust our code. Let\u2019s also keep in mind some exceptions, such as making fields exported so that a struct can be unmarshaled with encoding/json.
Organizing a project isn\u2019t straightforward, but following these rules should help make it easier to maintain. However, remember that consistency is also vital to ease maintainability. Therefore, let\u2019s make sure that we keep things as consistent as possible within a codebase.
Note
In 2023, the Go team has published an official guideline for organizing / structuring a Go project: go.dev/doc/modules/layout
Naming is a critical piece of application design. Creating packages such as common, util, and shared doesn\u2019t bring much value for the reader. Refactor such packages into meaningful and specific package names.
Also, bear in mind that naming a package after what it provides and not what it contains can be an efficient way to increase its expressiveness.
Source code
"},{"location":"#ignoring-package-name-collisions-14","title":"Ignoring package name collisions (#14)","text":"TL;DR
To avoid naming collisions between variables and packages, leading to confusion or perhaps even bugs, use unique names for each one. If this isn\u2019t feasible, use an import alias to change the qualifier to differentiate the package name from the variable name, or think of a better name.
Package collisions occur when a variable name collides with an existing package name, preventing the package from being reused. We should prevent variable name collisions to avoid ambiguity. If we face a collision, we should either find another meaningful name or use an import alias.
To help clients and maintainers understand your code\u2019s purpose, document exported elements.
Documentation is an important aspect of coding. It simplifies how clients can consume an API but can also help in maintaining a project. In Go, we should follow some rules to make our code idiomatic:
First, every exported element must be documented. Whether it is a structure, an interface, a function, or something else, if it\u2019s exported, it must be documented. The convention is to add comments, starting with the name of the exported element.
As a convention, each comment should be a complete sentence that ends with punctuation. Also bear in mind that when we document a function (or a method), we should highlight what the function intends to do, not how it does it; this belongs to the core of a function and comments, not documentation. Furthermore, the documentation should ideally provide enough information that the consumer does not have to look at our code to understand how to use an exported element.
When it comes to documenting a variable or a constant, we might be interested in conveying two aspects: its purpose and its content. The former should live as code documentation to be useful for external clients. The latter, though, shouldn\u2019t necessarily be public.
To help clients and maintainers understand a package\u2019s scope, we should also document each package. The convention is to start the comment with // Package followed by the package name. The first line of a package comment should be concise. That\u2019s because it will appear in the package. Then, we can provide all the information we need in the following lines.
Documenting our code shouldn\u2019t be a constraint. We should take the opportunity to make sure it helps clients and maintainers to understand the purpose of our code.
"},{"location":"#not-using-linters-16","title":"Not using linters (#16)","text":"TL;DR
To improve code quality and consistency, use linters and formatters.
A linter is an automatic tool to analyze code and catch errors. The scope of this section isn\u2019t to give an exhaustive list of the existing linters; otherwise, it will become deprecated pretty quickly. But we should understand and remember why linters are essential for most Go projects.
However, if you\u2019re not a regular user of linters, here is a list that you may want to use daily:
https://golang.org/cmd/vet\u2014A standard Go analyzer
https://github.com/jgautheron/goconst\u2014A repeated string constants analyzer * Besides linters, we should also use code formatters to fix code style. Here is a list of some code formatters for you to try:
https://golang.org/cmd/gofmt\u2014A standard Go code formatter
https://godoc.org/golang.org/x/tools/cmd/goimports\u2014A standard Go imports formatter * Meanwhile, we should also look at golangci-lint (https://github.com/golangci/golangci-lint). It\u2019s a linting tool that provides a facade on top of many useful linters and formatters. Also, it allows running the linters in parallel to improve analysis speed, which is quite handy.
Linters and formatters are a powerful way to improve the quality and consistency of our codebase. Let\u2019s take the time to understand which one we should use and make sure we automate their execution (such as a CI or Git precommit hook).
"},{"location":"#data-types","title":"Data Types","text":""},{"location":"#creating-confusion-with-octal-literals-17","title":"Creating confusion with octal literals (#17)","text":"TL;DR
When reading existing code, bear in mind that integer literals starting with 0 are octal numbers. Also, to improve readability, make octal integers explicit by prefixing them with 0o.
Octal numbers start with a 0 (e.g., 010 is equal to 8 in base 10). To improve readability and avoid potential mistakes for future code readers, we should make octal numbers explicit using the 0o prefix (e.g., 0o10).
We should also note the other integer literal representations:
Binary\u2014Uses a 0b or 0B prefix (for example, 0b100 is equal to 4 in base 10)
Hexadecimal\u2014Uses an 0x or 0X prefix (for example, 0xF is equal to 15 in base 10)
Imaginary\u2014Uses an i suffix (for example, 3i)
We can also use an underscore character (_) as a separator for readability. For example, we can write 1 billion this way: 1_000_000_000. We can also use the underscore character with other representations (for example, 0b00_00_01).
Because integer overflows and underflows are handled silently in Go, you can implement your own functions to catch them.
In Go, an integer overflow that can be detected at compile time generates a compilation error. For example,
var counter int32 = math.MaxInt32 + 1\n
constant 2147483648 overflows int32\n
However, at run time, an integer overflow or underflow is silent; this does not lead to an application panic. It is essential to keep this behavior in mind, because it can lead to sneaky bugs (for example, an integer increment or addition of positive integers that leads to a negative result).
Making floating-point comparisons within a given delta can ensure that your code is portable. When performing addition or subtraction, group the operations with a similar order of magnitude to favor accuracy. Also, perform multiplication and division before addition and subtraction.
In Go, there are two floating-point types (if we omit imaginary numbers): float32 and float64. The concept of a floating point was invented to solve the major problem with integers: their inability to represent fractional values. To avoid bad surprises, we need to know that floating-point arithmetic is an approximation of real arithmetic.
For that, we\u2019ll look at a multiplication example:
var n float32 = 1.0001\nfmt.Println(n * n)\n
We may expect this code to print the result of 1.0001 * 1.0001 = 1.00020001, right? However, running it on most x86 processors prints 1.0002, instead.
Because Go\u2019s float32 and float64 types are approximations, we have to bear a few rules in mind:
When comparing two floating-point numbers, check that their difference is within an acceptable range.
When performing additions or subtractions, group operations with a similar order of magnitude for better accuracy.
To favor accuracy, if a sequence of operations requires addition, subtraction, multiplication, or division, perform the multiplication and division operations first.
Source code
"},{"location":"#not-understanding-slice-length-and-capacity-20","title":"Not understanding slice length and capacity (#20)","text":"TL;DR
Understanding the difference between slice length and capacity should be part of a Go developer\u2019s core knowledge. The slice length is the number of available elements in the slice, whereas the slice capacity is the number of elements in the backing array.
When creating a slice, initialize it with a given length or capacity if its length is already known. This reduces the number of allocations and improves performance.
While initializing a slice using make, we can provide a length and an optional capacity. Forgetting to pass an appropriate value for both of these parameters when it makes sense is a widespread mistake. Indeed, it can lead to multiple copies and additional effort for the GC to clean the temporary backing arrays. Performance-wise, there\u2019s no good reason not to give the Go runtime a helping hand.
Our options are to allocate a slice with either a given capacity or a given length. Of these two solutions, we have seen that the second tends to be slightly faster. But using a given capacity and append can be easier to implement and read in some contexts.
Source code
"},{"location":"#being-confused-about-nil-vs-empty-slice-22","title":"Being confused about nil vs. empty slice (#22)","text":"TL;DR
To prevent common confusions such as when using the encoding/json or the reflect package, you need to understand the difference between nil and empty slices. Both are zero-length, zero-capacity slices, but only a nil slice doesn\u2019t require allocation.
In Go, there is a distinction between nil and empty slices. A nil slice is equals to nil, whereas an empty slice has a length of zero. A nil slice is empty, but an empty slice isn\u2019t necessarily nil. Meanwhile, a nil slice doesn\u2019t require any allocation. We have seen throughout this section how to initialize a slice depending on the context by using
var s []string if we aren\u2019t sure about the final length and the slice can be empty
[]string(nil) as syntactic sugar to create a nil and empty slice
make([]string, length) if the future length is known
The last option, []string{}, should be avoided if we initialize the slice without elements. Finally, let\u2019s check whether the libraries we use make the distinctions between nil and empty slices to prevent unexpected behaviors.
Source code
"},{"location":"#not-properly-checking-if-a-slice-is-empty-23","title":"Not properly checking if a slice is empty (#23)","text":"TL;DR
To check if a slice doesn\u2019t contain any element, check its length. This check works regardless of whether the slice is nil or empty. The same goes for maps. To design unambiguous APIs, you shouldn\u2019t distinguish between nil and empty slices.
To determine whether a slice has elements, we can either do it by checking if the slice is nil or if its length is equal to 0. Checking the length is the best option to follow as it will cover both if the slice is empty or is the slice is nil.
Meanwhile, when designing interfaces, we should avoid distinguishing nil and empty slices, which leads to subtle programming errors. When returning slices, it should make neither a semantic nor a technical difference if we return a nil or empty slice. Both should mean the same thing for the callers. This principle is the same with maps. To check if a map is empty, check its length, not whether it\u2019s nil.
Source code
"},{"location":"#not-making-slice-copies-correctly-24","title":"Not making slice copies correctly (#24)","text":"TL;DR
To copy one slice to another using the copy built-in function, remember that the number of copied elements corresponds to the minimum between the two slice\u2019s lengths.
Copying elements from one slice to another is a reasonably frequent operation. When using copy, we must recall that the number of elements copied to the destination corresponds to the minimum between the two slices\u2019 lengths. Also bear in mind that other alternatives exist to copy a slice, so we shouldn\u2019t be surprised if we find them in a codebase.
Source code
"},{"location":"#unexpected-side-effects-using-slice-append-25","title":"Unexpected side effects using slice append (#25)","text":"TL;DR
Using copy or the full slice expression is a way to prevent append from creating conflicts if two different functions use slices backed by the same array. However, only a slice copy prevents memory leaks if you want to shrink a large slice.
When using slicing, we must remember that we can face a situation leading to unintended side effects. If the resulting slice has a length smaller than its capacity, append can mutate the original slice. If we want to restrict the range of possible side effects, we can use either a slice copy or the full slice expression, which prevents us from doing a copy.
Note
s[low:high:max] (full slice expression): This statement creates a slice similar to the one created with s[low:high], except that the resulting slice\u2019s capacity is equal to max - low.
Source code
"},{"location":"#slices-and-memory-leaks-26","title":"Slices and memory leaks (#26)","text":"TL;DR
Working with a slice of pointers or structs with pointer fields, you can avoid memory leaks by marking as nil the elements excluded by a slicing operation.
Remember that slicing a large slice or array can lead to potential high memory consumption. The remaining space won\u2019t be reclaimed by the GC, and we can keep a large backing array despite using only a few elements. Using a slice copy is the solution to prevent such a case.
Source code
"},{"location":"#slice-and-pointers","title":"Slice and pointers","text":"
When we use the slicing operation with pointers or structs with pointer fields, we need to know that the GC won\u2019t reclaim these elements. In that case, the two options are to either perform a copy or explicitly mark the remaining elements or their fields to nil.
When creating a map, initialize it with a given length if its length is already known. This reduces the number of allocations and improves performance.
A map provides an unordered collection of key-value pairs in which all the keys are distinct. In Go, a map is based on the hash table data structure. Internally, a hash table is an array of buckets, and each bucket is a pointer to an array of key-value pairs.
If we know up front the number of elements a map will contain, we should create it by providing an initial size. Doing this avoids potential map growth, which is quite heavy computation-wise because it requires reallocating enough space and rebalancing all the elements.
Source code
"},{"location":"#maps-and-memory-leaks-28","title":"Maps and memory leaks (#28)","text":"TL;DR
A map can always grow in memory, but it never shrinks. Hence, if it leads to some memory issues, you can try different options, such as forcing Go to recreate the map or using pointers.
To compare types in Go, you can use the == and != operators if two types are comparable: Booleans, numerals, strings, pointers, channels, and structs are composed entirely of comparable types. Otherwise, you can either use reflect.DeepEqual and pay the price of reflection or use custom implementations and libraries.
It\u2019s essential to understand how to use == and != to make comparisons effectively. We can use these operators on operands that are comparable:
Booleans\u2014Compare whether two Booleans are equal.
Numerics (int, float, and complex types)\u2014Compare whether two numerics are equal.
Strings\u2014Compare whether two strings are equal.
Channels\u2014Compare whether two channels were created by the same call to make or if both are nil.
Interfaces\u2014Compare whether two interfaces have identical dynamic types and equal dynamic values or if both are nil.
Pointers\u2014Compare whether two pointers point to the same value in memory or if both are nil.
Structs and arrays\u2014Compare whether they are composed of similar types.
Note
We can also use the ?, >=, <, and > operators with numeric types to compare values and with strings to compare their lexical order.
If operands are not comparable (e.g., slices and maps), we have to use other options such as reflection. Reflection is a form of metaprogramming, and it refers to the ability of an application to introspect and modify its structure and behavior. For example, in Go, we can use reflect.DeepEqual. This function reports whether two elements are deeply equal by recursively traversing two values. The elements it accepts are basic types plus arrays, structs, slices, maps, pointers, interfaces, and functions. Yet, the main catch is the performance penalty.
If performance is crucial at run time, implementing our custom method might be the best solution. One additional note: we must remember that the standard library has some existing comparison methods. For example, we can use the optimized bytes.Compare function to compare two slices of bytes. Before implementing a custom method, we need to make sure we don\u2019t reinvent the wheel.
Source code
"},{"location":"#control-structures","title":"Control Structures","text":""},{"location":"#ignoring-that-elements-are-copied-in-range-loops-30","title":"Ignoring that elements are copied in range loops (#30)","text":"TL;DR
The value element in a range loop is a copy. Therefore, to mutate a struct, for example, access it via its index or via a classic for loop (unless the element or the field you want to modify is a pointer).
A range loop allows iterating over different data structures:
String
Array
Pointer to an array
Slice
Map
Receiving channel
Compared to a classic for loop, a range loop is a convenient way to iterate over all the elements of one of these data structures, thanks to its concise syntax.
Yet, we should remember that the value element in a range loop is a copy. Therefore, if the value is a struct we need to mutate, we will only update the copy, not the element itself, unless the value or field we modify is a pointer. The favored options are to access the element via the index using a range loop or a classic for loop.
Source code
"},{"location":"#ignoring-how-arguments-are-evaluated-in-range-loops-channels-and-arrays-31","title":"Ignoring how arguments are evaluated in range loops (channels and arrays) (#31)","text":"TL;DR
Understanding that the expression passed to the range operator is evaluated only once before the beginning of the loop can help you avoid common mistakes such as inefficient assignment in channel or slice iteration.
The range loop evaluates the provided expression only once, before the beginning of the loop, by doing a copy (regardless of the type). We should remember this behavior to avoid common mistakes that might, for example, lead us to access the wrong element. For example:
a := [3]int{0, 1, 2}\nfor i, v := range a {\na[2] = 10\nif i == 2 {\nfmt.Println(v)\n}\n}\n
This code updates the last index to 10. However, if we run this code, it does not print 10; it prints 2.
Source code
"},{"location":"#ignoring-the-impacts-of-using-pointer-elements-in-range-loops-32","title":"Ignoring the impacts of using pointer elements in range loops (#32)","text":"TL;DR
Using a local variable or accessing an element using an index, you can prevent mistakes while copying pointers inside a loop.
When iterating over a data structure using a range loop, we must recall that all the values are assigned to a unique variable with a single unique address. Therefore, if we store a pointer referencing this variable during each iteration, we will end up in a situation where we store the same pointer referencing the same element: the latest one. We can overcome this issue by forcing the creation of a local variable in the loop\u2019s scope or creating a pointer referencing a slice element via its index. Both solutions are fine.
Source code
"},{"location":"#making-wrong-assumptions-during-map-iterations-ordering-and-map-insert-during-iteration-33","title":"Making wrong assumptions during map iterations (ordering and map insert during iteration) (#33)","text":"TL;DR
To ensure predictable outputs when using maps, remember that a map data structure:
Doesn\u2019t order the data by keys
Doesn\u2019t preserve the insertion order
Doesn\u2019t have a deterministic iteration order
Doesn\u2019t guarantee that an element added during an iteration will be produced during this iteration
Source code
"},{"location":"#ignoring-how-the-break-statement-works-34","title":"Ignoring how the break statement works (#34)","text":"TL;DR
Using break or continue with a label enforces breaking a specific statement. This can be helpful with switch or select statements inside loops.
A break statement is commonly used to terminate the execution of a loop. When loops are used in conjunction with switch or select, developers frequently make the mistake of breaking the wrong statement. For example:
for i := 0; i < 5; i++ {\nfmt.Printf(\"%d \", i)\nswitch i {\ndefault:\ncase 2:\nbreak\n}\n}\n
The break statement doesn\u2019t terminate the for loop: it terminates the switch statement, instead. Hence, instead of iterating from 0 to 2, this code iterates from 0 to 4: 0 1 2 3 4.
One essential rule to keep in mind is that a break statement terminates the execution of the innermost for, switch, or select statement. In the previous example, it terminates the switch statement.
To break the loop instead of the switch statement, the most idiomatic way is to use a label:
loop:\nfor i := 0; i < 5; i++ {\nfmt.Printf(\"%d \", i)\nswitch i {\ndefault:\ncase 2:\nbreak loop\n}\n}\n
Here, we associate the loop label with the for loop. Then, because we provide the loop label to the break statement, it breaks the loop, not the switch. Therefore, this new version will print 0 1 2, as we expected.
Source code
"},{"location":"#using-defer-inside-a-loop-35","title":"Using defer inside a loop (#35)","text":"TL;DR
Extracting loop logic inside a function leads to executing a defer statement at the end of each iteration.
The defer statement delays a call\u2019s execution until the surrounding function returns. It\u2019s mainly used to reduce boilerplate code. For example, if a resource has to be closed eventually, we can use defer to avoid repeating the closure calls before every single return.
One common mistake with defer is to forget that it schedules a function call when the surrounding function returns. For example:
func readFiles(ch <-chan string) error {\nfor path := range ch {\nfile, err := os.Open(path)\nif err != nil {\nreturn err\n}\ndefer file.Close()\n// Do something with file\n}\nreturn nil\n}\n
The defer calls are executed not during each loop iteration but when the readFiles function returns. If readFiles doesn\u2019t return, the file descriptors will be kept open forever, causing leaks.
One common option to fix this problem is to create a surrounding function after defer, called during each iteration:
Another solution is to make the readFile function a closure but intrinsically, this remains the same solution: adding another surrounding function to execute the defer calls during each iteration.
Source code
"},{"location":"#strings","title":"Strings","text":""},{"location":"#not-understanding-the-concept-of-rune-36","title":"Not understanding the concept of rune (#36)","text":"TL;DR
Understanding that a rune corresponds to the concept of a Unicode code point and that it can be composed of multiple bytes should be part of the Go developer\u2019s core knowledge to work accurately with strings.
As runes are everywhere in Go, it's important to understand the following:
A charset is a set of characters, whereas an encoding describes how to translate a charset into binary.
In Go, a string references an immutable slice of arbitrary bytes.
Go source code is encoded using UTF-8. Hence, all string literals are UTF-8 strings. But because a string can contain arbitrary bytes, if it\u2019s obtained from somewhere else (not the source code), it isn\u2019t guaranteed to be based on the UTF-8 encoding.
A rune corresponds to the concept of a Unicode code point, meaning an item represented by a single value.
Using UTF-8, a Unicode code point can be encoded into 1 to 4 bytes.
Using len() on a string in Go returns the number of bytes, not the number of runes.
Iterating on a string with the range operator iterates on the runes with the index corresponding to the starting index of the rune\u2019s byte sequence. To access a specific rune index (such as the third rune), convert the string into a []rune.
Iterating on a string is a common operation for developers. Perhaps we want to perform an operation for each rune in the string or implement a custom function to search for a specific substring. In both cases, we have to iterate on the different runes of a string. But it\u2019s easy to get confused about how iteration works.
For example, consider the following example:
s := \"h\u00eallo\"\nfor i := range s {\nfmt.Printf(\"position %d: %c\\n\", i, s[i])\n}\nfmt.Printf(\"len=%d\\n\", len(s))\n
Let's highlight three points that might be confusing:
The second rune is \u00c3 in the output instead of \u00ea.
We jumped from position 1 to position 3: what is at position 2?
len returns a count of 6, whereas s contains only 5 runes.
Let\u2019s start with the last observation. We already mentioned that len returns the number of bytes in a string, not the number of runes. Because we assigned a string literal to s, s is a UTF-8 string. Meanwhile, the special character \"\u00ea\" isn\u2019t encoded in a single byte; it requires 2 bytes. Therefore, calling len(s) returns 6.
Meanwhile, in the previous example, we have to understand that we don't iterate over each rune; instead, we iterate over each starting index of a rune:
Printing s[i] doesn\u2019t print the ith rune; it prints the UTF-8 representation of the byte at index i. Hence, we printed \"h\u00c3llo\" instead of \"h\u00eallo\".
If we want to print all the different runes, we can either use the value element of the range operator:
s := \"h\u00eallo\"\nfor i, r := range s {\nfmt.Printf(\"position %d: %c\\n\", i, r)\n}\n
Or, we can convert the string into a slice of runes and iterate over it:
s := \"h\u00eallo\"\nrunes := []rune(s)\nfor i, r := range runes {\nfmt.Printf(\"position %d: %c\\n\", i, r)\n}\n
Note that this solution introduces a run-time overhead compared to the previous one. Indeed, converting a string into a slice of runes requires allocating an additional slice and converting the bytes into runes: an O(n) time complexity with n the number of bytes in the string. Therefore, if we want to iterate over all the runes, we should use the first solution.
However, if we want to access the ith rune of a string with the first option, we don\u2019t have access to the rune index; rather, we know the starting index of a rune in the byte sequence.
s := \"h\u00eallo\"\nr := []rune(s)[4]\nfmt.Printf(\"%c\\n\", r) // o\n
Source code
"},{"location":"#misusing-trim-functions-38","title":"Misusing trim functions (#38)","text":"TL;DR
strings.TrimRight/strings.TrimLeft removes all the trailing/leading runes contained in a given set, whereas strings.TrimSuffix/strings.TrimPrefix returns a string without a provided suffix/prefix.
Concatenating a list of strings should be done with strings.Builder to prevent allocating a new string during each iteration.
Let\u2019s consider a concat function that concatenates all the string elements of a slice using the += operator:
func concat(values []string) string {\ns := \"\"\nfor _, value := range values {\ns += value\n}\nreturn s\n}\n
During each iteration, the += operator concatenates s with the value string. At first sight, this function may not look wrong. But with this implementation, we forget one of the core characteristics of a string: its immutability. Therefore, each iteration doesn\u2019t update s; it reallocates a new string in memory, which significantly impacts the performance of this function.
Fortunately, there is a solution to deal with this problem, using strings.Builder:
func concat(values []string) string {\nsb := strings.Builder{}\nfor _, value := range values {\n_, _ = sb.WriteString(value)\n}\nreturn sb.String()\n}\n
During each iteration, we constructed the resulting string by calling the WriteString method that appends the content of value to its internal buffer, hence minimizing memory copying.
Note
WriteString returns an error as the second output, but we purposely ignore it. Indeed, this method will never return a non-nil error. So what\u2019s the purpose of this method returning an error as part of its signature? strings.Builder implements the io.StringWriter interface, which contains a single method: WriteString(s string) (n int, err error). Hence, to comply with this interface, WriteString must return an error.
Internally, strings.Builder holds a byte slice. Each call to WriteString results in a call to append on this slice. There are two impacts. First, this struct shouldn\u2019t be used concurrently, as the calls to append would lead to race conditions. The second impact is something that we saw in mistake #21, \"Inefficient slice initialization\": if the future length of a slice is already known, we should preallocate it. For that purpose, strings.Builder exposes a method Grow(n int) to guarantee space for another n bytes:
func concat(values []string) string {\ntotal := 0\nfor i := 0; i < len(values); i++ {\ntotal += len(values[i])\n}\nsb := strings.Builder{}\nsb.Grow(total) (2)\nfor _, value := range values {\n_, _ = sb.WriteString(value)\n}\nreturn sb.String()\n}\n
Let\u2019s run a benchmark to compare the three versions (v1 using +=; v2 using strings.Builder{} without preallocation; and v3 using strings.Builder{} with preallocation). The input slice contains 1,000 strings, and each string contains 1,000 bytes:
As we can see, the latest version is by far the most efficient: 99% faster than v1 and 78% faster than v2.
strings.Builder is the recommended solution to concatenate a list of strings. Usually, this solution should be used within a loop. Indeed, if we just have to concatenate a few strings (such as a name and a surname), using strings.Builder is not recommended as doing so will make the code a bit less readable than using the += operator or fmt.Sprintf.
Remembering that the bytes package offers the same operations as the strings package can help avoid extra byte/string conversions.
When choosing to work with a string or a []byte, most programmers tend to favor strings for convenience. But most I/O is actually done with []byte. For example, io.Reader, io.Writer, and io.ReadAll work with []byte, not strings.
When we\u2019re wondering whether we should work with strings or []byte, let\u2019s recall that working with []byte isn\u2019t necessarily less convenient. Indeed, all the exported functions of the strings package also have alternatives in the bytes package: Split, Count, Contains, Index, and so on. Hence, whether we\u2019re doing I/O or not, we should first check whether we could implement a whole workflow using bytes instead of strings and avoid the price of additional conversions.
Source code
"},{"location":"#substring-and-memory-leaks-41","title":"Substring and memory leaks (#41)","text":"TL;DR
Using copies instead of substrings can prevent memory leaks, as the string returned by a substring operation will be backed by the same byte array.
In mistake #26, \u201cSlices and memory leaks,\u201d we saw how slicing a slice or array may lead to memory leak situations. This principle also applies to string and substring operations.
We need to keep two things in mind while using the substring operation in Go. First, the interval provided is based on the number of bytes, not the number of runes. Second, a substring operation may lead to a memory leak as the resulting substring will share the same backing array as the initial string. The solutions to prevent this case from happening are to perform a string copy manually or to use strings.Clone from Go 1.18.
Source code
"},{"location":"#functions-and-methods","title":"Functions and Methods","text":""},{"location":"#not-knowing-which-type-of-receiver-to-use-42","title":"Not knowing which type of receiver to use (#42)","text":"TL;DR
The decision whether to use a value or a pointer receiver should be made based on factors such as the type, whether it has to be mutated, whether it contains a field that can\u2019t be copied, and how large the object is. When in doubt, use a pointer receiver.
Choosing between value and pointer receivers isn\u2019t always straightforward. Let\u2019s discuss some of the conditions to help us choose.
A receiver must be a pointer
If the method needs to mutate the receiver. This rule is also valid if the receiver is a slice and a method needs to append elements:
type slice []int\nfunc (s *slice) add(element int) {\n*s = append(*s, element)\n}\n
If the method receiver contains a field that cannot be copied: for example, a type part of the sync package (see #74, \u201cCopying a sync type\u201d).
A receiver should be a pointer
If the receiver is a large object. Using a pointer can make the call more efficient, as doing so prevents making an extensive copy. When in doubt about how large is large, benchmarking can be the solution; it\u2019s pretty much impossible to state a specific size, because it depends on many factors.
A receiver must be a value
If we have to enforce a receiver\u2019s immutability.
If the receiver is a map, function, or channel. Otherwise, a compilation error occurs.
A receiver should be a value
If the receiver is a slice that doesn\u2019t have to be mutated.
If the receiver is a small array or struct that is naturally a value type without mutable fields, such as time.Time.
If the receiver is a basic type such as int, float64, or string.
Of course, it\u2019s impossible to be exhaustive, as there will always be edge cases, but this section\u2019s goal was to provide guidance to cover most cases. By default, we can choose to go with a value receiver unless there\u2019s a good reason not to do so. In doubt, we should use a pointer receiver.
Source code
"},{"location":"#never-using-named-result-parameters-43","title":"Never using named result parameters (#43)","text":"TL;DR
Using named result parameters can be an efficient way to improve the readability of a function/method, especially if multiple result parameters have the same type. In some cases, this approach can also be convenient because named result parameters are initialized to their zero value. But be cautious about potential side effects.
When we return parameters in a function or a method, we can attach names to these parameters and use them as regular variables. When a result parameter is named, it\u2019s initialized to its zero value when the function/method begins. With named result parameters, we can also call a naked return statement (without arguments). In that case, the current values of the result parameters are used as the returned values.
Here\u2019s an example that uses a named result parameter b:
func f(a int) (b int) {\nb = a\nreturn\n}\n
In this example, we attach a name to the result parameter: b. When we call return without arguments, it returns the current value of b.
In some cases, named result parameters can also increase readability: for example, if two parameters have the same type. In other cases, they can also be used for convenience. Therefore, we should use named result parameters sparingly when there\u2019s a clear benefit.
Source code
"},{"location":"#unintended-side-effects-with-named-result-parameters-44","title":"Unintended side effects with named result parameters (#44)","text":"TL;DR
See #43.
We mentioned why named result parameters can be useful in some situations. But as these result parameters are initialized to their zero value, using them can sometimes lead to subtle bugs if we\u2019re not careful enough. For example, can you spot what\u2019s wrong with this code?
The error might not be obvious at first glance. Here, the error returned in the if ctx.Err() != nil scope is err. But we haven\u2019t assigned any value to the err variable. It\u2019s still assigned to the zero value of an error type: nil. Hence, this code will always return a nil error.
When using named result parameters, we must recall that each parameter is initialized to its zero value. As we have seen in this section, this can lead to subtle bugs that aren\u2019t always straightforward to spot while reading code. Therefore, let\u2019s remain cautious when using named result parameters, to avoid potential side effects.
Source code
"},{"location":"#returning-a-nil-receiver-45","title":"Returning a nil receiver (#45)","text":"TL;DR
When returning an interface, be cautious about returning not a nil pointer but an explicit nil value. Otherwise, unintended consequences may result because the caller will receive a non-nil value.
Source code
"},{"location":"#using-a-filename-as-a-function-input-46","title":"Using a filename as a function input (#46)","text":"TL;DR
Designing functions to receive io.Reader types instead of filenames improves the reusability of a function and makes testing easier.
Accepting a filename as a function input to read from a file should, in most cases, be considered a code smell (except in specific functions such as os.Open). Indeed, it makes unit tests more complex because we may have to create multiple files. It also reduces the reusability of a function (although not all functions are meant to be reused). Using the io.Reader interface abstracts the data source. Regardless of whether the input is a file, a string, an HTTP request, or a gRPC request, the implementation can be reused and easily tested.
Source code
"},{"location":"#ignoring-how-defer-arguments-and-receivers-are-evaluated-argument-evaluation-pointer-and-value-receivers-47","title":"Ignoring how defer arguments and receivers are evaluated (argument evaluation, pointer, and value receivers) (#47)","text":"TL;DR
Passing a pointer to a defer function and wrapping a call inside a closure are two possible solutions to overcome the immediate evaluation of arguments and receivers.
In a defer function the arguments are evaluated right away, not once the surrounding function returns. For example, in this code, we always call notify and incrementCounter with the same status: an empty string.
Indeed, we call notify(status) and incrementCounter(status) as defer functions. Therefore, Go will delay these calls to be executed once f returns with the current value of status at the stage we used defer, hence passing an empty string.
Two leading options if we want to keep using defer.
The first solution is to pass a string pointer:
func f() error {\nvar status string\ndefer notify(&status) defer incrementCounter(&status)\n// The rest of the function unchanged\n}\n
Using defer evaluates the arguments right away: here, the address of status. Yes, status itself is modified throughout the function, but its address remains constant, regardless of the assignments. Hence, if notify or incrementCounter uses the value referenced by the string pointer, it will work as expected. But this solution requires changing the signature of the two functions, which may not always be possible.
There\u2019s another solution: calling a closure (an anonymous function value that references variables from outside its body) as a defer statement:
func f() error {\nvar status string\ndefer func() {\nnotify(status)\nincrementCounter(status)\n}()\n// The rest of the function unchanged\n}\n
Here, we wrap the calls to both notify and incrementCounter within a closure. This closure references the status variable from outside its body. Therefore, status is evaluated once the closure is executed, not when we call defer. This solution also works and doesn\u2019t require notify and incrementCounter to change their signature.
Let's also note this behavior applies with method receiver: the receiver is evaluated immediately.
Using panic is an option to deal with errors in Go. However, it should only be used sparingly in unrecoverable conditions: for example, to signal a programmer error or when you fail to load a mandatory dependency.
In Go, panic is a built-in function that stops the ordinary flow:
Panicking in Go should be used sparingly. There are two prominent cases, one to signal a programmer error (e.g., sql.Register that panics if the driver is nil or has already been register) and another where our application fails to create a mandatory dependency. Hence, exceptional conditions that lead us to stop the application. In most other cases, error management should be done with a function that returns a proper error type as the last return argument.
Source code
"},{"location":"#ignoring-when-to-wrap-an-error-49","title":"Ignoring when to wrap an error (#49)","text":"TL;DR
Wrapping an error allows you to mark an error and/or provide additional context. However, error wrapping creates potential coupling as it makes the source error available for the caller. If you want to prevent that, don\u2019t use error wrapping.
Since Go 1.13, the %w directive allows us to wrap errors conveniently. Error wrapping is about wrapping or packing an error inside a wrapper container that also makes the source error available. In general, the two main use cases for error wrapping are the following:
Adding additional context to an error
Marking an error as a specific error
When handling an error, we can decide to wrap it. Wrapping is about adding additional context to an error and/or marking an error as a specific type. If we need to mark an error, we should create a custom error type. However, if we just want to add extra context, we should use fmt.Errorf with the %w directive as it doesn\u2019t require creating a new error type. Yet, error wrapping creates potential coupling as it makes the source error available for the caller. If we want to prevent it, we shouldn\u2019t use error wrapping but error transformation, for example, using fmt.Errorf with the %v directive.
Source code
"},{"location":"#comparing-an-error-type-inaccurately-50","title":"Comparing an error type inaccurately (#50)","text":"TL;DR
If you use Go 1.13 error wrapping with the %w directive and fmt.Errorf, comparing an error against a type has to be done using errors.As. Otherwise, if the returned error you want to check is wrapped, it will fail the checks.
Source code
"},{"location":"#comparing-an-error-value-inaccurately-51","title":"Comparing an error value inaccurately (#51)","text":"TL;DR
If you use Go 1.13 error wrapping with the %w directive and fmt.Errorf, comparing an error against or a value has to be done using errors.As. Otherwise, if the returned error you want to check is wrapped, it will fail the checks.
A sentinel error is an error defined as a global variable:
In general, the convention is to start with Err followed by the error type: here, ErrFoo. A sentinel error conveys an expected error, an error that clients will expect to check. As general guidelines:
Expected errors should be designed as error values (sentinel errors): var ErrFoo = errors.New(\"foo\").
Unexpected errors should be designed as error types: type BarError struct { ... }, with BarError implementing the error interface.
If we use error wrapping in our application with the %w directive and fmt.Errorf, checking an error against a specific value should be done using errors.Is instead of ==. Thus, even if the sentinel error is wrapped, errors.Is can recursively unwrap it and compare each error in the chain against the provided value.
Source code
"},{"location":"#handling-an-error-twice-52","title":"Handling an error twice (#52)","text":"TL;DR
In most situations, an error should be handled only once. Logging an error is handling an error. Therefore, you have to choose between logging or returning an error. In many cases, error wrapping is the solution as it allows you to provide additional context to an error and return the source error.
Handling an error multiple times is a mistake made frequently by developers, not specifically in Go. This can cause situations where the same error is logged multiple times make debugging harder.
Let's remind us that handling an error should be done only once. Logging an error is handling an error. Hence, we should either log or return an error. By doing this, we simplify our code and gain better insights into the error situation. Using error wrapping is the most convenient approach as it allows us to propagate the source error and add context to an error.
Source code
"},{"location":"#not-handling-an-error-53","title":"Not handling an error (#53)","text":"TL;DR
Ignoring an error, whether during a function call or in a defer function, should be done explicitly using the blank identifier. Otherwise, future readers may be confused about whether it was intentional or a miss.
In many cases, you shouldn\u2019t ignore an error returned by a defer function. Either handle it directly or propagate it to the caller, depending on the context. If you want to ignore it, use the blank identifier.
From a maintainability perspective, the code can lead to some issues. Let\u2019s consider a new reader looking at it. This reader notices that notify returns an error but that the error isn\u2019t handled by the parent function. How can they guess whether or not handling the error was intentional? How can they know whether the previous developer forgot to handle it or did it purposely?
For these reasons, when we want to ignore an error, there's only one way to do it, using the blank identifier (_):
_ = notify\n
In terms of compilation and run time, this approach doesn\u2019t change anything compared to the first piece of code. But this new version makes explicit that we aren\u2019t interested in the error. Also, we can add a comment that indicates the rationale for why an error is ignored:
// At-most once delivery.\n// Hence, it's accepted to miss some of them in case of errors.\n_ = notify()\n
Source code
"},{"location":"#concurrency-foundations","title":"Concurrency: Foundations","text":""},{"location":"#mixing-up-concurrency-and-parallelism-55","title":"Mixing up concurrency and parallelism (#55)","text":"TL;DR
Understanding the fundamental differences between concurrency and parallelism is a cornerstone of the Go developer\u2019s knowledge. Concurrency is about structure, whereas parallelism is about execution.
Concurrency and parallelism are not the same:
Concurrency is about structure. We can change a sequential implementation into a concurrent one by introducing different steps that separate concurrent goroutines can tackle.
Meanwhile, parallelism is about execution. We can use parallism at the steps level by adding more parallel goroutines.
In summary, concurrency provides a structure to solve a problem with parts that may be parallelized. Therefore, concurrency enables parallelism.
"},{"location":"#thinking-concurrency-is-always-faster-56","title":"Thinking concurrency is always faster (#56)","text":"TL;DR
To be a proficient developer, you must acknowledge that concurrency isn\u2019t always faster. Solutions involving parallelization of minimal workloads may not necessarily be faster than a sequential implementation. Benchmarking sequential versus concurrent solutions should be the way to validate assumptions.
Read the full section here.
Source code
"},{"location":"#being-puzzled-about-when-to-use-channels-or-mutexes-57","title":"Being puzzled about when to use channels or mutexes (#57)","text":"TL;DR
Being aware of goroutine interactions can also be helpful when deciding between channels and mutexes. In general, parallel goroutines require synchronization and hence mutexes. Conversely, concurrent goroutines generally require coordination and orchestration and hence channels.
Given a concurrency problem, it may not always be clear whether we can implement a solution using channels or mutexes. Because Go promotes sharing memory by communication, one mistake could be to always force the use of channels, regardless of the use case. However, we should see the two options as complementary.
When should we use channels or mutexes? We will use the example in the next figure as a backbone. Our example has three different goroutines with specific relationships:
G1 and G2 are parallel goroutines. They may be two goroutines executing the same function that keeps receiving messages from a channel, or perhaps two goroutines executing the same HTTP handler at the same time.
On the other hand, G1 and G3 are concurrent goroutines, as are G2 and G3. All the goroutines are part of an overall concurrent structure, but G1 and G2 perform the first step, whereas G3 does the next step.
In general, parallel goroutines have to synchronize: for example, when they need to access or mutate a shared resource such as a slice. Synchronization is enforced with mutexes but not with any channel types (not with buffered channels). Hence, in general, synchronization between parallel goroutines should be achieved via mutexes.
Conversely, in general, concurrent goroutines have to coordinate and orchestrate. For example, if G3 needs to aggregate results from both G1 and G2, G1 and G2 need to signal to G3 that a new intermediate result is available. This coordination falls under the scope of communication\u2014therefore, channels.
Regarding concurrent goroutines, there\u2019s also the case where we want to transfer the ownership of a resource from one step (G1 and G2) to another (G3); for example, if G1 and G2 are enriching a shared resource and at some point, we consider this job as complete. Here, we should use channels to signal that a specific resource is ready and handle the ownership transfer.
Mutexes and channels have different semantics. Whenever we want to share a state or access a shared resource, mutexes ensure exclusive access to this resource. Conversely, channels are a mechanic for signaling with or without data (chan struct{} or not). Coordination or ownership transfer should be achieved via channels. It\u2019s important to know whether goroutines are parallel or concurrent because, in general, we need mutexes for parallel goroutines and channels for concurrent ones.
"},{"location":"#not-understanding-race-problems-data-races-vs-race-conditions-and-the-go-memory-model-58","title":"Not understanding race problems (data races vs. race conditions and the Go memory model) (#58)","text":"TL;DR
Being proficient in concurrency also means understanding that data races and race conditions are different concepts. Data races occur when multiple goroutines simultaneously access the same memory location and at least one of them is writing. Meanwhile, being data-race-free doesn\u2019t necessarily mean deterministic execution. When a behavior depends on the sequence or the timing of events that can\u2019t be controlled, this is a race condition.
Race problems can be among the hardest and most insidious bugs a programmer can face. As Go developers, we must understand crucial aspects such as data races and race conditions, their possible impacts, and how to avoid them.
A data race occurs when two or more goroutines simultaneously access the same memory location and at least one is writing. In this case, the result can be hazardous. Even worse, in some situations, the memory location may end up holding a value containing a meaningless combination of bits.
We can prevent a data race from happening using different techniques. For example:
Using the sync/atomic package
In synchronizing the two goroutines with an ad how data structure like a mutex
Using channels to make the two goroutines communicating to ensure that a variable is updated by only one goroutine at a time
Depending on the operation we want to perform, does a data-race-free application necessarily mean a deterministic result? Not necessarily.
A race condition occurs when the behavior depends on the sequence or the timing of events that can\u2019t be controlled. Here, the timing of events is the goroutines\u2019 execution order.
In summary, when we work in concurrent applications, it\u2019s essential to understand that a data race is different from a race condition. A data race occurs when multiple goroutines simultaneously access the same memory location and at least one of them is writing. A data race means unexpected behavior. However, a data-race-free application doesn\u2019t necessarily mean deterministic results. An application can be free of data races but still have behavior that depends on uncontrolled events (such as goroutine execution, how fast a message is published to a channel, or how long a call to a database lasts); this is a race condition. Understanding both concepts is crucial to becoming proficient in designing concurrent applications.
Source code
"},{"location":"#not-understanding-the-concurrency-impacts-of-a-workload-type-59","title":"Not understanding the concurrency impacts of a workload type (#59)","text":"TL;DR
When creating a certain number of goroutines, consider the workload type. Creating CPU-bound goroutines means bounding this number close to the GOMAXPROCS variable (based by default on the number of CPU cores on the host). Creating I/O-bound goroutines depends on other factors, such as the external system.
In programming, the execution time of a workload is limited by one of the following:
The speed of the CPU\u2014For example, running a merge sort algorithm. The workload is called CPU-bound.
The speed of I/O\u2014For example, making a REST call or a database query. The workload is called I/O-bound.
The amount of available memory\u2014The workload is called memory-bound.
Note
The last is the rarest nowadays, given that memory has become very cheap in recent decades. Hence, this section focuses on the two first workload types: CPU- and I/O-bound.
If the workload executed by the workers is I/O-bound, the value mainly depends on the external system. Conversely, if the workload is CPU-bound, the optimal number of goroutines is close to the number of available CPU cores (a best practice can be to use runtime.GOMAXPROCS). Knowing the workload type (I/O or CPU) is crucial when designing concurrent applications.
Source code
"},{"location":"#misunderstanding-go-contexts-60","title":"Misunderstanding Go contexts (#60)","text":"TL;DR
Go contexts are also one of the cornerstones of concurrency in Go. A context allows you to carry a deadline, a cancellation signal, and/or a list of keys-values.
https://pkg.go.dev/context
A Context carries a deadline, a cancellation signal, and other values across API boundaries.
A deadline refers to a specific point in time determined with one of the following:
A time.Duration from now (for example, in 250 ms)
A time.Time (for example, 2023-02-07 00:00:00 UTC)
The semantics of a deadline convey that an ongoing activity should be stopped if this deadline is met. An activity is, for example, an I/O request or a goroutine waiting to receive a message from a channel.
Another use case for Go contexts is to carry a cancellation signal. Let\u2019s imagine that we want to create an application that calls CreateFileWatcher(ctx context.Context, filename string) within another goroutine. This function creates a specific file watcher that keeps reading from a file and catches updates. When the provided context expires or is canceled, this function handles it to close the file descriptor.
The last use case for Go contexts is to carry a key-value list. What\u2019s the point of having a context carrying a key-value list? Because Go contexts are generic and mainstream, there are infinite use cases.
For example, if we use tracing, we may want different subfunctions to share the same correlation ID. Some developers may consider this ID too invasive to be part of the function signature. In this regard, we could also decide to include it as part of the provided context.
"},{"location":"#catching-a-context-cancellation","title":"Catching a context cancellation","text":"
The context.Context type exports a Done method that returns a receive-only notification channel: <-chan struct{}. This channel is closed when the work associated with the context should be canceled. For example,
The Done channel related to a context created with context.WithCancel is closed when the cancel function is called.
The Done channel related to a context created with context.WithDeadline is closed when the deadline has expired.
One thing to note is that the internal channel should be closed when a context is canceled or has met a deadline, instead of when it receives a specific value, because the closure of a channel is the only channel action that all the consumer goroutines will receive. This way, all the consumers will be notified once a context is canceled or a deadline is reached.
In summary, to be a proficient Go developer, we have to understand what a context is and how to use it. In general, a function that users wait for should take a context, as doing so allows upstream callers to decide when calling this function should be aborted.
Source code
"},{"location":"#concurrency-practice","title":"Concurrency: Practice","text":""},{"location":"#propagating-an-inappropriate-context-61","title":"Propagating an inappropriate context (#61)","text":"TL;DR
Understanding the conditions when a context can be canceled should matter when propagating it: for example, an HTTP handler canceling the context when the response has been sent.
In many situations, it is recommended to propagate Go contexts. However, context propagation can sometimes lead to subtle bugs, preventing subfunctions from being correctly executed.
Let\u2019s consider the following example. We expose an HTTP handler that performs some tasks and returns a response. But just before returning the response, we also want to send it to a Kafka topic. We don\u2019t want to penalize the HTTP consumer latency-wise, so we want the publish action to be handled asynchronously within a new goroutine. We assume that we have at our disposal a publish function that accepts a context so the action of publishing a message can be interrupted if the context is canceled, for example. Here is a possible implementation:
func handler(w http.ResponseWriter, r *http.Request) {\nresponse, err := doSomeTask(r.Context(), r)\nif err != nil {\nhttp.Error(w, err.Error(), http.StatusInternalServerError)\nreturn\n}\ngo func() {\nerr := publish(r.Context(), response)\n// Do something with err\n}()\nwriteResponse(response)\n}\n
What\u2019s wrong with this piece of code? We have to know that the context attached to an HTTP request can cancel in different conditions:
When the client\u2019s connection closes
In the case of an HTTP/2 request, when the request is canceled
When the response has been written back to the client
In the first two cases, we probably handle things correctly. For example, if we get a response from doSomeTask but the client has closed the connection, it\u2019s probably OK to call publish with a context already canceled so the message isn\u2019t published. But what about the last case?
When the response has been written to the client, the context associated with the request will be canceled. Therefore, we are facing a race condition:
If the response is written after the Kafka publication, we both return a response and publish a message successfully
However, if the response is written before or during the Kafka publication, the message shouldn\u2019t be published.
In the latter case, calling publish will return an error because we returned the HTTP response quickly.
In summary, propagating a context should be done cautiously.
Source code
"},{"location":"#starting-a-goroutine-without-knowing-when-to-stop-it-62","title":"Starting a goroutine without knowing when to stop it (#62)","text":"TL;DR
Avoiding leaks means being mindful that whenever a goroutine is started, you should have a plan to stop it eventually.
Goroutines are easy and cheap to start\u2014so easy and cheap that we may not necessarily have a plan for when to stop a new goroutine, which can lead to leaks. Not knowing when to stop a goroutine is a design issue and a common concurrency mistake in Go.
Let\u2019s discuss a concrete example. We will design an application that needs to watch some external configuration (for example, using a database connection). Here\u2019s a first implementation:
func main() {\nnewWatcher()\n// Run the application\n}\ntype watcher struct { /* Some resources */ }\nfunc newWatcher() {\nw := watcher{}\ngo w.watch() // Creates a goroutine that watches some external configuration\n}\n
The problem with this code is that when the main goroutine exits (perhaps because of an OS signal or because it has a finite workload), the application is stopped. Hence, the resources created by watcher aren\u2019t closed gracefully. How can we prevent this from happening?
One option could be to pass to newWatcher a context that will be canceled when main returns:
func main() {\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nnewWatcher(ctx)\n// Run the application\n}\nfunc newWatcher(ctx context.Context) {\nw := watcher{}\ngo w.watch(ctx)\n}\n
We propagate the context created to the watch method. When the context is canceled, the watcher struct should close its resources. However, can we guarantee that watch will have time to do so? Absolutely not\u2014and that\u2019s a design flaw.
The problem is that we used signaling to convey that a goroutine had to be stopped. We didn\u2019t block the parent goroutine until the resources had been closed. Let\u2019s make sure we do:
func main() {\nw := newWatcher()\ndefer w.close()\n// Run the application\n}\nfunc newWatcher() watcher {\nw := watcher{}\ngo w.watch()\nreturn w\n}\nfunc (w watcher) close() {\n// Close the resources\n}\n
Instead of signaling watcher that it\u2019s time to close its resources, we now call this close method, using defer to guarantee that the resources are closed before the application exits.
In summary, let\u2019s be mindful that a goroutine is a resource like any other that must eventually be closed to free memory or other resources. Starting a goroutine without knowing when to stop it is a design issue. Whenever a goroutine is started, we should have a clear plan about when it will stop. Last but not least, if a goroutine creates resources and its lifetime is bound to the lifetime of the application, it\u2019s probably safer to wait for this goroutine to complete before exiting the application. This way, we can ensure that the resources can be freed.
Source code
"},{"location":"#not-being-careful-with-goroutines-and-loop-variables-63","title":"Not being careful with goroutines and loop variables (#63)","text":"Warning
This mistake isn't relevant anymore from Go 1.22 (source).
"},{"location":"#expecting-a-deterministic-behavior-using-select-and-channels-64","title":"Expecting a deterministic behavior using select and channels (#64)","text":"TL;DR
Understanding that select with multiple channels chooses the case randomly if multiple options are possible prevents making wrong assumptions that can lead to subtle concurrency bugs.
One common mistake made by Go developers while working with channels is to make wrong assumptions about how select behaves with multiple channels.
For example, let's consider the following case (disconnectCh is a buffered channel):
go func() {\nfor i := 0; i < 10; i++ {\nmessageCh <- i\n}\ndisconnectCh <- struct{}{}\n}()\nfor {\nselect {\ncase v := <-messageCh:\nfmt.Println(v)\ncase <-disconnectCh:\nfmt.Println(\"disconnection, return\")\nreturn\n}\n}\n
If we run this example multiple times, the result will be random:
Instead of consuming the 10 messages, we only received a few of them. What\u2019s the reason? It lies in the specification of the select statement with multiple channels (https:// go.dev/ref/spec):
Quote
If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.
Unlike a switch statement, where the first case with a match wins, the select state- ment selects randomly if multiple options are possible.
This behavior might look odd at first, but there\u2019s a good reason for it: to prevent possible starvation. Suppose the first possible communication chosen is based on the source order. In that case, we may fall into a situation where, for example, we only receive from one channel because of a fast sender. To prevent this, the language designers decided to use a random selection.
When using select with multiple channels, we must remember that if multiple options are possible, the first case in the source order does not automatically win. Instead, Go selects randomly, so there\u2019s no guarantee about which option will be chosen. To overcome this behavior, in the case of a single producer goroutine, we can use either unbuffered channels or a single channel.
Source code
"},{"location":"#not-using-notification-channels-65","title":"Not using notification channels (#65)","text":"TL;DR
Send notifications using a chan struct{} type.
Channels are a mechanism for communicating across goroutines via signaling. A signal can be either with or without data.
Let\u2019s look at a concrete example. We will create a channel that will notify us whenever a certain disconnection occurs. One idea is to handle it as a chan bool:
disconnectCh := make(chan bool)\n
Now, let\u2019s say we interact with an API that provides us with such a channel. Because it\u2019s a channel of Booleans, we can receive either true or false messages. It\u2019s probably clear what true conveys. But what does false mean? Does it mean we haven\u2019t been disconnected? And in this case, how frequently will we receive such a signal? Does it mean we have reconnected? Should we even expect to receive false? Perhaps we should only expect to receive true messages.
If that\u2019s the case, meaning we don\u2019t need a specific value to convey some information, we need a channel without data. The idiomatic way to handle it is a channel of empty structs: chan struct{}.
"},{"location":"#not-using-nil-channels-66","title":"Not using nil channels (#66)","text":"TL;DR
Using nil channels should be part of your concurrency toolset because it allows you to remove cases from select statements, for example.
What should this code do?
var ch chan int\n<-ch\n
ch is a chan int type. The zero value of a channel being nil, ch is nil. The goroutine won\u2019t panic; however, it will block forever.
The principle is the same if we send a message to a nil channel. This goroutine blocks forever:
var ch chan int\nch <- 0\n
Then what\u2019s the purpose of Go allowing messages to be received from or sent to a nil channel? For example, we can use nil channels to implement an idiomatic way to merge two channels:
func merge(ch1, ch2 <-chan int) <-chan int {\nch := make(chan int, 1)\ngo func() {\nfor ch1 != nil || ch2 != nil { // Continue if at least one channel isn\u2019t nil\nselect {\ncase v, open := <-ch1:\nif !open {\nch1 = nil // Assign ch1 to a nil channel once closed\nbreak\n}\nch <- v\ncase v, open := <-ch2:\nif !open {\nch2 = nil // Assigns ch2 to a nil channel once closed\nbreak\n}\nch <- v\n}\n}\nclose(ch)\n}()\nreturn ch\n}\n
This elegant solution relies on nil channels to somehow remove one case from the select statement.
Let\u2019s keep this idea in mind: nil channels are useful in some conditions and should be part of the Go developer\u2019s toolset when dealing with concurrent code.
Source code
"},{"location":"#being-puzzled-about-channel-size-67","title":"Being puzzled about channel size (#67)","text":"TL;DR
Carefully decide on the right channel type to use, given a problem. Only unbuffered channels provide strong synchronization guarantees.
You should have a good reason to specify a channel size other than one for buffered channels.
"},{"location":"#forgetting-about-possible-side-effects-with-string-formatting-etcd-data-race-example-and-deadlock-68","title":"Forgetting about possible side effects with string formatting (etcd data race example and deadlock) (#68)","text":"TL;DR
Being aware that string formatting may lead to calling existing functions means watching out for possible deadlocks and other data races.
Source code
"},{"location":"#creating-data-races-with-append-69","title":"Creating data races with append (#69)","text":"TL;DR
Calling append isn\u2019t always data-race-free; hence, it shouldn\u2019t be used concurrently on a shared slice.
Source code
"},{"location":"#using-mutexes-inaccurately-with-slices-and-maps-70","title":"Using mutexes inaccurately with slices and maps (#70)","text":"TL;DR
Remembering that slices and maps are pointers can prevent common data races.
To accurately use sync.WaitGroup, call the Add method before spinning up goroutines.
Source code
"},{"location":"#forgetting-about-synccond-72","title":"Forgetting about sync.Cond (#72)","text":"TL;DR
You can send repeated notifications to multiple goroutines with sync.Cond.
Source code
"},{"location":"#not-using-errgroup-73","title":"Not using errgroup (#73)","text":"TL;DR
You can synchronize a group of goroutines and handle errors and contexts with the errgroup package.
Source code
"},{"location":"#copying-a-sync-type-74","title":"Copying a sync type (#74)","text":"TL;DR
sync types shouldn\u2019t be copied.
Source code
"},{"location":"#standard-library","title":"Standard Library","text":""},{"location":"#providing-a-wrong-time-duration-75","title":"Providing a wrong time duration (#75)","text":"TL;DR
Remain cautious with functions accepting a time.Duration. Even though passing an integer is allowed, strive to use the time API to prevent any possible confusion.
Source code
"},{"location":"#timeafter-and-memory-leaks-76","title":"time.After and memory leaks (#76)","text":"TL;DR
Avoiding calls to time.After in repeated functions (such as loops or HTTP handlers) can avoid peak memory consumption. The resources created by time.After are released only when the timer expires.
Source code
"},{"location":"#json-handling-common-mistakes-77","title":"JSON handling common mistakes (#77)","text":"
Unexpected behavior because of type embedding
Be careful about using embedded fields in Go structs. Doing so may lead to sneaky bugs like an embedded time.Time field implementing the json.Marshaler interface, hence overriding the default marshaling behavior.
Source code
JSON and the monotonic clock
When comparing two time.Time structs, recall that time.Time contains both a wall clock and a monotonic clock, and the comparison using the == operator is done on both clocks.
Source code
Map of any
To avoid wrong assumptions when you provide a map while unmarshaling JSON data, remember that numerics are converted to float64 by default.
Forgetting that sql.Open doesn't necessarily establish connections to a database
Call the Ping or PingContext method if you need to test your configuration and make sure a database is reachable.
Source code
Forgetting about connections pooling
Configure the database connection parameters for production-grade applications.
Not using prepared statements
Using SQL prepared statements makes queries more efficient and more secure.
Source code
Mishandling null values
Deal with nullable columns in tables using pointers or sql.NullXXX types.
Source code
Not handling rows iteration errors
Call the Err method of sql.Rows after row iterations to ensure that you haven\u2019t missed an error while preparing the next row.
Source code
"},{"location":"#not-closing-transient-resources-http-body-sqlrows-and-osfile-79","title":"Not closing transient resources (HTTP body, sql.Rows, and os.File) (#79)","text":"TL;DR
Eventually close all structs implementing io.Closer to avoid possible leaks.
Source code
"},{"location":"#forgetting-the-return-statement-after-replying-to-an-http-request-80","title":"Forgetting the return statement after replying to an HTTP request (#80)","text":"TL;DR
To avoid unexpected behaviors in HTTP handler implementations, make sure you don\u2019t miss the return statement if you want a handler to stop after http.Error.
Source code
"},{"location":"#using-the-default-http-client-and-server-81","title":"Using the default HTTP client and server (#81)","text":"TL;DR
For production-grade applications, don\u2019t use the default HTTP client and server implementations. These implementations are missing timeouts and behaviors that should be mandatory in production.
Source code
"},{"location":"#testing","title":"Testing","text":""},{"location":"#not-categorizing-tests-build-tags-environment-variables-and-short-mode-82","title":"Not categorizing tests (build tags, environment variables, and short mode) (#82)","text":"TL;DR
Categorizing tests using build flags, environment variables, or short mode makes the testing process more efficient. You can create test categories using build flags or environment variables (for example, unit versus integration tests) and differentiate short from long-running tests to decide which kinds of tests to execute.
Source code
"},{"location":"#not-enabling-the-race-flag-83","title":"Not enabling the race flag (#83)","text":"TL;DR
Enabling the -race flag is highly recommended when writing concurrent applications. Doing so allows you to catch potential data races that can lead to software bugs.
"},{"location":"#not-using-test-execution-modes-parallel-and-shuffle-84","title":"Not using test execution modes (parallel and shuffle) (#84)","text":"TL;DR
Using the -parallel flag is an efficient way to speed up tests, especially long-running ones. Use the -shuffle flag to help ensure that a test suite doesn\u2019t rely on wrong assumptions that could hide bugs.
"},{"location":"#not-using-table-driven-tests-85","title":"Not using table-driven tests (#85)","text":"TL;DR
Table-driven tests are an efficient way to group a set of similar tests to prevent code duplication and make future updates easier to handle.
Source code
"},{"location":"#sleeping-in-unit-tests-86","title":"Sleeping in unit tests (#86)","text":"TL;DR
Avoid sleeps using synchronization to make a test less flaky and more robust. If synchronization isn\u2019t possible, consider a retry approach.
Source code
"},{"location":"#not-dealing-with-the-time-api-efficiently-87","title":"Not dealing with the time API efficiently (#87)","text":"TL;DR
Understanding how to deal with functions using the time API is another way to make a test less flaky. You can use standard techniques such as handling the time as part of a hidden dependency or asking clients to provide it.
Source code
"},{"location":"#not-using-testing-utility-packages-httptest-and-iotest-88","title":"Not using testing utility packages (httptest and iotest) (#88)","text":"
The httptest package is helpful for dealing with HTTP applications. It provides a set of utilities to test both clients and servers.
Source code
The iotest package helps write io.Reader and test that an application is tolerant to errors.
Use time methods to preserve the accuracy of a benchmark.
Increasing benchtime or using tools such as benchstat can be helpful when dealing with micro-benchmarks.
Be careful with the results of a micro-benchmark if the system that ends up running the application is different from the one running the micro-bench- mark.
Make sure the function under test leads to a side effect, to prevent compiler optimizations from fooling you about the benchmark results.
To prevent the observer effect, force a benchmark to re-create the data used by a CPU-bound function.
Read the full section here.
Source code
"},{"location":"#not-exploring-all-the-go-testing-features-90","title":"Not exploring all the Go testing features (#90)","text":"
Code coverage
Use code coverage with the -coverprofile flag to quickly see which part of the code needs more attention.
Testing from a different package
Place unit tests in a different package to enforce writing tests that focus on an exposed behavior, not internals.
Source code
Utility functions
Handling errors using the *testing.T variable instead of the classic if err != nil makes code shorter and easier to read.
Source code
Setup and teardown
You can use setup and teardown functions to configure a complex environment, such as in the case of integration tests.
Source code
"},{"location":"#not-using-fuzzing-community-mistake","title":"Not using fuzzing (community mistake)","text":"TL;DR
Fuzzing is an efficient strategy to detect random, unexpected, or malformed inputs to complex functions and methods in order to discover vulnerabilities, bugs, or even potential crashes.
Credits: @jeromedoucet
"},{"location":"#optimizations","title":"Optimizations","text":""},{"location":"#not-understanding-cpu-caches-91","title":"Not understanding CPU caches (#91)","text":"
CPU architecture
Understanding how to use CPU caches is important for optimizing CPU-bound applications because the L1 cache is about 50 to 100 times faster than the main memory.
Cache line
Being conscious of the cache line concept is critical to understanding how to organize data in data-intensive applications. A CPU doesn\u2019t fetch memory word by word; instead, it usually copies a memory block to a 64-byte cache line. To get the most out of each individual cache line, enforce spatial locality.
Source code
Slice of structs vs. struct of slices
Source code
Predictability
Making code predictable for the CPU can also be an efficient way to optimize certain functions. For example, a unit or constant stride is predictable for the CPU, but a non-unit stride (for example, a linked list) isn\u2019t predictable.
Source code
Cache placement policy
To avoid a critical stride, hence utilizing only a tiny portion of the cache, be aware that caches are partitioned.
"},{"location":"#writing-concurrent-code-that-leads-to-false-sharing-92","title":"Writing concurrent code that leads to false sharing (#92)","text":"TL;DR
Knowing that lower levels of CPU caches aren\u2019t shared across all the cores helps avoid performance-degrading patterns such as false sharing while writing concurrency code. Sharing memory is an illusion.
Source code
"},{"location":"#not-taking-into-account-instruction-level-parallelism-93","title":"Not taking into account instruction-level parallelism (#93)","text":"TL;DR
Use instruction-level parallelism (ILP) to optimize specific parts of your code to allow a CPU to execute as many parallel instructions as possible. Identifying data hazards is one of the main steps.
Source code
"},{"location":"#not-being-aware-of-data-alignment-94","title":"Not being aware of data alignment (#94)","text":"TL;DR
You can avoid common mistakes by remembering that in Go, basic types are aligned with their own size. For example, keep in mind that reorganizing the fields of a struct by size in descending order can lead to more compact structs (less memory allocation and potentially a better spatial locality).
Source code
"},{"location":"#not-understanding-stack-vs-heap-95","title":"Not understanding stack vs. heap (#95)","text":"TL;DR
Understanding the fundamental differences between heap and stack should also be part of your core knowledge when optimizing a Go application. Stack allocations are almost free, whereas heap allocations are slower and rely on the GC to clean the memory.
Source code
"},{"location":"#not-knowing-how-to-reduce-allocations-api-change-compiler-optimizations-and-syncpool-96","title":"Not knowing how to reduce allocations (API change, compiler optimizations, and sync.Pool) (#96)","text":"TL;DR
Reducing allocations is also an essential aspect of optimizing a Go application. This can be done in different ways, such as designing the API carefully to prevent sharing up, understanding the common Go compiler optimizations, and using sync.Pool.
Source code
"},{"location":"#not-relying-on-inlining-97","title":"Not relying on inlining (#97)","text":"TL;DR
Use the fast-path inlining technique to efficiently reduce the amortized time to call a function.
"},{"location":"#not-using-go-diagnostics-tooling-98","title":"Not using Go diagnostics tooling (#98)","text":"TL;DR
Rely on profiling and the execution tracer to understand how an application performs and the parts to optimize.
Read the full section here.
"},{"location":"#not-understanding-how-the-gc-works-99","title":"Not understanding how the GC works (#99)","text":"TL;DR
Understanding how to tune the GC can lead to multiple benefits such as handling sudden load increases more efficiently.
"},{"location":"#not-understanding-the-impacts-of-running-go-in-docker-and-kubernetes-100","title":"Not understanding the impacts of running Go in Docker and Kubernetes (#100)","text":"TL;DR
To help avoid CPU throttling when deployed in Docker and Kubernetes, keep in mind that Go isn\u2019t CFS-aware.
"},{"location":"20-slice/","title":"Not understanding slice length and capacity","text":"
It\u2019s pretty common for Go developers to mix slice length and capacity or not understand them thoroughly. Assimilating these two concepts is essential for efficiently handling core operations such as slice initialization and adding elements with append, copying, or slicing. This misunderstanding can lead to using slices suboptimally or even to memory leaks.
In Go, a slice is backed by an array. That means the slice\u2019s data is stored contiguously in an array data structure. A slice also handles the logic of adding an element if the backing array is full or shrinking the backing array if it\u2019s almost empty.
Internally, a slice holds a pointer to the backing array plus a length and a capacity. The length is the number of elements the slice contains, whereas the capacity is the number of elements in the backing array, counting from the first element in the slice. Let\u2019s go through a few examples to make things clearer. First, let\u2019s initialize a slice with a given length and capacity:
s := make([]int, 3, 6) // Three-length, six-capacity slice\n
The first argument, representing the length, is mandatory. However, the second argument representing the capacity is optional. Figure 1 shows the result of this code in memory.
Figure 1: A three-length, six-capacity slice.
In this case, make creates an array of six elements (the capacity). But because the length was set to 3, Go initializes only the first three elements. Also, because the slice is an []int type, the first three elements are initialized to the zeroed value of an int: 0. The grayed elements are allocated but not yet used.
If we print this slice, we get the elements within the range of the length, [0 0 0]. If we set s[1] to 1, the second element of the slice updates without impacting its length or capacity. Figure 2 illustrates this.
Figure 2: Updating the slice\u2019s second element: s[1] = 1.
However, accessing an element outside the length range is forbidden, even though it\u2019s already allocated in memory. For example, s[4] = 0 would lead to the following panic:
panic: runtime error: index out of range [4] with length 3\n
How can we use the remaining space of the slice? By using the append built-in function:
s = append(s, 2)\n
This code appends to the existing s slice a new element. It uses the first grayed element (which was allocated but not yet used) to store element 2, as figure 3 shows.
Figure 3: Appending an element to s.
The length of the slice is updated from 3 to 4 because the slice now contains four elements. Now, what happens if we add three more elements so that the backing array isn\u2019t large enough?
s = append(s, 3, 4, 5)\nfmt.Println(s)\n
If we run this code, we see that the slice was able to cope with our request:
[0 1 0 2 3 4 5]\n
Because an array is a fixed-size structure, it can store the new elements until element 4. When we want to insert element 5, the array is already full: Go internally creates another array by doubling the capacity, copying all the elements, and then inserting element 5. Figure 4 shows this process.
Figure 4: Because the initial backing array is full, Go creates another array and copies all the elements.
The slice now references the new backing array. What will happen to the previous backing array? If it\u2019s no longer referenced, it\u2019s eventually freed by the garbage collector (GC) if allocated on the heap. (We discuss heap memory in mistake #95, \u201cNot understanding stack vs. heap,\u201d and we look at how the GC works in mistake #99, \u201cNot understanding how the GC works.\u201d)
What happens with slicing? Slicing is an operation done on an array or a slice, providing a half-open range; the first index is included, whereas the second is excluded. The following example shows the impact, and figure 5 displays the result in memory:
s1 := make([]int, 3, 6) // Three-length, six-capacity slice\ns2 := s1[1:3] // Slicing from indices 1 to 3\n
Figure 5: The slices s1 and s2 reference the same backing array with different lengths and capacities.
First, s1 is created as a three-length, six-capacity slice. When s2 is created by slicing s1, both slices reference the same backing array. However, s2 starts from a different index, 1. Therefore, its length and capacity (a two-length, five-capacity slice) differ from s1. If we update s1[1] or s2[0], the change is made to the same array, hence, visible in both slices, as figure 6 shows.
Figure 6: Because s1 and s2 are backed by the same array, updating a common element makes the change visible in both slices.
Now, what happens if we append an element to s2? Does the following code change s1 as well?
s2 = append(s2, 2)\n
The shared backing array is modified, but only the length of s2 changes. Figure 7 shows the result of appending an element to s2.
Figure 7: Appending an element to s2.
s1 remains a three-length, six-capacity slice. Therefore, if we print s1 and s2, the added element is only visible for s2:
s1=[0 1 0], s2=[1 0 2]\n
It\u2019s important to understand this behavior so that we don\u2019t make wrong assumptions while using append.
Note
In these examples, the backing array is internal and not available directly to the Go developer. The only exception is when a slice is created from slicing an existing array.
One last thing to note: what if we keep appending elements to s2 until the backing array is full? What will the state be, memory-wise? Let\u2019s add three more elements so that the backing array will not have enough capacity:
s2 = append(s2, 3)\ns2 = append(s2, 4) // At this stage, the backing is already full\ns2 = append(s2, 5)\n
This code leads to creating another backing array. Figure 8 displays the results in memory.
Figure 8: Appending elements to s2 until the backing array is full.
s1 and s2 now reference two different arrays. As s1 is still a three-length, six-capacity slice, it still has some available buffer, so it keeps referencing the initial array. Also, the new backing array was made by copying the initial one from the first index of s2. That\u2019s why the new array starts with element 1, not 0.
To summarize, the slice length is the number of available elements in the slice, whereas the slice capacity is the number of elements in the backing array. Adding an element to a full slice (length == capacity) leads to creating a new backing array with a new capacity, copying all the elements from the previous array, and updating the slice pointer to the new array.
"},{"location":"28-maps-memory-leaks/","title":"Maps and memory leaks","text":"
When working with maps in Go, we need to understand some important characteristics of how a map grows and shrinks. Let\u2019s delve into this to prevent issues that can cause memory leaks.
First, to view a concrete example of this problem, let\u2019s design a scenario where we will work with the following map:
m := make(map[int][128]byte)\n
Each value of m is an array of 128 bytes. We will do the following:
Allocate an empty map.
Add 1 million elements.
Remove all the elements, and run a Garbage Collection (GC).
After each step, we want to print the size of the heap (using a printAlloc utility function). This shows us how this example behaves memory-wise:
func main() {\nn := 1_000_000\nm := make(map[int][128]byte)\nprintAlloc()\nfor i := 0; i < n; i++ { // Adds 1 million elements\nm[i] = [128]byte{}\n}\nprintAlloc()\nfor i := 0; i < n; i++ { // Deletes 1 million elements\ndelete(m, i)\n}\nruntime.GC() // Triggers a manual GC\nprintAlloc()\nruntime.KeepAlive(m) // Keeps a reference to m so that the map isn\u2019t collected\n}\nfunc printAlloc() {\nvar m runtime.MemStats\nruntime.ReadMemStats(&m)\nfmt.Printf(\"%d KB\\n\", m.Alloc/1024)\n}\n
We allocate an empty map, add 1 million elements, remove 1 million elements, and then run a GC. We also make sure to keep a reference to the map using runtime.KeepAlive so that the map isn\u2019t collected as well. Let\u2019s run this example:
0 MB <-- After m is allocated\n461 MB <-- After we add 1 million elements\n293 MB <-- After we remove 1 million elements\n
What can we observe? At first, the heap size is minimal. Then it grows significantly after having added 1 million elements to the map. But if we expected the heap size to decrease after removing all the elements, this isn\u2019t how maps work in Go. In the end, even though the GC has collected all the elements, the heap size is still 293 MB. So the memory shrunk, but not as we might have expected. What\u2019s the rationale? We need to delve into how a map works in Go.
A map provides an unordered collection of key-value pairs in which all the keys are distinct. In Go, a map is based on the hash table data structure: an array where each element is a pointer to a bucket of key-value pairs, as shown in figure 1.
Figure 1: A hash table example with a focus on bucket 0.
Each bucket is a fixed-size array of eight elements. In the case of an insertion into a bucket that is already full (a bucket overflow), Go creates another bucket of eight elements and links the previous one to it. Figure 2 shows an example:
Figure 2: In case of a bucket overflow, Go allocates a new bucket and links the previous bucket to it.
Under the hood, a Go map is a pointer to a runtime.hmap struct. This struct contains multiple fields, including a B field, giving the number of buckets in the map:
type hmap struct {\nB uint8 // log_2 of # of buckets\n// (can hold up to loadFactor * 2^B items)\n// ...\n}\n
After adding 1 million elements, the value of B equals 18, which means 2\u00b9\u2078 = 262,144 buckets. When we remove 1 million elements, what\u2019s the value of B? Still 18. Hence, the map still contains the same number of buckets.
The reason is that the number of buckets in a map cannot shrink. Therefore, removing elements from a map doesn\u2019t impact the number of existing buckets; it just zeroes the slots in the buckets. A map can only grow and have more buckets; it never shrinks.
In the previous example, we went from 461 MB to 293 MB because the elements were collected, but running the GC didn\u2019t impact the map itself. Even the number of extra buckets (the buckets created because of overflows) remains the same.
Let\u2019s take a step back and discuss when the fact that a map cannot shrink can be a problem. Imagine building a cache using a map[int][128]byte. This map holds per customer ID (the int), a sequence of 128 bytes. Now, suppose we want to save the last 1,000 customers. The map size will remain constant, so we shouldn\u2019t worry about the fact that a map cannot shrink.
However, let\u2019s say we want to store one hour of data. Meanwhile, our company has decided to have a big promotion for Black Friday: in one hour, we may have millions of customers connected to our system. But a few days after Black Friday, our map will contain the same number of buckets as during the peak time. This explains why we can experience high memory consumption that doesn\u2019t significantly decrease in such a scenario.
What are the solutions if we don\u2019t want to manually restart our service to clean the amount of memory consumed by the map? One solution could be to re-create a copy of the current map at a regular pace. For example, every hour, we can build a new map, copy all the elements, and release the previous one. The main drawback of this option is that following the copy and until the next garbage collection, we may consume twice the current memory for a short period.
Another solution would be to change the map type to store an array pointer: map[int]*[128]byte. It doesn\u2019t solve the fact that we will have a significant number of buckets; however, each bucket entry will reserve the size of a pointer for the value instead of 128 bytes (8 bytes on 64-bit systems and 4 bytes on 32-bit systems).
Coming back to the original scenario, let\u2019s compare the memory consumption for each map type following each step. The following table shows the comparison.
Step map[int][128]bytemap[int]*[128]byte Allocate an empty map 0 MB 0 MB Add 1 million elements 461 MB 182 MB Remove all the elements and run a GC 293 MB 38 MB Note
If a key or a value is over 128 bytes, Go won\u2019t store it directly in the map bucket. Instead, Go stores a pointer to reference the key or the value.
As we have seen, adding n elements to a map and then deleting all the elements means keeping the same number of buckets in memory. So, we must remember that because a Go map can only grow in size, so does its memory consumption. There is no automated strategy to shrink it. If this leads to high memory consumption, we can try different options such as forcing Go to re-create the map or using pointers to check if it can be optimized.
"},{"location":"56-concurrency-faster/","title":"Thinking concurrency is always faster","text":"
A misconception among many developers is believing that a concurrent solution is always faster than a sequential one. This couldn\u2019t be more wrong. The overall performance of a solution depends on many factors, such as the efficiency of our code structure (concurrency), which parts can be tackled in parallel, and the level of contention among the computation units. This post reminds us about some fundamental knowledge of concurrency in Go; then we will see a concrete example where a concurrent solution isn\u2019t necessarily faster.
A thread is the smallest unit of processing that an OS can perform. If a process wants to execute multiple actions simultaneously, it spins up multiple threads. These threads can be:
Concurrent \u2014 Two or more threads can start, run, and complete in overlapping time periods.
Parallel \u2014 The same task can be executed multiple times at once.
The OS is responsible for scheduling the thread\u2019s processes optimally so that:
All the threads can consume CPU cycles without being starved for too much time.
The workload is distributed as evenly as possible among the different CPU cores.
Note
The word thread can also have a different meaning at a CPU level. Each physical core can be composed of multiple logical cores (the concept of hyper-threading), and a logical core is also called a thread. In this post, when we use the word thread, we mean the unit of processing, not a logical core.
A CPU core executes different threads. When it switches from one thread to another, it executes an operation called context switching. The active thread consuming CPU cycles was in an executing state and moves to a runnable state, meaning it\u2019s ready to be executed pending an available core. Context switching is considered an expensive operation because the OS needs to save the current execution state of a thread before the switch (such as the current register values).
As Go developers, we can\u2019t create threads directly, but we can create goroutines, which can be thought of as application-level threads. However, whereas an OS thread is context-switched on and off a CPU core by the OS, a goroutine is context-switched on and off an OS thread by the Go runtime. Also, compared to an OS thread, a goroutine has a smaller memory footprint: 2 KB for goroutines from Go 1.4. An OS thread depends on the OS, but, for example, on Linux/x86\u201332, the default size is 2 MB (see https://man7.org/linux/man-pages/man3/pthread_create.3.html). Having a smaller size makes context switching faster.
Note
Context switching a goroutine versus a thread is about 80% to 90% faster, depending on the architecture.
Let\u2019s now discuss how the Go scheduler works to overview how goroutines are handled. Internally, the Go scheduler uses the following terminology (see proc.go):
G \u2014 Goroutine
M \u2014 OS thread (stands for machine)
P \u2014 CPU core (stands for processor)
Each OS thread (M) is assigned to a CPU core (P) by the OS scheduler. Then, each goroutine (G) runs on an M. The GOMAXPROCS variable defines the limit of Ms in charge of executing user-level code simultaneously. But if a thread is blocked in a system call (for example, I/O), the scheduler can spin up more Ms. As of Go 1.5, GOMAXPROCS is by default equal to the number of available CPU cores.
A goroutine has a simpler lifecycle than an OS thread. It can be doing one of the following:
Executing \u2014 The goroutine is scheduled on an M and executing its instructions.
Runnable \u2014 The goroutine is waiting to be in an executing state.
Waiting \u2014 The goroutine is stopped and pending something completing, such as a system call or a synchronization operation (such as acquiring a mutex).
There\u2019s one last stage to understand about the implementation of Go scheduling: when a goroutine is created but cannot be executed yet; for example, all the other Ms are already executing a G. In this scenario, what will the Go runtime do about it? The answer is queuing. The Go runtime handles two kinds of queues: one local queue per P and a global queue shared among all the Ps.
Figure 1 shows a given scheduling situation on a four-core machine with GOMAXPROCS equal to 4. The parts are the logical cores (Ps), goroutines (Gs), OS threads (Ms), local queues, and global queue:
Figure 1: An example of the current state of a Go application executed on a four-core machine. Goroutines that aren\u2019t in an executing state are either runnable (pending being executed) or waiting (pending a blocking operation)
First, we can see five Ms, whereas GOMAXPROCS is set to 4. But as we mentioned, if needed, the Go runtime can create more OS threads than the GOMAXPROCS value.
P0, P1, and P3 are currently busy executing Go runtime threads. But P2 is presently idle as M3 is switched off P2, and there\u2019s no goroutine to be executed. This isn\u2019t a good situation because six runnable goroutines are pending being executed, some in the global queue and some in other local queues. How will the Go runtime handle this situation? Here\u2019s the scheduling implementation in pseudocode (see proc.go):
runtime.schedule() {\n // Only 1/61 of the time, check the global runnable queue for a G.\n // If not found, check the local queue.\n // If not found,\n // Try to steal from other Ps.\n // If not, check the global runnable queue.\n // If not found, poll network.\n}\n
Every sixty-first execution, the Go scheduler will check whether goroutines from the global queue are available. If not, it will check its local queue. Meanwhile, if both the global and local queues are empty, the Go scheduler can pick up goroutines from other local queues. This principle in scheduling is called work stealing, and it allows an underutilized processor to actively look for another processor\u2019s goroutines and steal some.
One last important thing to mention: prior to Go 1.14, the scheduler was cooperative, which meant a goroutine could be context-switched off a thread only in specific blocking cases (for example, channel send or receive, I/O, waiting to acquire a mutex). Since Go 1.14, the Go scheduler is now preemptive: when a goroutine is running for a specific amount of time (10 ms), it will be marked preemptible and can be context-switched off to be replaced by another goroutine. This allows a long-running job to be forced to share CPU time.
Now that we understand the fundamentals of scheduling in Go, let\u2019s look at a concrete example: implementing a merge sort in a parallel manner.
First, let\u2019s briefly review how the merge sort algorithm works. Then we will implement a parallel version. Note that the objective isn\u2019t to implement the most efficient version but to support a concrete example showing why concurrency isn\u2019t always faster.
The merge sort algorithm works by breaking a list repeatedly into two sublists until each sublist consists of a single element and then merging these sublists so that the result is a sorted list (see figure 2). Each split operation splits the list into two sublists, whereas the merge operation merges two sublists into a sorted list.
Figure 2: Applying the merge sort algorithm repeatedly breaks each list into two sublists. Then the algorithm uses a merge operation such that the resulting list is sorted
Here is the sequential implementation of this algorithm. We don\u2019t include all of the code as it\u2019s not the main point of this section:
func sequentialMergesort(s []int) {\nif len(s) <= 1 {\nreturn\n}\nmiddle := len(s) / 2\nsequentialMergesort(s[:middle]) // First half\nsequentialMergesort(s[middle:]) // Second half\nmerge(s, middle) // Merges the two halves\n}\nfunc merge(s []int, middle int) {\n// ...\n}\n
This algorithm has a structure that makes it open to concurrency. Indeed, as each sequentialMergesort operation works on an independent set of data that doesn\u2019t need to be fully copied (here, an independent view of the underlying array using slicing), we could distribute this workload among the CPU cores by spinning up each sequentialMergesort operation in a different goroutine. Let\u2019s write a first parallel implementation:
func parallelMergesortV1(s []int) {\nif len(s) <= 1 {\nreturn\n}\nmiddle := len(s) / 2\nvar wg sync.WaitGroup\nwg.Add(2)\ngo func() { // Spins up the first half of the work in a goroutine\ndefer wg.Done()\nparallelMergesortV1(s[:middle])\n}()\ngo func() { // Spins up the second half of the work in a goroutine\ndefer wg.Done()\nparallelMergesortV1(s[middle:])\n}()\nwg.Wait()\nmerge(s, middle) // Merges the halves\n}\n
In this version, each half of the workload is handled in a separate goroutine. The parent goroutine waits for both parts by using sync.WaitGroup. Hence, we call the Wait method before the merge operation.
We now have a parallel version of the merge sort algorithm. Therefore, if we run a benchmark to compare this version against the sequential one, the parallel version should be faster, correct? Let\u2019s run it on a four-core machine with 10,000 elements:
Surprisingly, the parallel version is almost an order of magnitude slower. How can we explain this result? How is it possible that a parallel version that distributes a workload across four cores is slower than a sequential version running on a single machine? Let\u2019s analyze the problem.
If we have a slice of, say, 1,024 elements, the parent goroutine will spin up two goroutines, each in charge of handling a half consisting of 512 elements. Each of these goroutines will spin up two new goroutines in charge of handling 256 elements, then 128, and so on, until we spin up a goroutine to compute a single element.
If the workload that we want to parallelize is too small, meaning we\u2019re going to compute it too fast, the benefit of distributing a job across cores is destroyed: the time it takes to create a goroutine and have the scheduler execute it is much too high compared to directly merging a tiny number of items in the current goroutine. Although goroutines are lightweight and faster to start than threads, we can still face cases where a workload is too small.
So what can we conclude from this result? Does it mean the merge sort algorithm cannot be parallelized? Wait, not so fast.
Let\u2019s try another approach. Because merging a tiny number of elements within a new goroutine isn\u2019t efficient, let\u2019s define a threshold. This threshold will represent how many elements a half should contain in order to be handled in a parallel manner. If the number of elements in the half is fewer than this value, we will handle it sequentially. Here\u2019s a new version:
const max = 2048 // Defines the threshold\nfunc parallelMergesortV2(s []int) {\nif len(s) <= 1 {\nreturn\n}\nif len(s) <= max {\nsequentialMergesort(s) // Calls our initial sequential version\n} else { // If bigger than the threshold, keeps the parallel version\nmiddle := len(s) / 2\nvar wg sync.WaitGroup\nwg.Add(2)\ngo func() {\ndefer wg.Done()\nparallelMergesortV2(s[:middle])\n}()\ngo func() {\ndefer wg.Done()\nparallelMergesortV2(s[middle:])\n}()\nwg.Wait()\nmerge(s, middle)\n}\n}\n
If the number of elements in the s slice is smaller than max, we call the sequential version. Otherwise, we keep calling our parallel implementation. Does this approach impact the result? Yes, it does:
Our v2 parallel implementation is more than 40% faster than the sequential one, thanks to this idea of defining a threshold to indicate when parallel should be more efficient than sequential.
Note
Why did I set the threshold to 2,048? Because it was the optimal value for this specific workload on my machine. In general, such magic values should be defined carefully with benchmarks (running on an execution environment similar to production). It\u2019s also pretty interesting to note that running the same algorithm in a programming language that doesn\u2019t implement the concept of goroutines has an impact on the value. For example, running the same example in Java using threads means an optimal value closer to 8,192. This tends to illustrate how goroutines are more efficient than threads.
We have seen throughout this post the fundamental concepts of scheduling in Go: the differences between a thread and a goroutine and how the Go runtime schedules goroutines. Meanwhile, using the parallel merge sort example, we illustrated that concurrency isn\u2019t always necessarily faster. As we have seen, spinning up goroutines to handle minimal workloads (merging only a small set of elements) demolishes the benefit we could get from parallelism.
So, where should we go from here? We must keep in mind that concurrency isn\u2019t always faster and shouldn\u2019t be considered the default way to go for all problems. First, it makes things more complex. Also, modern CPUs have become incredibly efficient at executing sequential code and predictable code. For example, a superscalar processor can parallelize instruction execution over a single core with high efficiency.
Does this mean we shouldn\u2019t use concurrency? Of course not. However, it\u2019s essential to keep these conclusions in mind. If we\u2019re not sure that a parallel version will be faster, the right approach may be to start with a simple sequential version and build from there using profiling (mistake #98, \u201cNot using Go diagnostics tooling\u201d) and benchmarks (mistake #89, \u201cWriting inaccurate benchmarks\u201d), for example. It can be the only way to ensure that a concurrent implementation is worth it.
In general, we should never guess about performance. When writing optimizations, so many factors may come into play that even if we have a strong opinion about the results, it\u2019s rarely a bad idea to test them. However, writing benchmarks isn\u2019t straightforward. It can be pretty simple to write inaccurate benchmarks and make wrong assumptions based on them. The goal of this post is to examine four common and concrete traps leading to inaccuracy:
Before discussing these traps, let\u2019s briefly review how benchmarks work in Go. The skeleton of a benchmark is as follows:
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nfoo()\n}\n}\n
The function name starts with the Benchmark prefix. The function under test (foo) is called within the for loop. b.N represents a variable number of iterations. When running a benchmark, Go tries to make it match the requested benchmark time. The benchmark time is set by default to 1 second and can be changed with the -benchtime flag. b.N starts at 1; if the benchmark completes in under 1 second, b.N is increased, and the benchmark runs again until b.N roughly matches benchtime:
$ go test -bench=.\ncpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkFoo-4 73 16511228 ns/op\n
Here, the benchmark took about 1 second, and foo was executed 73 times, for an average execution time of 16,511,228 nanoseconds. We can change the benchmark time using -benchtime:
$ go test -bench=. -benchtime=2s\nBenchmarkFoo-4 150 15832169 ns/op\n
foo was executed roughly twice more than during the previous benchmark.
Next, let\u2019s look at some common traps.
"},{"location":"89-benchmarks/#not-resetting-or-pausing-the-timer","title":"Not resetting or pausing the timer","text":"
In some cases, we need to perform operations before the benchmark loop. These operations may take quite a while (for example, generating a large slice of data) and may significantly impact the benchmark results:
func BenchmarkFoo(b *testing.B) {\nexpensiveSetup()\nfor i := 0; i < b.N; i++ {\nfunctionUnderTest()\n}\n}\n
In this case, we can use the ResetTimer method before entering the loop:
func BenchmarkFoo(b *testing.B) {\nexpensiveSetup()\nb.ResetTimer() // Reset the benchmark timer\nfor i := 0; i < b.N; i++ {\nfunctionUnderTest()\n}\n}\n
Calling ResetTimer zeroes the elapsed benchmark time and memory allocation counters since the beginning of the test. This way, an expensive setup can be discarded from the test results.
What if we have to perform an expensive setup not just once but within each loop iteration?
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nexpensiveSetup()\nfunctionUnderTest()\n}\n}\n
We can\u2019t reset the timer, because that would be executed during each loop iteration. But we can stop and resume the benchmark timer, surrounding the call to expensiveSetup:
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nb.StopTimer() // Pause the benchmark timer\nexpensiveSetup()\nb.StartTimer() // Resume the benchmark timer\nfunctionUnderTest()\n}\n}\n
Here, we pause the benchmark timer to perform the expensive setup and then resume the timer.
Note
There\u2019s one catch to remember about this approach: if the function under test is too fast to execute compared to the setup function, the benchmark may take too long to complete. The reason is that it would take much longer than 1 second to reach benchtime. Calculating the benchmark time is based solely on the execution time of functionUnderTest. So, if we wait a significant time in each loop iteration, the benchmark will be much slower than 1 second. If we want to keep the benchmark, one possible mitigation is to decrease benchtime.
We must be sure to use the timer methods to preserve the accuracy of a benchmark.
"},{"location":"89-benchmarks/#making-wrong-assumptions-about-micro-benchmarks","title":"Making wrong assumptions about micro-benchmarks","text":"
A micro-benchmark measures a tiny computation unit, and it can be extremely easy to make wrong assumptions about it. Let\u2019s say, for example, that we aren\u2019t sure whether to use atomic.StoreInt32 or atomic.StoreInt64 (assuming that the values we handle will always fit in 32 bits). We want to write a benchmark to compare both functions:
func BenchmarkAtomicStoreInt32(b *testing.B) {\nvar v int32\nfor i := 0; i < b.N; i++ {\natomic.StoreInt32(&v, 1)\n}\n}\nfunc BenchmarkAtomicStoreInt64(b *testing.B) {\nvar v int64\nfor i := 0; i < b.N; i++ {\natomic.StoreInt64(&v, 1)\n}\n}\n
If we run this benchmark, here\u2019s some example output:
We could easily take this benchmark for granted and decide to use atomic.StoreInt64 because it appears to be faster. Now, for the sake of doing a fair benchmark, we reverse the order and test atomic.StoreInt64 first, followed by atomic.StoreInt32. Here is some example output:
This time, atomic.StoreInt32 has better results. What happened?
In the case of micro-benchmarks, many factors can impact the results, such as machine activity while running the benchmarks, power management, thermal scaling, and better cache alignment of a sequence of instructions. We must remember that many factors, even outside the scope of our Go project, can impact the results.
Note
We should make sure the machine executing the benchmark is idle. However, external processes may run in the background, which may affect benchmark results. For that reason, tools such as perflock can limit how much CPU a benchmark can consume. For example, we can run a benchmark with 70% of the total available CPU, giving 30% to the OS and other processes and reducing the impact of the machine activity factor on the results.
One option is to increase the benchmark time using the -benchtime option. Similar to the law of large numbers in probability theory, if we run a benchmark a large number of times, it should tend to approach its expected value (assuming we omit the benefits of instructions caching and similar mechanics).
Another option is to use external tools on top of the classic benchmark tooling. For instance, the benchstat tool, which is part of the golang.org/x repository, allows us to compute and compare statistics about benchmark executions.
Let\u2019s run the benchmark 10 times using the -count option and pipe the output to a specific file:
$ go test -bench=. -count=10 | tee stats.txt\ncpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkAtomicStoreInt32-4 234935682 5.124 ns/op\nBenchmarkAtomicStoreInt32-4 235307204 5.112 ns/op\n// ...\nBenchmarkAtomicStoreInt64-4 235548591 5.107 ns/op\nBenchmarkAtomicStoreInt64-4 235210292 5.090 ns/op\n// ...\n
The results are the same: both functions take on average 5.10 nanoseconds to complete. We also see the percent variation between the executions of a given benchmark: \u00b1 1%. This metric tells us that both benchmarks are stable, giving us more confidence in the computed average results. Therefore, instead of concluding that atomic.StoreInt32 is faster or slower, we can conclude that its execution time is similar to that of atomic.StoreInt64 for the usage we tested (in a specific Go version on a particular machine).
In general, we should be cautious about micro-benchmarks. Many factors can significantly impact the results and potentially lead to wrong assumptions. Increasing the benchmark time or repeating the benchmark executions and computing stats with tools such as benchstat can be an efficient way to limit external factors and get more accurate results, leading to better conclusions.
Let\u2019s also highlight that we should be careful about using the results of a micro-benchmark executed on a given machine if another system ends up running the application. The production system may act quite differently from the one on which we ran the micro-benchmark.
"},{"location":"89-benchmarks/#not-being-careful-about-compiler-optimizations","title":"Not being careful about compiler optimizations","text":"
Another common mistake related to writing benchmarks is being fooled by compiler optimizations, which can also lead to wrong benchmark assumptions. In this section, we look at Go issue 14813 (https://github.com/golang/go/issues/14813, also discussed by Go project member Dave Cheney) with a population count function (a function that counts the number of bits set to 1):
This function takes and returns a uint64. To benchmark this function, we can write the following:
func BenchmarkPopcnt1(b *testing.B) {\nfor i := 0; i < b.N; i++ {\npopcnt(uint64(i))\n}\n}\n
However, if we execute this benchmark, we get a surprisingly low result:
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkPopcnt1-4 1000000000 0.2858 ns/op\n
A duration of 0.28 nanoseconds is roughly one clock cycle, so this number is unreasonably low. The problem is that the developer wasn\u2019t careful enough about compiler optimizations. In this case, the function under test is simple enough to be a candidate for inlining: an optimization that replaces a function call with the body of the called function and lets us prevent a function call, which has a small footprint. Once the function is inlined, the compiler notices that the call has no side effects and replaces it with the following benchmark:
func BenchmarkPopcnt1(b *testing.B) {\nfor i := 0; i < b.N; i++ {\n// Empty\n}\n}\n
The benchmark is now empty \u2014 which is why we got a result close to one clock cycle. To prevent this from happening, a best practice is to follow this pattern:
During each loop iteration, assign the result to a local variable (local in the context of the benchmark function).
Assign the latest result to a global variable.
In our case, we write the following benchmark:
var global uint64 // Define a global variable\nfunc BenchmarkPopcnt2(b *testing.B) {\nvar v uint64 // Define a local variable\nfor i := 0; i < b.N; i++ {\nv = popcnt(uint64(i)) // Assign the result to the local variable\n}\nglobal = v // Assign the result to the global variable\n}\n
global is a global variable, whereas v is a local variable whose scope is the benchmark function. During each loop iteration, we assign the result of popcnt to the local variable. Then we assign the latest result to the global variable.
Note
Why not assign the result of the popcnt call directly to global to simplify the test? Writing to a global variable is slower than writing to a local variable (these concepts are discussed in 100 Go Mistakes, mistake #95: \u201cNot understanding stack vs. heap\u201d). Therefore, we should write each result to a local variable to limit the footprint during each loop iteration.
If we run these two benchmarks, we now get a significant difference in the results:
BenchmarkPopcnt2 is the accurate version of the benchmark. It guarantees that we avoid the inlining optimizations, which can artificially lower the execution time or even remove the call to the function under test. Relying on the results of BenchmarkPopcnt1 could have led to wrong assumptions.
Let\u2019s remember the pattern to avoid compiler optimizations fooling benchmark results: assign the result of the function under test to a local variable, and then assign the latest result to a global variable. This best practice also prevents us from making incorrect assumptions.
"},{"location":"89-benchmarks/#being-fooled-by-the-observer-effect","title":"Being fooled by the observer effect","text":"
In physics, the observer effect is the disturbance of an observed system by the act of observation. This effect can also be seen in benchmarks and can lead to wrong assumptions about results. Let\u2019s look at a concrete example and then try to mitigate it.
We want to implement a function receiving a matrix of int64 elements. This matrix has a fixed number of 512 columns, and we want to compute the total sum of the first eight columns, as shown in figure 1.
Figure 1: Computing the sum of the first eight columns.
For the sake of optimizations, we also want to determine whether varying the number of columns has an impact, so we also implement a second function with 513 columns. The implementation is the following:
func calculateSum512(s [][512]int64) int64 {\nvar sum int64\nfor i := 0; i < len(s); i++ { // Iterate over each row\nfor j := 0; j < 8; j++ { // Iterate over the first eight columns\nsum += s[i][j] // Increment sum\n}\n}\nreturn sum\n}\nfunc calculateSum513(s [][513]int64) int64 {\n// Same implementation as calculateSum512\n}\n
We iterate over each row and then over the first eight columns, and we increment a sum variable that we return. The implementation in calculateSum513 remains the same.
We want to benchmark these functions to decide which one is the most performant given a fixed number of rows:
const rows = 1000\nvar res int64\nfunc BenchmarkCalculateSum512(b *testing.B) {\nvar sum int64\ns := createMatrix512(rows) // Create a matrix of 512 columns\nb.ResetTimer()\nfor i := 0; i < b.N; i++ {\nsum = calculateSum512(s) // Create a matrix of 512 columns\n}\nres = sum\n}\nfunc BenchmarkCalculateSum513(b *testing.B) {\nvar sum int64\ns := createMatrix513(rows) // Create a matrix of 513 columns\nb.ResetTimer()\nfor i := 0; i < b.N; i++ {\nsum = calculateSum513(s) // Calculate the sum\n}\nres = sum\n}\n
We want to create the matrix only once, to limit the footprint on the results. Therefore, we call createMatrix512 and createMatrix513 outside of the loop. We may expect the results to be similar as again we only want to iterate on the first eight columns, but this isn\u2019t the case (on my machine):
The second benchmark with 513 columns is about 50% faster. Again, because we iterate only over the first eight columns, this result is quite surprising.
To understand this difference, we need to understand the basics of CPU caches. In a nutshell, a CPU is composed of different caches (usually L1, L2, and L3). These caches reduce the average cost of accessing data from the main memory. In some conditions, the CPU can fetch data from the main memory and copy it to L1. In this case, the CPU tries to fetch into L1 the matrix\u2019s subset that calculateSum is interested in (the first eight columns of each row). However, the matrix fits in memory in one case (513 columns) but not in the other case (512 columns).
Note
This isn\u2019t in the scope of this post to explain why, but we look at this problem in 100 Go Mistakes, mistake #91: \u201cNot understanding CPU caches.\u201d
Coming back to the benchmark, the main issue is that we keep reusing the same matrix in both cases. Because the function is repeated thousands of times, we don\u2019t measure the function\u2019s execution when it receives a plain new matrix. Instead, we measure a function that gets a matrix that already has a subset of the cells present in the cache. Therefore, because calculateSum513 leads to fewer cache misses, it has a better execution time.
This is an example of the observer effect. Because we keep observing a repeatedly called CPU-bound function, CPU caching may come into play and significantly affect the results. In this example, to prevent this effect, we should create a matrix during each test instead of reusing one:
func BenchmarkCalculateSum512(b *testing.B) {\nvar sum int64\nfor i := 0; i < b.N; i++ {\nb.StopTimer()\ns := createMatrix512(rows) // Create a new matrix during each loop iteration\nb.StartTimer()\nsum = calculateSum512(s)\n}\nres = sum\n}\n
A new matrix is now created during each loop iteration. If we run the benchmark again (and adjust benchtime \u2014 otherwise, it takes too long to execute), the results are closer to each other:
Instead of making the incorrect assumption that calculateSum513 is faster, we see that both benchmarks lead to similar results when receiving a new matrix.
As we have seen in this post, because we were reusing the same matrix, CPU caches significantly impacted the results. To prevent this, we had to create a new matrix during each loop iteration. In general, we should remember that observing a function under test may lead to significant differences in results, especially in the context of micro-benchmarks of CPU-bound functions where low-level optimizations matter. Forcing a benchmark to re-create data during each iteration can be a good way to prevent this effect.
"},{"location":"9-generics/","title":"Being confused about when to use generics","text":"
Generics is a fresh addition to the language. In a nutshell, it allows writing code with types that can be specified later and instantiated when needed. However, it can be pretty easy to be confused about when to use generics and when not to. Throughout this post, we will describe the concept of generics in Go and then delve into common use and misuses.
Consider the following function that extracts all the keys from a map[string]int type:
func getKeys(m map[string]int) []string {\nvar keys []string\nfor k := range m {\nkeys = append(keys, k)\n}\nreturn keys\n}\n
What if we would like to use a similar feature for another map type such as a map[int]string? Before generics, Go developers had a couple of options: using code generation, reflection, or duplicating code.
For example, we could write two functions, one for each map type, or even try to extend getKeys to accept different map types:
func getKeys(m any) ([]any, error) {\nswitch t := m.(type) {\ndefault:\nreturn nil, fmt.Errorf(\"unknown type: %T\", t)\ncase map[string]int:\nvar keys []any\nfor k := range t {\nkeys = append(keys, k)\n}\nreturn keys, nil\ncase map[int]string:\n// Copy the extraction logic\n}\n}\n
We can start noticing a couple of issues:
First, it increases boilerplate code. Indeed, whenever we want to add a case, it will require duplicating the range loop.
Meanwhile, the function now accepts an empty interface, which means we are losing some of the benefits of Go being a typed language. Indeed, checking whether a type is supported is done at runtime instead of compile-time. Hence, we also need to return an error if the provided type is unknown.
Last but not least, as the key type can be either int or string, we are obliged to return a slice of empty interfaces to factor out key types. This approach increases the effort on the caller-side as the client may also have to perform a type check of the keys or extra conversion.
Thanks to generics, we can now refactor this code using type parameters.
Type parameters are generic types we can use with functions and types. For example, the following function accepts a type parameter:
func foo[T any](t T) {\n// ...\n}\n
When calling foo, we will pass a type argument of any type. Passing a type argument is called instantiation because the work is done at compile time which keeps type safety as part of the core language features and avoids runtime overheads.
Let\u2019s get back to the getKeys function and use type parameters to write a generic version that would accept any kind of map:
func getKeys[K comparable, V any](m map[K]V) []K { <1>\nvar keys []K <2>\nfor k := range m {\nkeys = append(keys, k)\n}\nreturn keys\n}\n
To handle the map, we defined two kinds of type parameters. First, the values can be of any type: V any. However, in Go, the map keys can\u2019t be of any type. For example, we cannot use slices:
var m map[[]byte]int\n
This code leads to a compilation error: invalid map key type []byte. Therefore, instead of accepting any key type, we are obliged to restrict type arguments so that the key type meets specific requirements. Here, being comparable (we can use == or !=). Hence, we defined K as comparable instead of a``ny.
Restricting type arguments to match specific requirements is called a constraint. A constraint is an interface type that can contain:
A set of behaviors (methods)
But also arbitrary type
Let\u2019s see a concrete example for the latter. Imagine we don\u2019t want to accept any comparable type for map key type. For instance, we would like to restrict it to either int or string types. We can define a custom constraint this way:
type customConstraint interface {\n~int | ~string // Define a custom type that will restrict types to int and string\n}\n// Change the type parameter K to be custom\nfunc getKeys[K customConstraint, V any](m map[K]V) []K {\n// Same implementation\n}\n
First, we define a customConstraint interface to restrict the types to be either int or string using the union operator | (we will discuss the use of ~ a bit later). Then, K is now a customConstraint instead of a comparable as before.
Now, the signature of getKeys enforces that we can call it with a map of any value type, but the key type has to be an int or a string. For example, on the caller-side:
m = map[string]int{\n\"one\": 1,\n\"two\": 2,\n\"three\": 3,\n}\nkeys := getKeys(m)\n
Note that Go can infer that getKeys is called with a string type argument. The previous call was similar to this:
keys := getKeys[string](m)\n
Note
What\u2019s the difference between a constraint using ~int or int? Using int restricts it to that type, whereas ~int restricts all the types whose underlying type is an int.
To illustrate it, let\u2019s imagine a constraint where we would like to restrict a type to any int type implementing the String() string method:
type customConstraint interface {\n~int\nString() string\n}\n
Using this constraint will restrict type arguments to custom types like this one:
type customInt int\nfunc (i customInt) String() string {\nreturn strconv.Itoa(int(i))\n}\n
As customInt is an int and implements the String() string method, the customInt type satisfies the constraint defined.
However, if we change the constraint to contain an int instead of an ~int, using customInt would lead to a compilation error because the int type doesn\u2019t implement String() string.
Let\u2019s also note the constraints package contains a set of common constraints such as Signed that includes all the signed integer types. Let\u2019s ensure that a constraint doesn\u2019t already exist in this package before creating a new one.
So far, we have discussed examples using generics for functions. However, we can also use generics with data structures.
For example, we will create a linked list containing values of any type. Meanwhile, we will write an Add method to append a node:
type Node[T any] struct { // Use type parameter\nVal T\nnext *Node[T]\n}\nfunc (n *Node[T]) Add(next *Node[T]) { // Instantiate type receiver\nn.next = next\n}\n
We use type parameters to define T and use both fields in Node. Regarding the method, the receiver is instantiated. Indeed, because Node is generic, it has to follow also the type parameter defined.
One last thing to note about type parameters: they can\u2019t be used on methods, only on functions. For example, the following method wouldn\u2019t compile:
type Foo struct {}\nfunc (Foo) bar[T any](t T) {}\n
./main.go:29:15: methods cannot have type parameters\n
Now, let\u2019s delve into concrete cases where we should and shouldn\u2019t use generics.
"},{"location":"9-generics/#common-uses-and-misuses","title":"Common uses and misuses","text":"
So when are generics useful? Let\u2019s discuss a couple of common uses where generics are recommended:
Data structures. For example, we can use generics to factor out the element type if we implement a binary tree, a linked list, or a heap.
Functions working with slices, maps, and channels of any type. For example, a function to merge two channels would work with any channel type. Hence, we could use type parameters to factor out the channel type:
func merge[T any](ch1, ch2 <-chan T) <-chan T {\n// ...\n}\n
Meanwhile, instead of factoring out a type, we can factor out behaviors. For example, the sort package contains functions to sort different slice types such as sort.Ints or sort.Float64s. Using type parameters, we can factor out the sorting behaviors that rely on three methods, Len, Less, and Swap:
type sliceFn[T any] struct { // Use type parameter\ns []T\ncompare func(T, T) bool // Compare two T elements\n}\nfunc (s sliceFn[T]) Len() int { return len(s.s) }\nfunc (s sliceFn[T]) Less(i, j int) bool { return s.compare(s.s[i], s.s[j]) }\nfunc (s sliceFn[T]) Swap(i, j int) { s.s[i], s.s[j] = s.s[j], s.s[i] }\n
Conversely, when is it recommended not to use generics?
When just calling a method of the type argument. For example, consider a function that receives an io.Writer and call the Write method:
When it makes our code more complex. Generics are never mandatory, and as Go developers, we have been able to live without them for more than a decade. If writing generic functions or structures we figure out that it doesn\u2019t make our code clearer, we should probably reconsider our decision for this particular use case.
Though generics can be very helpful in particular conditions, we should be cautious about when to use them and not use them.
In general, when we want to answer when not to use generics, we can find similarities with when not to use interfaces. Indeed, generics introduce a form of abstraction, and we have to remember that unnecessary abstractions introduce complexity.
Let\u2019s not pollute our code with needless abstractions, and let\u2019s focus on solving concrete problems for now. It means that we shouldn\u2019t use type parameters prematurely. Let\u2019s wait until we are about to write boilerplate code to consider using generics.
"},{"location":"98-profiling-execution-tracing/","title":"Not using Go diagnostics tooling","text":"
Go offers a few excellent diagnostics tools to help us get insights into how an application performs. This post focuses on the most important ones: profiling and the execution tracer. Both tools are so important that they should be part of the core toolset of any Go developer who is interested in optimization. First, let\u2019s discuss profiling.
Profiling provides insights into the execution of an application. It allows us to resolve performance issues, detect contention, locate memory leaks, and more. These insights can be collected via several profiles:
CPU\u2014 Determines where an application spends its time
Goroutine\u2014 Reports the stack traces of the ongoing goroutines
Heap\u2014 Reports heap memory allocation to monitor current memory usage and check for possible memory leaks
Mutex\u2014 Reports lock contentions to see the behaviors of the mutexes used in our code and whether an application spends too much time in locking calls
Block\u2014 Shows where goroutines block waiting on synchronization primitives
Profiling is achieved via instrumentation using a tool called a profiler, in Go: pprof. First, let\u2019s understand how and when to enable pprof; then, we discuss the most critical profile types.
There are several ways to enable pprof. For example, we can use the net/http/pprof package to serve the profiling data via HTTP:
package main\nimport (\n\"fmt\"\n\"log\"\n\"net/http\"\n_ \"net/http/pprof\" // Blank import to pprof\n)\nfunc main() {\n// Exposes an HTTP endpoint\nhttp.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\nfmt.Fprintf(w, \"\")\n})\nlog.Fatal(http.ListenAndServe(\":80\", nil))\n}\n
Importing net/http/pprof leads to a side effect that allows us to reach the pprof URL: http://host/debug/pprof. Note that enabling pprof is safe even in production (https://go.dev/doc/diagnostics#profiling). The profiles that impact performance, such as CPU profiling, aren\u2019t enabled by default, nor do they run continuously: they are activated only for a specific period.
Now that we have seen how to expose a pprof endpoint, let\u2019s discuss the most common profiles.
The CPU profiler relies on the OS and signaling. When it is activated, the application asks the OS to interrupt it every 10 ms by default via a SIGPROF signal. When the application receives a SIGPROF, it suspends the current activity and transfers the execution to the profiler. The profiler collects data such as the current goroutine activity and aggregates execution statistics that we can retrieve. Then it stops, and the execution resumes until the next SIGPROF.
We can access the /debug/pprof/profile endpoint to activate CPU profiling. Accessing this endpoint executes CPU profiling for 30 seconds by default. For 30 seconds, our application is interrupted every 10 ms. Note that we can change these two default values: we can use the seconds parameter to pass to the endpoint how long the profiling should last (for example, /debug/pprof/profile?seconds=15), and we can change the interruption rate (even to less than 10 ms). But in most cases, 10 ms should be enough, and in decreasing this value (meaning increasing the rate), we should be careful not to harm performance. After 30 seconds, we download the results of the CPU profiler.
Note
We can also enable the CPU profiler using the -cpuprofile flag, such as when running a benchmark. For example, the following command produces the same type of file that can be downloaded via /debug/ pprof/profile.
$ go test -bench=. -cpuprofile profile.out\n
From this file, we can navigate to the results using go tool:
$ go tool pprof -http=:8080 <file>\n
This command opens a web UI showing the call graph. The next figure shows an example taken from an application. The larger the arrow, the more it was a hot path. We can then navigate into this graph and get execution insights.
Figure 1: The call graph of an application during 30 seconds.
For example, the graph in the next figure tells us that during 30 seconds, 0.06 seconds were spent in the decode method (*FetchResponse receiver). Of these 0.06 seconds, 0.02 were spent in RecordBatch.decode and 0.01 in makemap (creating a map).
Figure 2: Example call graph.
We can also access this kind of information from the web UI with different representations. For example, the Top view sorts the functions per execution time, and Flame Graph visualizes the execution time hierarchy. The UI can even display the expensive parts of the source code line by line.
Note
We can also delve into profiling data via a command line. However, we focus on the web UI in this post.
Thanks to this data, we can get a general idea of how an application behaves:
Too many calls to runtime.mallogc can mean an excessive number of small heap allocations that we can try to minimize.
Too much time spent in channel operations or mutex locks can indicate excessive contention that is harming the application\u2019s performance.
Too much time spent on syscall.Read or syscall.Write means the application spends a significant amount of time in Kernel mode. Working on I/O buffering may be an avenue for improvement.
These are the kinds of insights we can get from the CPU profiler. It\u2019s valuable to understand the hottest code path and identify bottlenecks. But it won\u2019t determine more than the configured rate because the CPU profiler is executed at a fixed pace (by default, 10 ms). To get finer-grained insights, we should use tracing, which we discuss later in this post.
Note
We can also attach labels to the different functions. For example, imagine a common function called from different clients. To track the time spent for both clients, we can use pprof.Labels.
Heap profiling allows us to get statistics about the current heap usage. Like CPU profiling, heap profiling is sample-based. We can change this rate, but we shouldn\u2019t be too granular because the more we decrease the rate, the more effort heap profiling will require to collect data. By default, samples are profiled at one allocation for every 512 KB of heap allocation.
If we reach /debug/pprof/heap/, we get raw data that can be hard to read. However, we can download a heap profile using /debug/pprof/heap/?debug=0 and then open it with go tool (the same command as in the previous section) to navigate into the data using the web UI.
The next figure shows an example of a heap graph. Calling the MetadataResponse.decode method leads to allocating 1536 KB of heap data (which represents 6.32% of the total heap). However, 0 out of these 1536 KB were allocated by this function directly, so we need to inspect the second call. The TopicMetadata.decode method allocated 512 KB out of the 1536 KB; the rest \u2014 1024 KB \u2014 were allocated in another method.
Figure 3: A heap graph.
This is how we can navigate the call chain to understand what part of an application is responsible for most of the heap allocations. We can also look at different sample types:
alloc_objects\u2014 Total number of objects allocated
alloc_space\u2014 Total amount of memory allocated
inuse_objects \u2014 Number of objects allocated and not yet released
inuse_space\u2014 Amount of memory allocated and not yet released
Another very helpful capability with heap profiling is tracking memory leaks. With a GC-based language, the usual procedure is the following:
Trigger a GC.
Download heap data.
Wait for a few seconds/minutes.
Trigger another GC.
Download another heap data.
Compare.
Forcing a GC before downloading data is a way to prevent false assumptions. For example, if we see a peak of retained objects without running a GC first, we cannot be sure whether it\u2019s a leak or objects that the next GC will collect.
Using pprof, we can download a heap profile and force a GC in the meantime. The procedure in Go is the following:
Go to /debug/pprof/heap?gc=1 (trigger the GC and download the heap profile).
Wait for a few seconds/minutes.
Go to /debug/pprof/heap?gc=1 again.
Use go tool to compare both heap profiles:
$ go tool pprof -http=:8080 -diff_base <file2> <file1>\n
The next figure shows the kind of data we can access. For example, the amount of heap memory held by the newTopicProducer method (top left) has decreased (\u2013513 KB). In contrast, the amount held by updateMetadata (bottom right) has increased (+512 KB). Slow increases are normal. The second heap profile may have been calculated in the middle of a service call, for example. We can repeat this process or wait longer; the important part is to track steady increases in allocations of a specific object.
Figure 4: The differences between the two heap profiles. Note
Another type of profiling related to the heap is allocs, which reports allocations. Heap profiling shows the current state of the heap memory. To get insights about past memory allocations since the application started, we can use allocations profiling. As discussed, because stack allocations are cheap, they aren\u2019t part of this profiling, which only focuses on the heap.
The goroutine profile reports the stack trace of all the current goroutines in an application. We can download a file using /debug/pprof/goroutine/?debug=0 and use go tool again. The next figure shows the kind of information we can get.
Figure 5: Goroutine graph.
We can see the current state of the application and how many goroutines were created per function. In this case, withRecover has created 296 ongoing goroutines (63%), and 29 were related to a call to responseFeeder.
This kind of information is also beneficial if we suspect goroutine leaks. We can look at goroutine profiler data to know which part of a system is the suspect.
The block profile reports where ongoing goroutines block waiting on synchronization primitives. Possibilities include
Sending or receiving on an unbuffered channel
Sending to a full channel
Receiving from an empty channel
Mutex contention
Network or filesystem waits
Block profiling also records the amount of time a goroutine has been waiting and is accessible via /debug/pprof/block. This profile can be extremely helpful if we suspect that performance is being harmed by blocking calls.
The block profile isn\u2019t enabled by default: we have to call runtime.SetBlockProfileRate to enable it. This function controls the fraction of goroutine blocking events that are reported. Once enabled, the profiler will keep collecting data in the background even if we don\u2019t call the /debug/pprof/block endpoint. Let\u2019s be cautious if we want to set a high rate so we don\u2019t harm performance.
Note
If we face a deadlock or suspect that goroutines are in a blocked state, the full goroutine stack dump (/debug/pprof/goroutine/?debug=2) creates a dump of all the current goroutine stack traces. This can be helpful as a first analysis step. For example, the following dump shows a Sarama goroutine blocked for 1,420 minutes on a channel-receive operation:
The last profile type is related to blocking but only regarding mutexes. If we suspect that our application spends significant time waiting for locking mutexes, thus harming execution, we can use mutex profiling. It\u2019s accessible via /debug/pprof/mutex.
This profile works in a manner similar to that for blocking. It\u2019s disabled by default: we have to enable it using runtime.SetMutexProfileFraction, which controls the fraction of mutex contention events reported.
Following are a few additional notes about profiling:
We haven\u2019t mentioned the threadcreate profile because it\u2019s been broken since 2013 (https://github.com/golang/go/issues/6104).
Be sure to enable only one profiler at a time: for example, do not enable CPU and heap profiling simultaneously. Doing so can lead to erroneous observations.
pprof is extensible, and we can create our own custom profiles using pprof.Profile.
We have seen the most important profiles that we can enable to help us understand how an application performs and possible avenues for optimization. In general, enabling pprof is recommended, even in production, because in most cases it offers an excellent balance between its footprint and the amount of insight we can get from it. Some profiles, such as the CPU profile, lead to performance penalties but only during the time they are enabled.
The execution tracer is a tool that captures a wide range of runtime events with go tool to make them available for visualization. It is helpful for the following:
Understanding runtime events such as how the GC performs
Understanding how goroutines execute
Identifying poorly parallelized execution
Let\u2019s try it with an example given the Concurrency isn\u2019t Always Faster in Go section. We discussed two parallel versions of the merge sort algorithm. The issue with the first version was poor parallelization, leading to the creation of too many goroutines. Let\u2019s see how the tracer can help us in validating this statement.
We will write a benchmark for the first version and execute it with the -trace flag to enable the execution tracer:
$ go test -bench=. -v -trace=trace.out\n
Note
We can also download a remote trace file using the /debug/pprof/ trace?debug=0 pprof endpoint.
This command creates a trace.out file that we can open using go tool:
$ go tool trace trace.out\n2021/11/26 21:36:03 Parsing trace...\n2021/11/26 21:36:31 Splitting trace...\n2021/11/26 21:37:00 Opening browser. Trace viewer is listening on\n http://127.0.0.1:54518\n
The web browser opens, and we can click View Trace to see all the traces during a specific timeframe, as shown in the next figure. This figure represents about 150 ms. We can see multiple helpful metrics, such as the goroutine count and the heap size. The heap size grows steadily until a GC is triggered. We can also observe the activity of the Go application per CPU core. The timeframe starts with user-level code; then a \u201cstop the world\u201d is executed, which occupies the four CPU cores for approximately 40 ms.
Figure 6: Showing goroutine activity and runtime events such as a GC phase.
Regarding concurrency, we can see that this version uses all the available CPU cores on the machine. However, the next figure zooms in on a portion of 1 ms. Each bar corresponds to a single goroutine execution. Having too many small bars doesn\u2019t look right: it means execution that is poorly parallelized.
Figure 7: Too many small bars mean poorly parallelized execution.
The next figure zooms even closer to see how these goroutines are orchestrated. Roughly 50% of the CPU time isn\u2019t spent executing application code. The white spaces represent the time the Go runtime takes to spin up and orchestrate new goroutines.
Figure 8: About 50% of CPU time is spent handling goroutine switches.
Let\u2019s compare this with the second parallel implementation, which was about an order of magnitude faster. The next figure again zooms to a 1 ms timeframe.
Figure 9: The number of white spaces has been significantly reduced, proving that the CPU is more fully occupied.
Each goroutine takes more time to execute, and the number of white spaces has been significantly reduced. Hence, the CPU is much more occupied executing application code than it was in the first version. Each millisecond of CPU time is spent more efficiently, explaining the benchmark differences.
Note that the granularity of the traces is per goroutine, not per function like CPU profiling. However, it\u2019s possible to define user-level tasks to get insights per function or group of functions using the runtime/trace package.
For example, imagine a function that computes a Fibonacci number and then writes it to a global variable using atomic. We can define two different tasks:
var v int64\n// Creates a fibonacci task\nctx, fibTask := trace.NewTask(context.Background(), \"fibonacci\")\ntrace.WithRegion(ctx, \"main\", func() {\nv = fibonacci(10)\n})\nfibTask.End()\n// Creates a store task\nctx, fibStore := trace.NewTask(ctx, \"store\")\ntrace.WithRegion(ctx, \"main\", func() {\natomic.StoreInt64(&result, v)\n})\nfibStore.End()\n
Using go tool, we can get more precise information about how these two tasks perform. In the previous trace UI, we can see the boundaries for each task per goroutine. In User-Defined Tasks, we can follow the duration distribution:
Figure 10: Distribution of user-level tasks.
We see that in most cases, the fibonacci task is executed in less than 15 microseconds, whereas the store task takes less than 6309 nanoseconds.
In the previous section, we discussed the kinds of information we can get from CPU profiling. What are the main differences compared to the data we can get from user-level traces?
CPU profiling:
Sample-based
Per function
Doesn\u2019t go below the sampling rate (10 ms by default)
User-level traces:
Not sample-based
Per-goroutine execution (unless we use the runtime/trace package)
Time executions aren\u2019t bound by any rate
In summary, the execution tracer is a powerful tool for understanding how an application performs. As we have seen with the merge sort example, we can identify poorly parallelized execution. However, the tracer\u2019s granularity remains per goroutine unless we manually use runtime/trace compared to a CPU profile, for example. We can use both profiling and the execution tracer to get the most out of the standard Go diagnostics tools when optimizing an application.
"},{"location":"book/","title":"100 Go Mistakes and How to Avoid Them","text":"
Community space of \ud83d\udcd6 100 Go Mistakes and How to Avoid Them, published by Manning in 2022.
If you're a Go developer looking to improve your skills, this book is for you. With a focus on practical examples, 100 Go Mistakes and How to Avoid Them covers a wide range of topics from concurrency and error handling to testing and code organization. You'll learn to write more idiomatic, efficient, and maintainable code and become a proficient Go developer.
Read a summary of the 100 mistakes here.
"},{"location":"book/#quotes-and-ratings","title":"Quotes and Ratings","text":"
Krystian (Goodreads user)
This is an exceptional book. Usually, if a book contains either high-quality explanations or is written succinctly, I consider myself lucky to have found it. This one combines these two characteristics, which is super rare. It's another Go book for me and I still had quite a lot of \"a-ha!\" moments while reading it, and all of that without the unnecessary fluff, just straight to the point.
Akash Chetty
The book is completely exceptional, especially the examples carved out for each topic are really great. There is one topic that I struggled to understand is Concurrency but the way it is explained in this book is truly an art of genius.
Neeraj Shah
This should be the required reading for all Golang developers before they touch code in Production... It's the Golang equivalent of the legendary 'Effective Java' by Joshua Bloch.
Anupam Sengupta
Not having this will be the 101st mistake a Go programmer could make.
Manning, Goodreads, and Amazon reviews: 4.7/5 avg rating"},{"location":"book/#where-to-buy","title":"Where to Buy?","text":"
100 Go Mistakes and How to Avoid Them (\ud83c\uddec\ud83c\udde7 edition: paper, digital, or audiobook)
Manning (make sure to use my personal discount code for -35%: au35har)
Go\u8a00\u8a9e100Tips \u958b\u767a\u8005\u306b\u3042\u308a\u304c\u3061\u306a\u9593\u9055\u3044\u3078\u306e\u5bfe\u51e6\u6cd5 (\ud83c\uddef\ud83c\uddf5 edition: paper or digital)
Amazon: .co.jp
English and Japanese covers"},{"location":"book/#about-the-author","title":"About the Author","text":"
Teiva Harsanyi is a senior software engineer at Google. He has worked in various domains, including insurance, transportation, and safety-critical industries like air traffic management. He is passionate about Go and how to design and implement reliable systems.
"},{"location":"chapter-1/","title":"Go: Simple to learn but hard to master","text":"
This chapter covers
What makes Go an efficient, scalable, and productive language
Exploring why Go is simple to learn but hard to master
Presenting the common types of mistakes made by developers
Making mistakes is part of everyone\u2019s life. As Albert Einstein once said,
Albert Einstein
A person who never made a mistake never tried anything new.
What matters in the end isn\u2019t the number of mistakes we make, but our capacity to learn from them. This assertion also applies to programming. The seniority we acquire in a language isn\u2019t a magical process; it involves making many mistakes and learning from them. The purpose of this book is centered around this idea. It will help you, the reader, become a more proficient Go developer by looking at and learning from 100 common mistakes people make in many areas of the language.
This chapter presents a quick refresher as to why Go has become mainstream over the years. We\u2019ll discuss why, despite Go being considered simple to learn, mastering its nuances can be challenging. Finally, we\u2019ll introduce the concepts this book covers.
If you are reading this book, it\u2019s likely that you\u2019re already sold on Go. Therefore, this section provides a brief reminder about what makes Go such a powerful language.
Software engineering has evolved considerably during the past decades. Most modern systems are no longer written by a single person but by teams consisting of multiple programmers\u2014sometimes even hundreds, if not thousands. Nowadays, code must be readable, expressive, and maintainable to guarantee a system\u2019s durability over the years. Meanwhile, in our fast-moving world, maximizing agility and reducing the time to market is critical for most organizations. Programming should also follow this trend, and companies strive to ensure that software engineers are as productive as possible when reading, writing, and maintaining code.
In response to these challenges, Google created the Go programming language in 2007. Since then, many organizations have adopted the language to support various use cases: APIs, automation, databases, CLIs (command-line interfaces), and so on. Many today consider Go the language of the cloud.
Feature-wise, Go has no type inheritance, no exceptions, no macros, no partial functions, no support for lazy variable evaluation or immutability, no operator overloading, no pattern matching, and on and on. Why are these features missing from the language? The official Go FAQ gives us some insight:
Go FAQ
Why does Go not have feature X? Your favorite feature may be missing because it doesn\u2019t fit, because it affects compilation speed or clarity of design, or because it would make the fundamental system model too difficult.
Judging the quality of a programming language via its number of features is probably not an accurate metric. At least, it\u2019s not an objective of Go. Instead, Go utilizes a few essential characteristics when adopting a language at scale for an organization. These include the following:
Stability\u2014Even though Go receives frequent updates (including improvements and security patches), it remains a stable language. Some may even consider this one of the best features of the language.
Expressivity\u2014We can define expressivity in a programming language by how naturally and intuitively we can write and read code. A reduced number of keywords and limited ways to solve common problems make Go an expressive language for large codebases.
Compilation\u2014As developers, what can be more exasperating than having to wait for a build to test our application? Targeting fast compilation times has always been a conscious goal for the language designers. This, in turn, enables productivity.
Safety\u2014Go is a strong, statically typed language. Hence, it has strict compiletime rules, which ensure the code is type-safe in most cases.
Go was built from the ground up with solid features such as outstanding concurrency primitives with goroutines and channels. There\u2019s not a strong need to rely on external libraries to build efficient concurrent applications. Observing how important concurrency is these days also demonstrates why Go is such a suitable language for the present and probably for the foreseeable future.
Some also consider Go a simple language. And, in a sense, this isn\u2019t necessarily wrong. For example, a newcomer can learn the language\u2019s main features in less than a day. So why read a book centered on the concept of mistakes if Go is simple?
"},{"location":"chapter-1/#simple-doesnt-mean-easy","title":"Simple doesn\u2019t mean easy","text":"
There is a subtle difference between simple and easy. Simple, applied to a technology, means not complicated to learn or understand. However, easy means that we can achieve anything without much effort. Go is simple to learn but not necessarily easy to master.
Let\u2019s take concurrency, for example. In 2019, a study focusing on concurrency bugs was published: Understanding Real-World Concurrency Bugs in Go. This study was the first systematic analysis of concurrency bugs. It focused on multiple popular Go repositories such as Docker, gRPC, and Kubernetes. One of the most important takeaways from this study is that most of the blocking bugs are caused by inaccurate use of the message-passing paradigm via channels, despite the belief that message passing is easier to handle and less error-prone than sharing memory.
What should be an appropriate reaction to such a takeaway? Should we consider that the language designers were wrong about message passing? Should we reconsider how we deal with concurrency in our project? Of course not.
It\u2019s not a question of confronting message passing versus sharing memory and determining the winner. However, it\u2019s up to us as Go developers to thoroughly understand how to use concurrency, its implications on modern processors, when to favor one approach over the other, and how to avoid common traps. This example highlights that although a concept such as channels and goroutines can be simple to learn, it isn\u2019t an easy topic in practice.
This leitmotif\u2014simple doesn\u2019t mean easy\u2014can be generalized to many aspects of Go, not only concurrency. Hence, to be proficient Go developers, we must have a thorough understanding of many aspects of the language, which requires time, effort, and mistakes.
This book aims to help accelerate our journey toward proficiency by delving into 100 Go mistakes.
"},{"location":"chapter-1/#100-go-mistakes","title":"100 Go mistakes","text":"
Why should we read a book about common Go mistakes? Why not deepen our knowledge with an ordinary book that would dig into different topics?
In a 2011 article, neuroscientists proved that the best time for brain growth is when we\u2019re facing mistakes. 1 Haven\u2019t we all experienced the process of learning from a mistake and recalling that occasion after months or even years, when some context related to it? As presented in another article, by Janet Metcalfe, this happens because mistakes have a facilitative effect. 2 The main idea is that we can remember not only the error but also the context surrounding the mistake. This is one of the reasons why learning from mistakes is so efficient.
To strengthen this facilitative effect, this book accompanies each mistake as much as possible with real-world examples. This book isn\u2019t only about theory; it also helps us get better at avoiding mistakes and making more well-informed, conscious decisions because we now understand the rationale behind them.
Unknown
Tell me and I forget. Teach me and I remember. Involve me and I learn.
This book presents seven main categories of mistakes. Overall, the mistakes can be classified as
The first type of mistake and probably the most obvious is software bugs. In 2020, a study conducted by Synopsys estimated the cost of software bugs in the U.S. alone to be over $2 trillion. 3
Furthermore, bugs can also lead to tragic impacts. We can, for example, mention cases such as Therac-25, a radiation therapy machine produced by Atomic Energy of Canada Limited (AECL). Because of a race condition, the machine gave its patients radiation doses that were hundreds of times greater than expected, leading to the death of three patients. Hence, software bugs aren\u2019t only about money. As developers, we should remember how impactful our jobs are.
This book covers plenty of cases that could lead to various software bugs, including data races, leaks, logic errors, and other defects. Although accurate tests should be a way to discover such bugs as early as possible, we may sometimes miss cases because of different factors such as time constraints or complexity. Therefore, as a Go developer, it\u2019s essential to make sure we avoid common bugs.
The next category of mistakes is related to unnecessary complexity. A significant part of software complexity comes from the fact that, as developers, we strive to think about imaginary futures. Instead of solving concrete problems right now, it can be tempting to build evolutionary software that could tackle whatever future use case arises. However, this leads to more drawbacks than benefits in most cases because it can make a codebase more complex to understand and reason about.
Getting back to Go, we can think of plenty of use cases where developers might be tempted to design abstractions for future needs, such as interfaces or generics. This book discusses topics where we should remain careful not to harm a codebase with needless complexity.
Another kind of mistake is to weaken readability. As Robert C. Martin wrote in his book Clean Code: A Handbook of Agile Software Craftsmanship, the ratio of time spent reading versus writing is well over 10 to 1. Most of us started to program on solo projects where readability wasn\u2019t that important. However, today\u2019s software engineering is programming with a time dimension: making sure we can still work with and maintain an application months, years, or perhaps even decades later.
When programming in Go, we can make many mistakes that can harm readability. These mistakes may include nested code, data type representations, or not using named result parameters in some cases. Throughout this book, we will learn how to write readable code and care for future readers (including our future selves).
"},{"location":"chapter-1/#suboptimal-or-unidiomatic-organization","title":"Suboptimal or unidiomatic organization","text":"
Be it while working on a new project or because we acquire inaccurate reflexes, another type of mistake is organizing our code and a project suboptimally and unidiomatically. Such issues can make a project harder to reason about and maintain. This book covers some of these common mistakes in Go. For example, we\u2019ll look at how to structure a project and deal with utility packages or init functions. All in all, looking at these mistakes should help us organize our code and projects more efficiently and idiomatically.
"},{"location":"chapter-1/#lack-of-api-convenience","title":"Lack of API convenience","text":"
Making common mistakes that weaken how convenient an API is for our clients is another type of mistake. If an API isn\u2019t user-friendly, it will be less expressive and, hence, harder to understand and more error-prone.
We can think about many situations such as overusing any types, using the wrong creational pattern to deal with options, or blindly applying standard practices from object-oriented programming that affect the usability of our APIs. This book covers common mistakes that prevent us from exposing convenient APIs for our users.
Under-optimized code is another type of mistake made by developers. It can happen for various reasons, such as not understanding language features or even a lack of fundamental knowledge. Performance is one of the most obvious impacts of this mistake, but not the only one.
We can think about optimizing code for other goals, such as accuracy. For example, this book provides some common techniques to ensure that floating-point operations are accurate. Meanwhile, we will cover plenty of cases that can negatively impact performance code because of poorly parallelized executions, not knowing how to reduce allocations, or the impacts of data alignment, for example. We will tackle optimization via different prisms.
"},{"location":"chapter-1/#lack-of-productivity","title":"Lack of productivity","text":"
In most cases, what\u2019s the best language we can choose when working on a new project? The one we\u2019re the most productive with. Being comfortable with how a language works and exploiting it to get the best out of it is crucial to reach proficiency.
In this book, we will cover many cases and concrete examples that will help us to be more productive while working in Go. For instance, we\u2019ll look at writing efficient tests to ensure that our code works, relying on the standard library to be more effective, and getting the best out of the profiling tools and linters. Now, it\u2019s time to delve into those 100 common Go mistakes.
Go is a modern programming language that enables developer productivity, which is crucial for most companies today.
Go is simple to learn but not easy to master. This is why we need to deepen our knowledge to make the most effective use of the language.
Learning via mistakes and concrete examples is a powerful way to be proficient in a language. This book will accelerate our path to proficiency by exploring 100 common mistakes.
J. S. Moser, H. S. Schroder, et al., \u201cMind Your Errors: Evidence for a Neural Mechanism Linking Growth Mindset to Adaptive Posterror Adjustments,\u201d Psychological Science, vol. 22, no. 12, pp. 1484\u20131489, Dec. 2011.\u00a0\u21a9
J. Metcalfe, \u201cLearning from Errors,\u201d Annual Review of Psychology, vol. 68, pp. 465\u2013489, Jan. 2017.\u00a0\u21a9
Synopsys, \u201cThe Cost of Poor Software Quality in the US: A 2020 Report.\u201d 2020. https://news.synopsys.com/2021-01-06-Synopsys-Sponsored-CISQ-Research-Estimates-Cost-of-Poor-Software-Quality-in-the-US-2-08-Trillion-in-2020.\u00a0\u21a9
Is your company hiring? Sponsor this repository and let a significant audience of Go developers know about your opportunities in this section (book sales: +8k, website traffic: +2000 views and +400 unique visitors per week).
\u53ea\u6709\u5728\u9700\u8981\u63a5\u53d7\u6216\u8fd4\u56de\u4efb\u610f\u7c7b\u578b\u65f6\uff0c\u624d\u4f7f\u7528 any\uff0c\u4f8b\u5982 json.Marshal\u3002\u5176\u4ed6\u60c5\u51b5\u4e0b\uff0c\u56e0\u4e3a any \u4e0d\u63d0\u4f9b\u6709\u610f\u4e49\u7684\u4fe1\u606f\uff0c\u53ef\u80fd\u4f1a\u5bfc\u81f4\u7f16\u8bd1\u65f6\u95ee\u9898\uff0c\u5982\u5141\u8bb8\u8c03\u7528\u8005\u8c03\u7528\u65b9\u6cd5\u5904\u7406\u4efb\u610f\u7c7b\u578b\u6570\u636e\u3002
\u5728 Go \u4e2d\u6574\u6570\u4e0a\u6ea2\u51fa\u548c\u4e0b\u6ea2\u662f\u9759\u9ed8\u5904\u7406\u7684\uff0c\u6240\u4ee5\u4f60\u53ef\u4ee5\u5b9e\u73b0\u81ea\u5df1\u7684\u51fd\u6570\u6765\u6355\u83b7\u5b83\u4eec\u3002
"},{"location":"zh/#_3","title":"\u63a7\u5236\u7ed3\u6784","text":""},{"location":"zh/#range-30","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u53d8\u91cf\u662f\u4e00\u4e2a\u62f7\u8d1d (#30)","text":"
range \u5faa\u73af\u4e2d\u7684\u5faa\u73af\u53d8\u91cf\u662f\u904d\u5386\u5bb9\u5668\u4e2d\u5143\u7d20\u503c\u7684\u4e00\u4e2a\u62f7\u8d1d\u3002\u56e0\u6b64\uff0c\u5982\u679c\u5143\u7d20\u503c\u662f\u4e00\u4e2astruct\u5e76\u4e14\u60f3\u5728 range \u4e2d\u4fee\u6539\u5b83\uff0c\u53ef\u4ee5\u901a\u8fc7\u7d22\u5f15\u503c\u6765\u8bbf\u95ee\u5e76\u4fee\u6539\u5b83\uff0c\u6216\u8005\u4f7f\u7528\u7ecf\u5178\u7684for\u5faa\u73af+\u7d22\u5f15\u503c\u7684\u5199\u6cd5\uff08\u9664\u975e\u904d\u5386\u7684\u5143\u7d20\u662f\u4e00\u4e2a\u6307\u9488\uff09\u3002
"},{"location":"zh/#range-channels-arrays-31","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u4e2d\u8fed\u4ee3\u76ee\u6807\u503c\u7684\u8ba1\u7b97\u65b9\u5f0f (channels \u548c arrays) (#31)","text":"
\u4f20\u9012\u7ed9 range \u64cd\u4f5c\u7684\u8fed\u4ee3\u76ee\u6807\u5bf9\u5e94\u7684\u8868\u8fbe\u5f0f\u7684\u503c\uff0c\u53ea\u4f1a\u5728\u5faa\u73af\u6267\u884c\u524d\u88ab\u8ba1\u7b97\u4e00\u6b21\uff0c\u7406\u89e3\u8fd9\u4e2a\u6709\u52a9\u4e8e\u907f\u514d\u72af\u4e00\u4e9b\u5e38\u89c1\u7684\u9519\u8bef\uff0c\u4f8b\u5982\u4e0d\u9ad8\u6548\u7684channel\u8d4b\u503c\u64cd\u4f5c\u3001slice\u8fed\u4ee3\u64cd\u4f5c\u3002
"},{"location":"zh/#range-range-loops-32","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u4e2d\u6307\u9488\u5143\u7d20\u7684\u5f71\u54cd range loops (#32)","text":"
\u8fd9\u91cc\u5176\u5b9e\u5f3a\u8c03\u7684\u662f range \u8fed\u4ee3\u8fc7\u7a0b\u4e2d\uff0c\u8fed\u4ee3\u53d8\u91cf\u5b9e\u9645\u4e0a\u662f\u4e00\u4e2a\u62f7\u8d1d\uff0c\u5047\u8bbe\u7ed9\u53e6\u5916\u4e00\u4e2a\u5bb9\u5668\u5143\u7d20\uff08\u6307\u9488\u7c7b\u578b\uff09\u8d4b\u503c\uff0c\u4e14\u9700\u8981\u5bf9\u8fed\u4ee3\u53d8\u91cf\u53d6\u5730\u5740\u8f6c\u6362\u6210\u6307\u9488\u518d\u8d4b\u503c\u7684\u8bdd\uff0c\u8fd9\u91cc\u6f5c\u85cf\u7740\u4e00\u4e2a\u9519\u8bef\uff0c\u5c31\u662ffor\u5faa\u73af\u8fed\u4ee3\u53d8\u91cf\u662f per-variable-per-loop \u800c\u4e0d\u662f per-variable-per-iteration\u3002\u5982\u679c\u662f\u901a\u8fc7\u5c40\u90e8\u53d8\u91cf\uff08\u7528\u8fed\u4ee3\u53d8\u91cf\u6765\u521d\u59cb\u5316\uff09\u6216\u8005\u4f7f\u7528\u7d22\u5f15\u503c\u6765\u76f4\u63a5\u5f15\u7528\u8fed\u4ee3\u7684\u5143\u7d20\uff0c\u5c06\u6709\u52a9\u4e8e\u907f\u514d\u62f7\u8d1d\u6307\u9488(\u8fed\u4ee3\u53d8\u91cf\u7684\u5730\u5740)\u4e4b\u7c7b\u7684bug\u3002
ps\uff1a\u6570\u636e\u7ade\u4e89\u662f\u7ade\u6001\u6761\u4ef6\u7684\u5b50\u96c6\uff0c\u7ade\u6001\u6761\u4ef6\u4e0d\u4ec5\u5c40\u9650\u4e8e\u8bbf\u5b58\u672a\u540c\u6b65\uff0c\u5b83\u53ef\u4ee5\u53d1\u751f\u5728\u66f4\u9ad8\u7684\u5c42\u9762\u3002go test -race \u68c0\u6d4b\u7684\u662f\u6570\u636e\u7ade\u4e89\uff0c\u9700\u8981\u540c\u6b65\u6765\u89e3\u51b3\uff0c\u800c\u5f00\u53d1\u8005\u8fd8\u9700\u8981\u5173\u6ce8\u9762\u66f4\u5e7f\u7684\u7ade\u6001\u6761\u4ef6\uff0c\u5b83\u9700\u8981\u5bf9\u591a\u4e2agoroutines\u7684\u6267\u884c\u8fdb\u884c\u7f16\u6392\u3002
\u7406\u89e3 Go \u7684\u5185\u5b58\u6a21\u578b\u4ee5\u53ca\u6709\u5173\u987a\u5e8f\u548c\u540c\u6b65\u7684\u5e95\u5c42\u4fdd\u8bc1\u662f\u9632\u6b62\u53ef\u80fd\u7684\u6570\u636e\u7ade\u4e89\u548c\u7ade\u6001\u6761\u4ef6\u7684\u5173\u952e\u3002
Go \u7684\u4e0a\u4e0b\u6587\uff08context\uff09\u4e5f\u662f Go \u5e76\u53d1\u7f16\u7a0b\u7684\u57fa\u77f3\u4e4b\u4e00\u3002\u4e0a\u4e0b\u6587\u5141\u8bb8\u60a8\u643a\u5e26\u622a\u6b62\u65f6\u95f4\u3001\u53d6\u6d88\u4fe1\u53f7\u548c\u952e\u503c\u5217\u8868\u3002
\u8981\u660e\u767d\uff0cselect \u591a\u4e2achannels\u65f6\uff0c\u5982\u679c\u591a\u4e2achannels\u4e0a\u7684\u64cd\u4f5c\u90fd\u5c31\u7eea\uff0c\u90a3\u4e48\u4f1a\u968f\u673a\u9009\u62e9\u4e00\u4e2a case \u5206\u652f\u6765\u6267\u884c\uff0c\u56e0\u6b64\u8981\u907f\u514d\u6709\u5206\u652f\u9009\u62e9\u987a\u5e8f\u662f\u4ece\u4e0a\u5230\u4e0b\u7684\u8fd9\u79cd\u9519\u8bef\u9884\u8bbe\uff0c\u8fd9\u53ef\u80fd\u4f1a\u5bfc\u81f4\u8bbe\u8ba1\u4e0a\u7684bug\u3002
\u6ce8\u610f\u6709\u4e9b\u51fd\u6570\u63a5\u6536\u4e00\u4e2a time.Duration \u7c7b\u578b\u7684\u53c2\u6570\u65f6\uff0c\u5c3d\u7ba1\u76f4\u63a5\u4f20\u9012\u4e00\u4e2a\u6574\u6570\u662f\u53ef\u4ee5\u7684\uff0c\u4f46\u6700\u597d\u8fd8\u662f\u4f7f\u7528 time API \u4e2d\u7684\u65b9\u6cd5\u6765\u4f20\u9012 duration\uff0c\u4ee5\u907f\u514d\u53ef\u80fd\u9020\u6210\u7684\u56f0\u60d1\u3001bug\u3002
\u5f53\u63d0\u4f9b\u4e00\u4e2amap\u7528\u6765unmarshal JSON\u6570\u636e\u65f6\uff0c\u4e3a\u4e86\u907f\u514d\u4e0d\u786e\u5b9a\u7684value\u7ed3\u6784\u6211\u4eec\u4f1a\u4f7f\u7528 any \u6765\u4f5c\u4e3avalue\u7684\u7c7b\u578b\u800c\u4e0d\u662f\u5b9a\u4e49\u4e00\u4e2astruct\uff0c\u8fd9\u79cd\u60c5\u51b5\u4e0b\u9700\u8981\u8bb0\u5f97\u6570\u503c\u9ed8\u8ba4\u4f1a\u88ab\u8f6c\u6362\u4e3a float64\u3002
"},{"location":"zh/#time-api-87","title":"\u6ca1\u6709\u9ad8\u6548\u5730\u5904\u7406 time API (#87)","text":"
\u7406\u89e3\u5982\u4f55\u5904\u7406\u4f7f\u7528 time API \u7684\u51fd\u6570\uff0c\u662f\u4f7f\u6d4b\u8bd5\u66f4\u52a0\u7a33\u5b9a\u7684\u53e6\u4e00\u79cd\u65b9\u5f0f\u3002\u60a8\u53ef\u4ee5\u4f7f\u7528\u6807\u51c6\u6280\u672f\uff0c\u4f8b\u5982\u5c06\u65f6\u95f4\u4f5c\u4e3a\u9690\u85cf\u4f9d\u8d56\u9879\u7684\u4e00\u90e8\u5206\u6765\u5904\u7406\uff0c\u6216\u8005\u8981\u6c42\u5ba2\u6237\u7aef\u63d0\u4f9b\u65f6\u95f4\u3002
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Common Go Mistakes","text":"
This page is a summary of all the mistakes in the 100 Go Mistakes book. Meanwhile, it's also a page open to the community. If you believe that a mistake should be added, please create a community mistake issue.
Warning
You're currently viewing a new version that I'm enriching with significantly more content. Yet, this version is still under development; please be gentle if you find an issue, and feel free to create a PR.
"},{"location":"#code-and-project-organization","title":"Code and Project Organization","text":""},{"location":"#unintended-variable-shadowing-1","title":"Unintended variable shadowing (#1)","text":"TL;DR
Avoiding shadowed variables can help prevent mistakes like referencing the wrong variable or confusing readers.
Variable shadowing occurs when a variable name is redeclared in an inner block, but this practice is prone to mistakes. Imposing a rule to forbid shadowed variables depends on personal taste. For example, sometimes it can be convenient to reuse an existing variable name like err for errors. Yet, in general, we should remain cautious because we now know that we can face a scenario where the code compiles, but the variable that receives the value is not the one expected.
Avoiding nested levels and keeping the happy path aligned on the left makes building a mental code model easier.
In general, the more nested levels a function requires, the more complex it is to read and understand. Let\u2019s see some different applications of this rule to optimize our code for readability:
When an if block returns, we should omit the else block in all cases. For example, we shouldn\u2019t write:
if foo() {\n// ...\nreturn true\n} else {\n// ...\n}\n
Instead, we omit the else block like this:
if foo() {\n// ...\nreturn true\n}\n// ...\n
We can also follow this logic with a non-happy path:
if s != \"\" {\n// ...\n} else {\nreturn errors.New(\"empty string\")\n}\n
Here, an empty s represents the non-happy path. Hence, we should flip the condition like so:
if s == \"\" {\nreturn errors.New(\"empty string\")\n}\n// ...\n
Writing readable code is an important challenge for every developer. Striving to reduce the number of nested blocks, aligning the happy path on the left, and returning as early as possible are concrete means to improve our code\u2019s readability.
When initializing variables, remember that init functions have limited error handling and make state handling and testing more complex. In most cases, initializations should be handled as specific functions.
An init function is a function used to initialize the state of an application. It takes no arguments and returns no result (a func() function). When a package is initialized, all the constant and variable declarations in the package are evaluated. Then, the init functions are executed.
Init functions can lead to some issues:
They can limit error management.
They can complicate how to implement tests (for example, an external dependency must be set up, which may not be necessary for the scope of unit tests).
If the initialization requires us to set a state, that has to be done through global variables.
We should be cautious with init functions. They can be helpful in some situations, however, such as defining static configuration. Otherwise, and in most cases, we should handle initializations through ad hoc functions.
Source code
"},{"location":"#overusing-getters-and-setters-4","title":"Overusing getters and setters (#4)","text":"TL;DR
Forcing the use of getters and setters isn\u2019t idiomatic in Go. Being pragmatic and finding the right balance between efficiency and blindly following certain idioms should be the way to go.
Data encapsulation refers to hiding the values or state of an object. Getters and setters are means to enable encapsulation by providing exported methods on top of unexported object fields.
In Go, there is no automatic support for getters and setters as we see in some languages. It is also considered neither mandatory nor idiomatic to use getters and setters to access struct fields. We shouldn\u2019t overwhelm our code with getters and setters on structs if they don\u2019t bring any value. We should be pragmatic and strive to find the right balance between efficiency and following idioms that are sometimes considered indisputable in other programming paradigms.
Remember that Go is a unique language designed for many characteristics, including simplicity. However, if we find a need for getters and setters or, as mentioned, foresee a future need while guaranteeing forward compatibility, there\u2019s nothing wrong with using them.
Abstractions should be discovered, not created. To prevent unnecessary complexity, create an interface when you need it and not when you foresee needing it, or if you can at least prove the abstraction to be a valid one.
An interface provides a way to specify the behavior of an object. We use interfaces to create common abstractions that multiple objects can implement. What makes Go interfaces so different is that they are satisfied implicitly. There is no explicit keyword like implements to mark that an object X implements interface Y.
In general, we can define three main use cases where interfaces are generally considered as bringing value: factoring out a common behavior, creating some decoupling, and restricting a type to a certain behavior. Yet, this list isn't exhaustive and will also depend on the context we face.
In many occasions, interfaces are made to create abstractions. And the main caveat when programming meets abstractions is remembering that abstractions should be discovered, not created. What does this mean? It means we shouldn\u2019t start creating abstractions in our code if there is no immediate reason to do so. We shouldn\u2019t design with interfaces but wait for a concrete need. Said differently, we should create an interface when we need it, not when we foresee that we could need it. What\u2019s the main problem if we overuse interfaces? The answer is that they make the code flow more complex. Adding a useless level of indirection doesn\u2019t bring any value; it creates a worthless abstraction making the code more difficult to read, understand, and reason about. If we don\u2019t have a strong reason for adding an interface and it\u2019s unclear how an interface makes a code better, we should challenge this interface\u2019s purpose. Why not call the implementation directly?
We should be cautious when creating abstractions in our code (abstractions should be discovered, not created). It\u2019s common for us, software developers, to overengineer our code by trying to guess what the perfect level of abstraction is, based on what we think we might need later. This process should be avoided because, in most cases, it pollutes our code with unnecessary abstractions, making it more complex to read.
Rob Pike
Don\u2019t design with interfaces, discover them.
Let\u2019s not try to solve a problem abstractly but solve what has to be solved now. Last, but not least, if it\u2019s unclear how an interface makes the code better, we should probably consider removing it to make our code simpler.
Source code
"},{"location":"#interface-on-the-producer-side-6","title":"Interface on the producer side (#6)","text":"TL;DR
Keeping interfaces on the client side avoids unnecessary abstractions.
Interfaces are satisfied implicitly in Go, which tends to be a gamechanger compared to languages with an explicit implementation. In most cases, the approach to follow is similar to what we described in the previous section: abstractions should be discovered, not created. This means that it\u2019s not up to the producer to force a given abstraction for all the clients. Instead, it\u2019s up to the client to decide whether it needs some form of abstraction and then determine the best abstraction level for its needs.
An interface should live on the consumer side in most cases. However, in particular contexts (for example, when we know\u2014not foresee\u2014that an abstraction will be helpful for consumers), we may want to have it on the producer side. If we do, we should strive to keep it as minimal as possible, increasing its reusability potential and making it more easily composable.
To prevent being restricted in terms of flexibility, a function shouldn\u2019t return interfaces but concrete implementations in most cases. Conversely, a function should accept interfaces whenever possible.
In most cases, we shouldn\u2019t return interfaces but concrete implementations. Otherwise, it can make our design more complex due to package dependencies and can restrict flexibility because all the clients would have to rely on the same abstraction. Again, the conclusion is similar to the previous sections: if we know (not foresee) that an abstraction will be helpful for clients, we can consider returning an interface. Otherwise, we shouldn\u2019t force abstractions; they should be discovered by clients. If a client needs to abstract an implementation for whatever reason, it can still do that on the client\u2019s side.
Only use any if you need to accept or return any possible type, such as json.Marshal. Otherwise, any doesn\u2019t provide meaningful information and can lead to compile-time issues by allowing a caller to call methods with any data type.
The any type can be helpful if there is a genuine need for accepting or returning any possible type (for instance, when it comes to marshaling or formatting). In general, we should avoid overgeneralizing the code we write at all costs. Perhaps a little bit of duplicated code might occasionally be better if it improves other aspects such as code expressiveness.
Source code
"},{"location":"#being-confused-about-when-to-use-generics-9","title":"Being confused about when to use generics (#9)","text":"TL;DR
Relying on generics and type parameters can prevent writing boilerplate code to factor out elements or behaviors. However, do not use type parameters prematurely, but only when you see a concrete need for them. Otherwise, they introduce unnecessary abstractions and complexity.
Read the full section here.
Source code
"},{"location":"#not-being-aware-of-the-possible-problems-with-type-embedding-10","title":"Not being aware of the possible problems with type embedding (#10)","text":"TL;DR
Using type embedding can also help avoid boilerplate code; however, ensure that doing so doesn\u2019t lead to visibility issues where some fields should have remained hidden.
When creating a struct, Go offers the option to embed types. But this can sometimes lead to unexpected behaviors if we don\u2019t understand all the implications of type embedding. Throughout this section, we look at how to embed types, what these bring, and the possible issues.
In Go, a struct field is called embedded if it\u2019s declared without a name. For example,
type Foo struct {\nBar // Embedded field\n}\ntype Bar struct {\nBaz int\n}\n
In the Foo struct, the Bar type is declared without an associated name; hence, it\u2019s an embedded field.
We use embedding to promote the fields and methods of an embedded type. Because Bar contains a Baz field, this field is promoted to Foo. Therefore, Baz becomes available from Foo.
What can we say about type embedding? First, let\u2019s note that it\u2019s rarely a necessity, and it means that whatever the use case, we can probably solve it as well without type embedding. Type embedding is mainly used for convenience: in most cases, to promote behaviors.
If we decide to use type embedding, we need to keep two main constraints in mind:
It shouldn\u2019t be used solely as some syntactic sugar to simplify accessing a field (such as Foo.Baz() instead of Foo.Bar.Baz()). If this is the only rationale, let\u2019s not embed the inner type and use a field instead.
It shouldn\u2019t promote data (fields) or a behavior (methods) we want to hide from the outside: for example, if it allows clients to access a locking behavior that should remain private to the struct.
Using type embedding consciously by keeping these constraints in mind can help avoid boilerplate code with additional forwarding methods. However, let\u2019s make sure we don\u2019t do it solely for cosmetics and not promote elements that should remain hidden.
Source code
"},{"location":"#not-using-the-functional-options-pattern-11","title":"Not using the functional options pattern (#11)","text":"TL;DR
To handle options conveniently and in an API-friendly manner, use the functional options pattern.
Although there are different implementations with minor variations, the main idea is as follows:
An unexported struct holds the configuration: options.
Each option is a function that returns the same type: type Option func(options *options) error. For example, WithPort accepts an int argument that represents the port and returns an Option type that represents how to update the options struct.
type options struct {\nport *int\n}\ntype Option func(options *options) error\nfunc WithPort(port int) Option {\nreturn func(options *options) error {\nif port < 0 {\nreturn errors.New(\"port should be positive\")\n}\noptions.port = &port\nreturn nil\n}\n}\nfunc NewServer(addr string, opts ...Option) ( *http.Server, error) { <1>\nvar options options <2>\nfor _, opt := range opts { <3>\nerr := opt(&options) <4>\nif err != nil {\nreturn nil, err\n}\n}\n// At this stage, the options struct is built and contains the config\n// Therefore, we can implement our logic related to port configuration\nvar port int\nif options.port == nil {\nport = defaultHTTPPort\n} else {\nif *options.port == 0 {\nport = randomPort()\n} else {\nport = *options.port\n}\n}\n// ...\n}\n
The functional options pattern provides a handy and API-friendly way to handle options. Although the builder pattern can be a valid option, it has some minor downsides (having to pass a config struct that can be empty or a less handy way to handle error management) that tend to make the functional options pattern the idiomatic way to deal with these kind of problems in Go.
Source code
"},{"location":"#project-misorganization-project-structure-and-package-organization-12","title":"Project misorganization (project structure and package organization) (#12)","text":"
Regarding the overall organization, there are different schools of thought. For example, should we organize our application by context or by layer? It depends on our preferences. We may favor grouping code per context (such as the customer context, the contract context, etc.), or we may favor following hexagonal architecture principles and group per technical layer. If the decision we make fits our use case, it cannot be a wrong decision, as long as we remain consistent with it.
Regarding packages, there are multiple best practices that we should follow. First, we should avoid premature packaging because it might cause us to overcomplicate a project. Sometimes, it\u2019s better to use a simple organization and have our project evolve when we understand what it contains rather than forcing ourselves to make the perfect structure up front. Granularity is another essential thing to consider. We should avoid having dozens of nano packages containing only one or two files. If we do, it\u2019s because we have probably missed some logical connections across these packages, making our project harder for readers to understand. Conversely, we should also avoid huge packages that dilute the meaning of a package name.
Package naming should also be considered with care. As we all know (as developers), naming is hard. To help clients understand a Go project, we should name our packages after what they provide, not what they contain. Also, naming should be meaningful. Therefore, a package name should be short, concise, expressive, and, by convention, a single lowercase word.
Regarding what to export, the rule is pretty straightforward. We should minimize what should be exported as much as possible to reduce the coupling between packages and keep unnecessary exported elements hidden. If we are unsure whether to export an element or not, we should default to not exporting it. Later, if we discover that we need to export it, we can adjust our code. Let\u2019s also keep in mind some exceptions, such as making fields exported so that a struct can be unmarshaled with encoding/json.
Organizing a project isn\u2019t straightforward, but following these rules should help make it easier to maintain. However, remember that consistency is also vital to ease maintainability. Therefore, let\u2019s make sure that we keep things as consistent as possible within a codebase.
Note
In 2023, the Go team has published an official guideline for organizing / structuring a Go project: go.dev/doc/modules/layout
Naming is a critical piece of application design. Creating packages such as common, util, and shared doesn\u2019t bring much value for the reader. Refactor such packages into meaningful and specific package names.
Also, bear in mind that naming a package after what it provides and not what it contains can be an efficient way to increase its expressiveness.
Source code
"},{"location":"#ignoring-package-name-collisions-14","title":"Ignoring package name collisions (#14)","text":"TL;DR
To avoid naming collisions between variables and packages, leading to confusion or perhaps even bugs, use unique names for each one. If this isn\u2019t feasible, use an import alias to change the qualifier to differentiate the package name from the variable name, or think of a better name.
Package collisions occur when a variable name collides with an existing package name, preventing the package from being reused. We should prevent variable name collisions to avoid ambiguity. If we face a collision, we should either find another meaningful name or use an import alias.
To help clients and maintainers understand your code\u2019s purpose, document exported elements.
Documentation is an important aspect of coding. It simplifies how clients can consume an API but can also help in maintaining a project. In Go, we should follow some rules to make our code idiomatic:
First, every exported element must be documented. Whether it is a structure, an interface, a function, or something else, if it\u2019s exported, it must be documented. The convention is to add comments, starting with the name of the exported element.
As a convention, each comment should be a complete sentence that ends with punctuation. Also bear in mind that when we document a function (or a method), we should highlight what the function intends to do, not how it does it; this belongs to the core of a function and comments, not documentation. Furthermore, the documentation should ideally provide enough information that the consumer does not have to look at our code to understand how to use an exported element.
When it comes to documenting a variable or a constant, we might be interested in conveying two aspects: its purpose and its content. The former should live as code documentation to be useful for external clients. The latter, though, shouldn\u2019t necessarily be public.
To help clients and maintainers understand a package\u2019s scope, we should also document each package. The convention is to start the comment with // Package followed by the package name. The first line of a package comment should be concise. That\u2019s because it will appear in the package. Then, we can provide all the information we need in the following lines.
Documenting our code shouldn\u2019t be a constraint. We should take the opportunity to make sure it helps clients and maintainers to understand the purpose of our code.
"},{"location":"#not-using-linters-16","title":"Not using linters (#16)","text":"TL;DR
To improve code quality and consistency, use linters and formatters.
A linter is an automatic tool to analyze code and catch errors. The scope of this section isn\u2019t to give an exhaustive list of the existing linters; otherwise, it will become deprecated pretty quickly. But we should understand and remember why linters are essential for most Go projects.
However, if you\u2019re not a regular user of linters, here is a list that you may want to use daily:
https://golang.org/cmd/vet\u2014A standard Go analyzer
https://github.com/jgautheron/goconst\u2014A repeated string constants analyzer * Besides linters, we should also use code formatters to fix code style. Here is a list of some code formatters for you to try:
https://golang.org/cmd/gofmt\u2014A standard Go code formatter
https://godoc.org/golang.org/x/tools/cmd/goimports\u2014A standard Go imports formatter * Meanwhile, we should also look at golangci-lint (https://github.com/golangci/golangci-lint). It\u2019s a linting tool that provides a facade on top of many useful linters and formatters. Also, it allows running the linters in parallel to improve analysis speed, which is quite handy.
Linters and formatters are a powerful way to improve the quality and consistency of our codebase. Let\u2019s take the time to understand which one we should use and make sure we automate their execution (such as a CI or Git precommit hook).
"},{"location":"#data-types","title":"Data Types","text":""},{"location":"#creating-confusion-with-octal-literals-17","title":"Creating confusion with octal literals (#17)","text":"TL;DR
When reading existing code, bear in mind that integer literals starting with 0 are octal numbers. Also, to improve readability, make octal integers explicit by prefixing them with 0o.
Octal numbers start with a 0 (e.g., 010 is equal to 8 in base 10). To improve readability and avoid potential mistakes for future code readers, we should make octal numbers explicit using the 0o prefix (e.g., 0o10).
We should also note the other integer literal representations:
Binary\u2014Uses a 0b or 0B prefix (for example, 0b100 is equal to 4 in base 10)
Hexadecimal\u2014Uses an 0x or 0X prefix (for example, 0xF is equal to 15 in base 10)
Imaginary\u2014Uses an i suffix (for example, 3i)
We can also use an underscore character (_) as a separator for readability. For example, we can write 1 billion this way: 1_000_000_000. We can also use the underscore character with other representations (for example, 0b00_00_01).
Because integer overflows and underflows are handled silently in Go, you can implement your own functions to catch them.
In Go, an integer overflow that can be detected at compile time generates a compilation error. For example,
var counter int32 = math.MaxInt32 + 1\n
constant 2147483648 overflows int32\n
However, at run time, an integer overflow or underflow is silent; this does not lead to an application panic. It is essential to keep this behavior in mind, because it can lead to sneaky bugs (for example, an integer increment or addition of positive integers that leads to a negative result).
Making floating-point comparisons within a given delta can ensure that your code is portable. When performing addition or subtraction, group the operations with a similar order of magnitude to favor accuracy. Also, perform multiplication and division before addition and subtraction.
In Go, there are two floating-point types (if we omit imaginary numbers): float32 and float64. The concept of a floating point was invented to solve the major problem with integers: their inability to represent fractional values. To avoid bad surprises, we need to know that floating-point arithmetic is an approximation of real arithmetic.
For that, we\u2019ll look at a multiplication example:
var n float32 = 1.0001\nfmt.Println(n * n)\n
We may expect this code to print the result of 1.0001 * 1.0001 = 1.00020001, right? However, running it on most x86 processors prints 1.0002, instead.
Because Go\u2019s float32 and float64 types are approximations, we have to bear a few rules in mind:
When comparing two floating-point numbers, check that their difference is within an acceptable range.
When performing additions or subtractions, group operations with a similar order of magnitude for better accuracy.
To favor accuracy, if a sequence of operations requires addition, subtraction, multiplication, or division, perform the multiplication and division operations first.
Source code
"},{"location":"#not-understanding-slice-length-and-capacity-20","title":"Not understanding slice length and capacity (#20)","text":"TL;DR
Understanding the difference between slice length and capacity should be part of a Go developer\u2019s core knowledge. The slice length is the number of available elements in the slice, whereas the slice capacity is the number of elements in the backing array.
When creating a slice, initialize it with a given length or capacity if its length is already known. This reduces the number of allocations and improves performance.
While initializing a slice using make, we can provide a length and an optional capacity. Forgetting to pass an appropriate value for both of these parameters when it makes sense is a widespread mistake. Indeed, it can lead to multiple copies and additional effort for the GC to clean the temporary backing arrays. Performance-wise, there\u2019s no good reason not to give the Go runtime a helping hand.
Our options are to allocate a slice with either a given capacity or a given length. Of these two solutions, we have seen that the second tends to be slightly faster. But using a given capacity and append can be easier to implement and read in some contexts.
Source code
"},{"location":"#being-confused-about-nil-vs-empty-slice-22","title":"Being confused about nil vs. empty slice (#22)","text":"TL;DR
To prevent common confusions such as when using the encoding/json or the reflect package, you need to understand the difference between nil and empty slices. Both are zero-length, zero-capacity slices, but only a nil slice doesn\u2019t require allocation.
In Go, there is a distinction between nil and empty slices. A nil slice is equals to nil, whereas an empty slice has a length of zero. A nil slice is empty, but an empty slice isn\u2019t necessarily nil. Meanwhile, a nil slice doesn\u2019t require any allocation. We have seen throughout this section how to initialize a slice depending on the context by using
var s []string if we aren\u2019t sure about the final length and the slice can be empty
[]string(nil) as syntactic sugar to create a nil and empty slice
make([]string, length) if the future length is known
The last option, []string{}, should be avoided if we initialize the slice without elements. Finally, let\u2019s check whether the libraries we use make the distinctions between nil and empty slices to prevent unexpected behaviors.
Source code
"},{"location":"#not-properly-checking-if-a-slice-is-empty-23","title":"Not properly checking if a slice is empty (#23)","text":"TL;DR
To check if a slice doesn\u2019t contain any element, check its length. This check works regardless of whether the slice is nil or empty. The same goes for maps. To design unambiguous APIs, you shouldn\u2019t distinguish between nil and empty slices.
To determine whether a slice has elements, we can either do it by checking if the slice is nil or if its length is equal to 0. Checking the length is the best option to follow as it will cover both if the slice is empty or is the slice is nil.
Meanwhile, when designing interfaces, we should avoid distinguishing nil and empty slices, which leads to subtle programming errors. When returning slices, it should make neither a semantic nor a technical difference if we return a nil or empty slice. Both should mean the same thing for the callers. This principle is the same with maps. To check if a map is empty, check its length, not whether it\u2019s nil.
Source code
"},{"location":"#not-making-slice-copies-correctly-24","title":"Not making slice copies correctly (#24)","text":"TL;DR
To copy one slice to another using the copy built-in function, remember that the number of copied elements corresponds to the minimum between the two slice\u2019s lengths.
Copying elements from one slice to another is a reasonably frequent operation. When using copy, we must recall that the number of elements copied to the destination corresponds to the minimum between the two slices\u2019 lengths. Also bear in mind that other alternatives exist to copy a slice, so we shouldn\u2019t be surprised if we find them in a codebase.
Source code
"},{"location":"#unexpected-side-effects-using-slice-append-25","title":"Unexpected side effects using slice append (#25)","text":"TL;DR
Using copy or the full slice expression is a way to prevent append from creating conflicts if two different functions use slices backed by the same array. However, only a slice copy prevents memory leaks if you want to shrink a large slice.
When using slicing, we must remember that we can face a situation leading to unintended side effects. If the resulting slice has a length smaller than its capacity, append can mutate the original slice. If we want to restrict the range of possible side effects, we can use either a slice copy or the full slice expression, which prevents us from doing a copy.
Note
s[low:high:max] (full slice expression): This statement creates a slice similar to the one created with s[low:high], except that the resulting slice\u2019s capacity is equal to max - low.
Source code
"},{"location":"#slices-and-memory-leaks-26","title":"Slices and memory leaks (#26)","text":"TL;DR
Working with a slice of pointers or structs with pointer fields, you can avoid memory leaks by marking as nil the elements excluded by a slicing operation.
Remember that slicing a large slice or array can lead to potential high memory consumption. The remaining space won\u2019t be reclaimed by the GC, and we can keep a large backing array despite using only a few elements. Using a slice copy is the solution to prevent such a case.
Source code
"},{"location":"#slice-and-pointers","title":"Slice and pointers","text":"
When we use the slicing operation with pointers or structs with pointer fields, we need to know that the GC won\u2019t reclaim these elements. In that case, the two options are to either perform a copy or explicitly mark the remaining elements or their fields to nil.
When creating a map, initialize it with a given length if its length is already known. This reduces the number of allocations and improves performance.
A map provides an unordered collection of key-value pairs in which all the keys are distinct. In Go, a map is based on the hash table data structure. Internally, a hash table is an array of buckets, and each bucket is a pointer to an array of key-value pairs.
If we know up front the number of elements a map will contain, we should create it by providing an initial size. Doing this avoids potential map growth, which is quite heavy computation-wise because it requires reallocating enough space and rebalancing all the elements.
Source code
"},{"location":"#maps-and-memory-leaks-28","title":"Maps and memory leaks (#28)","text":"TL;DR
A map can always grow in memory, but it never shrinks. Hence, if it leads to some memory issues, you can try different options, such as forcing Go to recreate the map or using pointers.
To compare types in Go, you can use the == and != operators if two types are comparable: Booleans, numerals, strings, pointers, channels, and structs are composed entirely of comparable types. Otherwise, you can either use reflect.DeepEqual and pay the price of reflection or use custom implementations and libraries.
It\u2019s essential to understand how to use == and != to make comparisons effectively. We can use these operators on operands that are comparable:
Booleans\u2014Compare whether two Booleans are equal.
Numerics (int, float, and complex types)\u2014Compare whether two numerics are equal.
Strings\u2014Compare whether two strings are equal.
Channels\u2014Compare whether two channels were created by the same call to make or if both are nil.
Interfaces\u2014Compare whether two interfaces have identical dynamic types and equal dynamic values or if both are nil.
Pointers\u2014Compare whether two pointers point to the same value in memory or if both are nil.
Structs and arrays\u2014Compare whether they are composed of similar types.
Note
We can also use the ?, >=, <, and > operators with numeric types to compare values and with strings to compare their lexical order.
If operands are not comparable (e.g., slices and maps), we have to use other options such as reflection. Reflection is a form of metaprogramming, and it refers to the ability of an application to introspect and modify its structure and behavior. For example, in Go, we can use reflect.DeepEqual. This function reports whether two elements are deeply equal by recursively traversing two values. The elements it accepts are basic types plus arrays, structs, slices, maps, pointers, interfaces, and functions. Yet, the main catch is the performance penalty.
If performance is crucial at run time, implementing our custom method might be the best solution. One additional note: we must remember that the standard library has some existing comparison methods. For example, we can use the optimized bytes.Compare function to compare two slices of bytes. Before implementing a custom method, we need to make sure we don\u2019t reinvent the wheel.
Source code
"},{"location":"#control-structures","title":"Control Structures","text":""},{"location":"#ignoring-that-elements-are-copied-in-range-loops-30","title":"Ignoring that elements are copied in range loops (#30)","text":"TL;DR
The value element in a range loop is a copy. Therefore, to mutate a struct, for example, access it via its index or via a classic for loop (unless the element or the field you want to modify is a pointer).
A range loop allows iterating over different data structures:
String
Array
Pointer to an array
Slice
Map
Receiving channel
Compared to a classic for loop, a range loop is a convenient way to iterate over all the elements of one of these data structures, thanks to its concise syntax.
Yet, we should remember that the value element in a range loop is a copy. Therefore, if the value is a struct we need to mutate, we will only update the copy, not the element itself, unless the value or field we modify is a pointer. The favored options are to access the element via the index using a range loop or a classic for loop.
Source code
"},{"location":"#ignoring-how-arguments-are-evaluated-in-range-loops-channels-and-arrays-31","title":"Ignoring how arguments are evaluated in range loops (channels and arrays) (#31)","text":"TL;DR
Understanding that the expression passed to the range operator is evaluated only once before the beginning of the loop can help you avoid common mistakes such as inefficient assignment in channel or slice iteration.
The range loop evaluates the provided expression only once, before the beginning of the loop, by doing a copy (regardless of the type). We should remember this behavior to avoid common mistakes that might, for example, lead us to access the wrong element. For example:
a := [3]int{0, 1, 2}\nfor i, v := range a {\na[2] = 10\nif i == 2 {\nfmt.Println(v)\n}\n}\n
This code updates the last index to 10. However, if we run this code, it does not print 10; it prints 2.
Source code
"},{"location":"#ignoring-the-impacts-of-using-pointer-elements-in-range-loops-32","title":"Ignoring the impacts of using pointer elements in range loops (#32)","text":"TL;DR
Using a local variable or accessing an element using an index, you can prevent mistakes while copying pointers inside a loop.
When iterating over a data structure using a range loop, we must recall that all the values are assigned to a unique variable with a single unique address. Therefore, if we store a pointer referencing this variable during each iteration, we will end up in a situation where we store the same pointer referencing the same element: the latest one. We can overcome this issue by forcing the creation of a local variable in the loop\u2019s scope or creating a pointer referencing a slice element via its index. Both solutions are fine.
Source code
"},{"location":"#making-wrong-assumptions-during-map-iterations-ordering-and-map-insert-during-iteration-33","title":"Making wrong assumptions during map iterations (ordering and map insert during iteration) (#33)","text":"TL;DR
To ensure predictable outputs when using maps, remember that a map data structure:
Doesn\u2019t order the data by keys
Doesn\u2019t preserve the insertion order
Doesn\u2019t have a deterministic iteration order
Doesn\u2019t guarantee that an element added during an iteration will be produced during this iteration
Source code
"},{"location":"#ignoring-how-the-break-statement-works-34","title":"Ignoring how the break statement works (#34)","text":"TL;DR
Using break or continue with a label enforces breaking a specific statement. This can be helpful with switch or select statements inside loops.
A break statement is commonly used to terminate the execution of a loop. When loops are used in conjunction with switch or select, developers frequently make the mistake of breaking the wrong statement. For example:
for i := 0; i < 5; i++ {\nfmt.Printf(\"%d \", i)\nswitch i {\ndefault:\ncase 2:\nbreak\n}\n}\n
The break statement doesn\u2019t terminate the for loop: it terminates the switch statement, instead. Hence, instead of iterating from 0 to 2, this code iterates from 0 to 4: 0 1 2 3 4.
One essential rule to keep in mind is that a break statement terminates the execution of the innermost for, switch, or select statement. In the previous example, it terminates the switch statement.
To break the loop instead of the switch statement, the most idiomatic way is to use a label:
loop:\nfor i := 0; i < 5; i++ {\nfmt.Printf(\"%d \", i)\nswitch i {\ndefault:\ncase 2:\nbreak loop\n}\n}\n
Here, we associate the loop label with the for loop. Then, because we provide the loop label to the break statement, it breaks the loop, not the switch. Therefore, this new version will print 0 1 2, as we expected.
Source code
"},{"location":"#using-defer-inside-a-loop-35","title":"Using defer inside a loop (#35)","text":"TL;DR
Extracting loop logic inside a function leads to executing a defer statement at the end of each iteration.
The defer statement delays a call\u2019s execution until the surrounding function returns. It\u2019s mainly used to reduce boilerplate code. For example, if a resource has to be closed eventually, we can use defer to avoid repeating the closure calls before every single return.
One common mistake with defer is to forget that it schedules a function call when the surrounding function returns. For example:
func readFiles(ch <-chan string) error {\nfor path := range ch {\nfile, err := os.Open(path)\nif err != nil {\nreturn err\n}\ndefer file.Close()\n// Do something with file\n}\nreturn nil\n}\n
The defer calls are executed not during each loop iteration but when the readFiles function returns. If readFiles doesn\u2019t return, the file descriptors will be kept open forever, causing leaks.
One common option to fix this problem is to create a surrounding function after defer, called during each iteration:
Another solution is to make the readFile function a closure but intrinsically, this remains the same solution: adding another surrounding function to execute the defer calls during each iteration.
Source code
"},{"location":"#strings","title":"Strings","text":""},{"location":"#not-understanding-the-concept-of-rune-36","title":"Not understanding the concept of rune (#36)","text":"TL;DR
Understanding that a rune corresponds to the concept of a Unicode code point and that it can be composed of multiple bytes should be part of the Go developer\u2019s core knowledge to work accurately with strings.
As runes are everywhere in Go, it's important to understand the following:
A charset is a set of characters, whereas an encoding describes how to translate a charset into binary.
In Go, a string references an immutable slice of arbitrary bytes.
Go source code is encoded using UTF-8. Hence, all string literals are UTF-8 strings. But because a string can contain arbitrary bytes, if it\u2019s obtained from somewhere else (not the source code), it isn\u2019t guaranteed to be based on the UTF-8 encoding.
A rune corresponds to the concept of a Unicode code point, meaning an item represented by a single value.
Using UTF-8, a Unicode code point can be encoded into 1 to 4 bytes.
Using len() on a string in Go returns the number of bytes, not the number of runes.
Iterating on a string with the range operator iterates on the runes with the index corresponding to the starting index of the rune\u2019s byte sequence. To access a specific rune index (such as the third rune), convert the string into a []rune.
Iterating on a string is a common operation for developers. Perhaps we want to perform an operation for each rune in the string or implement a custom function to search for a specific substring. In both cases, we have to iterate on the different runes of a string. But it\u2019s easy to get confused about how iteration works.
For example, consider the following example:
s := \"h\u00eallo\"\nfor i := range s {\nfmt.Printf(\"position %d: %c\\n\", i, s[i])\n}\nfmt.Printf(\"len=%d\\n\", len(s))\n
Let's highlight three points that might be confusing:
The second rune is \u00c3 in the output instead of \u00ea.
We jumped from position 1 to position 3: what is at position 2?
len returns a count of 6, whereas s contains only 5 runes.
Let\u2019s start with the last observation. We already mentioned that len returns the number of bytes in a string, not the number of runes. Because we assigned a string literal to s, s is a UTF-8 string. Meanwhile, the special character \"\u00ea\" isn\u2019t encoded in a single byte; it requires 2 bytes. Therefore, calling len(s) returns 6.
Meanwhile, in the previous example, we have to understand that we don't iterate over each rune; instead, we iterate over each starting index of a rune:
Printing s[i] doesn\u2019t print the ith rune; it prints the UTF-8 representation of the byte at index i. Hence, we printed \"h\u00c3llo\" instead of \"h\u00eallo\".
If we want to print all the different runes, we can either use the value element of the range operator:
s := \"h\u00eallo\"\nfor i, r := range s {\nfmt.Printf(\"position %d: %c\\n\", i, r)\n}\n
Or, we can convert the string into a slice of runes and iterate over it:
s := \"h\u00eallo\"\nrunes := []rune(s)\nfor i, r := range runes {\nfmt.Printf(\"position %d: %c\\n\", i, r)\n}\n
Note that this solution introduces a run-time overhead compared to the previous one. Indeed, converting a string into a slice of runes requires allocating an additional slice and converting the bytes into runes: an O(n) time complexity with n the number of bytes in the string. Therefore, if we want to iterate over all the runes, we should use the first solution.
However, if we want to access the ith rune of a string with the first option, we don\u2019t have access to the rune index; rather, we know the starting index of a rune in the byte sequence.
s := \"h\u00eallo\"\nr := []rune(s)[4]\nfmt.Printf(\"%c\\n\", r) // o\n
Source code
"},{"location":"#misusing-trim-functions-38","title":"Misusing trim functions (#38)","text":"TL;DR
strings.TrimRight/strings.TrimLeft removes all the trailing/leading runes contained in a given set, whereas strings.TrimSuffix/strings.TrimPrefix returns a string without a provided suffix/prefix.
Concatenating a list of strings should be done with strings.Builder to prevent allocating a new string during each iteration.
Let\u2019s consider a concat function that concatenates all the string elements of a slice using the += operator:
func concat(values []string) string {\ns := \"\"\nfor _, value := range values {\ns += value\n}\nreturn s\n}\n
During each iteration, the += operator concatenates s with the value string. At first sight, this function may not look wrong. But with this implementation, we forget one of the core characteristics of a string: its immutability. Therefore, each iteration doesn\u2019t update s; it reallocates a new string in memory, which significantly impacts the performance of this function.
Fortunately, there is a solution to deal with this problem, using strings.Builder:
func concat(values []string) string {\nsb := strings.Builder{}\nfor _, value := range values {\n_, _ = sb.WriteString(value)\n}\nreturn sb.String()\n}\n
During each iteration, we constructed the resulting string by calling the WriteString method that appends the content of value to its internal buffer, hence minimizing memory copying.
Note
WriteString returns an error as the second output, but we purposely ignore it. Indeed, this method will never return a non-nil error. So what\u2019s the purpose of this method returning an error as part of its signature? strings.Builder implements the io.StringWriter interface, which contains a single method: WriteString(s string) (n int, err error). Hence, to comply with this interface, WriteString must return an error.
Internally, strings.Builder holds a byte slice. Each call to WriteString results in a call to append on this slice. There are two impacts. First, this struct shouldn\u2019t be used concurrently, as the calls to append would lead to race conditions. The second impact is something that we saw in mistake #21, \"Inefficient slice initialization\": if the future length of a slice is already known, we should preallocate it. For that purpose, strings.Builder exposes a method Grow(n int) to guarantee space for another n bytes:
func concat(values []string) string {\ntotal := 0\nfor i := 0; i < len(values); i++ {\ntotal += len(values[i])\n}\nsb := strings.Builder{}\nsb.Grow(total) (2)\nfor _, value := range values {\n_, _ = sb.WriteString(value)\n}\nreturn sb.String()\n}\n
Let\u2019s run a benchmark to compare the three versions (v1 using +=; v2 using strings.Builder{} without preallocation; and v3 using strings.Builder{} with preallocation). The input slice contains 1,000 strings, and each string contains 1,000 bytes:
As we can see, the latest version is by far the most efficient: 99% faster than v1 and 78% faster than v2.
strings.Builder is the recommended solution to concatenate a list of strings. Usually, this solution should be used within a loop. Indeed, if we just have to concatenate a few strings (such as a name and a surname), using strings.Builder is not recommended as doing so will make the code a bit less readable than using the += operator or fmt.Sprintf.
Remembering that the bytes package offers the same operations as the strings package can help avoid extra byte/string conversions.
When choosing to work with a string or a []byte, most programmers tend to favor strings for convenience. But most I/O is actually done with []byte. For example, io.Reader, io.Writer, and io.ReadAll work with []byte, not strings.
When we\u2019re wondering whether we should work with strings or []byte, let\u2019s recall that working with []byte isn\u2019t necessarily less convenient. Indeed, all the exported functions of the strings package also have alternatives in the bytes package: Split, Count, Contains, Index, and so on. Hence, whether we\u2019re doing I/O or not, we should first check whether we could implement a whole workflow using bytes instead of strings and avoid the price of additional conversions.
Source code
"},{"location":"#substring-and-memory-leaks-41","title":"Substring and memory leaks (#41)","text":"TL;DR
Using copies instead of substrings can prevent memory leaks, as the string returned by a substring operation will be backed by the same byte array.
In mistake #26, \u201cSlices and memory leaks,\u201d we saw how slicing a slice or array may lead to memory leak situations. This principle also applies to string and substring operations.
We need to keep two things in mind while using the substring operation in Go. First, the interval provided is based on the number of bytes, not the number of runes. Second, a substring operation may lead to a memory leak as the resulting substring will share the same backing array as the initial string. The solutions to prevent this case from happening are to perform a string copy manually or to use strings.Clone from Go 1.18.
Source code
"},{"location":"#functions-and-methods","title":"Functions and Methods","text":""},{"location":"#not-knowing-which-type-of-receiver-to-use-42","title":"Not knowing which type of receiver to use (#42)","text":"TL;DR
The decision whether to use a value or a pointer receiver should be made based on factors such as the type, whether it has to be mutated, whether it contains a field that can\u2019t be copied, and how large the object is. When in doubt, use a pointer receiver.
Choosing between value and pointer receivers isn\u2019t always straightforward. Let\u2019s discuss some of the conditions to help us choose.
A receiver must be a pointer
If the method needs to mutate the receiver. This rule is also valid if the receiver is a slice and a method needs to append elements:
type slice []int\nfunc (s *slice) add(element int) {\n*s = append(*s, element)\n}\n
If the method receiver contains a field that cannot be copied: for example, a type part of the sync package (see #74, \u201cCopying a sync type\u201d).
A receiver should be a pointer
If the receiver is a large object. Using a pointer can make the call more efficient, as doing so prevents making an extensive copy. When in doubt about how large is large, benchmarking can be the solution; it\u2019s pretty much impossible to state a specific size, because it depends on many factors.
A receiver must be a value
If we have to enforce a receiver\u2019s immutability.
If the receiver is a map, function, or channel. Otherwise, a compilation error occurs.
A receiver should be a value
If the receiver is a slice that doesn\u2019t have to be mutated.
If the receiver is a small array or struct that is naturally a value type without mutable fields, such as time.Time.
If the receiver is a basic type such as int, float64, or string.
Of course, it\u2019s impossible to be exhaustive, as there will always be edge cases, but this section\u2019s goal was to provide guidance to cover most cases. By default, we can choose to go with a value receiver unless there\u2019s a good reason not to do so. In doubt, we should use a pointer receiver.
Source code
"},{"location":"#never-using-named-result-parameters-43","title":"Never using named result parameters (#43)","text":"TL;DR
Using named result parameters can be an efficient way to improve the readability of a function/method, especially if multiple result parameters have the same type. In some cases, this approach can also be convenient because named result parameters are initialized to their zero value. But be cautious about potential side effects.
When we return parameters in a function or a method, we can attach names to these parameters and use them as regular variables. When a result parameter is named, it\u2019s initialized to its zero value when the function/method begins. With named result parameters, we can also call a naked return statement (without arguments). In that case, the current values of the result parameters are used as the returned values.
Here\u2019s an example that uses a named result parameter b:
func f(a int) (b int) {\nb = a\nreturn\n}\n
In this example, we attach a name to the result parameter: b. When we call return without arguments, it returns the current value of b.
In some cases, named result parameters can also increase readability: for example, if two parameters have the same type. In other cases, they can also be used for convenience. Therefore, we should use named result parameters sparingly when there\u2019s a clear benefit.
Source code
"},{"location":"#unintended-side-effects-with-named-result-parameters-44","title":"Unintended side effects with named result parameters (#44)","text":"TL;DR
See #43.
We mentioned why named result parameters can be useful in some situations. But as these result parameters are initialized to their zero value, using them can sometimes lead to subtle bugs if we\u2019re not careful enough. For example, can you spot what\u2019s wrong with this code?
The error might not be obvious at first glance. Here, the error returned in the if ctx.Err() != nil scope is err. But we haven\u2019t assigned any value to the err variable. It\u2019s still assigned to the zero value of an error type: nil. Hence, this code will always return a nil error.
When using named result parameters, we must recall that each parameter is initialized to its zero value. As we have seen in this section, this can lead to subtle bugs that aren\u2019t always straightforward to spot while reading code. Therefore, let\u2019s remain cautious when using named result parameters, to avoid potential side effects.
Source code
"},{"location":"#returning-a-nil-receiver-45","title":"Returning a nil receiver (#45)","text":"TL;DR
When returning an interface, be cautious about not returning a nil pointer but an explicit nil value. Otherwise, unintended consequences may occur and the caller will receive a non-nil value.
Source code
"},{"location":"#using-a-filename-as-a-function-input-46","title":"Using a filename as a function input (#46)","text":"TL;DR
Designing functions to receive io.Reader types instead of filenames improves the reusability of a function and makes testing easier.
Accepting a filename as a function input to read from a file should, in most cases, be considered a code smell (except in specific functions such as os.Open). Indeed, it makes unit tests more complex because we may have to create multiple files. It also reduces the reusability of a function (although not all functions are meant to be reused). Using the io.Reader interface abstracts the data source. Regardless of whether the input is a file, a string, an HTTP request, or a gRPC request, the implementation can be reused and easily tested.
Source code
"},{"location":"#ignoring-how-defer-arguments-and-receivers-are-evaluated-argument-evaluation-pointer-and-value-receivers-47","title":"Ignoring how defer arguments and receivers are evaluated (argument evaluation, pointer, and value receivers) (#47)","text":"TL;DR
Passing a pointer to a defer function and wrapping a call inside a closure are two possible solutions to overcome the immediate evaluation of arguments and receivers.
In a defer function the arguments are evaluated right away, not once the surrounding function returns. For example, in this code, we always call notify and incrementCounter with the same status: an empty string.
Indeed, we call notify(status) and incrementCounter(status) as defer functions. Therefore, Go will delay these calls to be executed once f returns with the current value of status at the stage we used defer, hence passing an empty string.
Two leading options if we want to keep using defer.
The first solution is to pass a string pointer:
func f() error {\nvar status string\ndefer notify(&status) defer incrementCounter(&status)\n// The rest of the function unchanged\n}\n
Using defer evaluates the arguments right away: here, the address of status. Yes, status itself is modified throughout the function, but its address remains constant, regardless of the assignments. Hence, if notify or incrementCounter uses the value referenced by the string pointer, it will work as expected. But this solution requires changing the signature of the two functions, which may not always be possible.
There\u2019s another solution: calling a closure (an anonymous function value that references variables from outside its body) as a defer statement:
func f() error {\nvar status string\ndefer func() {\nnotify(status)\nincrementCounter(status)\n}()\n// The rest of the function unchanged\n}\n
Here, we wrap the calls to both notify and incrementCounter within a closure. This closure references the status variable from outside its body. Therefore, status is evaluated once the closure is executed, not when we call defer. This solution also works and doesn\u2019t require notify and incrementCounter to change their signature.
Let's also note this behavior applies with method receiver: the receiver is evaluated immediately.
Using panic is an option to deal with errors in Go. However, it should only be used sparingly in unrecoverable conditions: for example, to signal a programmer error or when you fail to load a mandatory dependency.
In Go, panic is a built-in function that stops the ordinary flow:
Panicking in Go should be used sparingly. There are two prominent cases, one to signal a programmer error (e.g., sql.Register that panics if the driver is nil or has already been register) and another where our application fails to create a mandatory dependency. Hence, exceptional conditions that lead us to stop the application. In most other cases, error management should be done with a function that returns a proper error type as the last return argument.
Source code
"},{"location":"#ignoring-when-to-wrap-an-error-49","title":"Ignoring when to wrap an error (#49)","text":"TL;DR
Wrapping an error allows you to mark an error and/or provide additional context. However, error wrapping creates potential coupling as it makes the source error available for the caller. If you want to prevent that, don\u2019t use error wrapping.
Since Go 1.13, the %w directive allows us to wrap errors conveniently. Error wrapping is about wrapping or packing an error inside a wrapper container that also makes the source error available. In general, the two main use cases for error wrapping are the following:
Adding additional context to an error
Marking an error as a specific error
When handling an error, we can decide to wrap it. Wrapping is about adding additional context to an error and/or marking an error as a specific type. If we need to mark an error, we should create a custom error type. However, if we just want to add extra context, we should use fmt.Errorf with the %w directive as it doesn\u2019t require creating a new error type. Yet, error wrapping creates potential coupling as it makes the source error available for the caller. If we want to prevent it, we shouldn\u2019t use error wrapping but error transformation, for example, using fmt.Errorf with the %v directive.
Source code
"},{"location":"#comparing-an-error-type-inaccurately-50","title":"Comparing an error type inaccurately (#50)","text":"TL;DR
If you use Go 1.13 error wrapping with the %w directive and fmt.Errorf, comparing an error against a type has to be done using errors.As. Otherwise, if the returned error you want to check is wrapped, it will fail the checks.
Source code
"},{"location":"#comparing-an-error-value-inaccurately-51","title":"Comparing an error value inaccurately (#51)","text":"TL;DR
If you use Go 1.13 error wrapping with the %w directive and fmt.Errorf, comparing an error against or a value has to be done using errors.As. Otherwise, if the returned error you want to check is wrapped, it will fail the checks.
A sentinel error is an error defined as a global variable:
In general, the convention is to start with Err followed by the error type: here, ErrFoo. A sentinel error conveys an expected error, an error that clients will expect to check. As general guidelines:
Expected errors should be designed as error values (sentinel errors): var ErrFoo = errors.New(\"foo\").
Unexpected errors should be designed as error types: type BarError struct { ... }, with BarError implementing the error interface.
If we use error wrapping in our application with the %w directive and fmt.Errorf, checking an error against a specific value should be done using errors.Is instead of ==. Thus, even if the sentinel error is wrapped, errors.Is can recursively unwrap it and compare each error in the chain against the provided value.
Source code
"},{"location":"#handling-an-error-twice-52","title":"Handling an error twice (#52)","text":"TL;DR
In most situations, an error should be handled only once. Logging an error is handling an error. Therefore, you have to choose between logging or returning an error. In many cases, error wrapping is the solution as it allows you to provide additional context to an error and return the source error.
Handling an error multiple times is a mistake made frequently by developers, not specifically in Go. This can cause situations where the same error is logged multiple times make debugging harder.
Let's remind us that handling an error should be done only once. Logging an error is handling an error. Hence, we should either log or return an error. By doing this, we simplify our code and gain better insights into the error situation. Using error wrapping is the most convenient approach as it allows us to propagate the source error and add context to an error.
Source code
"},{"location":"#not-handling-an-error-53","title":"Not handling an error (#53)","text":"TL;DR
Ignoring an error, whether during a function call or in a defer function, should be done explicitly using the blank identifier. Otherwise, future readers may be confused about whether it was intentional or a miss.
In many cases, you shouldn\u2019t ignore an error returned by a defer function. Either handle it directly or propagate it to the caller, depending on the context. If you want to ignore it, use the blank identifier.
From a maintainability perspective, the code can lead to some issues. Let\u2019s consider a new reader looking at it. This reader notices that notify returns an error but that the error isn\u2019t handled by the parent function. How can they guess whether or not handling the error was intentional? How can they know whether the previous developer forgot to handle it or did it purposely?
For these reasons, when we want to ignore an error, there's only one way to do it, using the blank identifier (_):
_ = notify\n
In terms of compilation and run time, this approach doesn\u2019t change anything compared to the first piece of code. But this new version makes explicit that we aren\u2019t interested in the error. Also, we can add a comment that indicates the rationale for why an error is ignored:
// At-most once delivery.\n// Hence, it's accepted to miss some of them in case of errors.\n_ = notify()\n
Source code
"},{"location":"#concurrency-foundations","title":"Concurrency: Foundations","text":""},{"location":"#mixing-up-concurrency-and-parallelism-55","title":"Mixing up concurrency and parallelism (#55)","text":"TL;DR
Understanding the fundamental differences between concurrency and parallelism is a cornerstone of the Go developer\u2019s knowledge. Concurrency is about structure, whereas parallelism is about execution.
Concurrency and parallelism are not the same:
Concurrency is about structure. We can change a sequential implementation into a concurrent one by introducing different steps that separate concurrent goroutines can tackle.
Meanwhile, parallelism is about execution. We can use parallism at the steps level by adding more parallel goroutines.
In summary, concurrency provides a structure to solve a problem with parts that may be parallelized. Therefore, concurrency enables parallelism.
"},{"location":"#thinking-concurrency-is-always-faster-56","title":"Thinking concurrency is always faster (#56)","text":"TL;DR
To be a proficient developer, you must acknowledge that concurrency isn\u2019t always faster. Solutions involving parallelization of minimal workloads may not necessarily be faster than a sequential implementation. Benchmarking sequential versus concurrent solutions should be the way to validate assumptions.
Read the full section here.
Source code
"},{"location":"#being-puzzled-about-when-to-use-channels-or-mutexes-57","title":"Being puzzled about when to use channels or mutexes (#57)","text":"TL;DR
Being aware of goroutine interactions can also be helpful when deciding between channels and mutexes. In general, parallel goroutines require synchronization and hence mutexes. Conversely, concurrent goroutines generally require coordination and orchestration and hence channels.
Given a concurrency problem, it may not always be clear whether we can implement a solution using channels or mutexes. Because Go promotes sharing memory by communication, one mistake could be to always force the use of channels, regardless of the use case. However, we should see the two options as complementary.
When should we use channels or mutexes? We will use the example in the next figure as a backbone. Our example has three different goroutines with specific relationships:
G1 and G2 are parallel goroutines. They may be two goroutines executing the same function that keeps receiving messages from a channel, or perhaps two goroutines executing the same HTTP handler at the same time.
On the other hand, G1 and G3 are concurrent goroutines, as are G2 and G3. All the goroutines are part of an overall concurrent structure, but G1 and G2 perform the first step, whereas G3 does the next step.
In general, parallel goroutines have to synchronize: for example, when they need to access or mutate a shared resource such as a slice. Synchronization is enforced with mutexes but not with any channel types (not with buffered channels). Hence, in general, synchronization between parallel goroutines should be achieved via mutexes.
Conversely, in general, concurrent goroutines have to coordinate and orchestrate. For example, if G3 needs to aggregate results from both G1 and G2, G1 and G2 need to signal to G3 that a new intermediate result is available. This coordination falls under the scope of communication\u2014therefore, channels.
Regarding concurrent goroutines, there\u2019s also the case where we want to transfer the ownership of a resource from one step (G1 and G2) to another (G3); for example, if G1 and G2 are enriching a shared resource and at some point, we consider this job as complete. Here, we should use channels to signal that a specific resource is ready and handle the ownership transfer.
Mutexes and channels have different semantics. Whenever we want to share a state or access a shared resource, mutexes ensure exclusive access to this resource. Conversely, channels are a mechanic for signaling with or without data (chan struct{} or not). Coordination or ownership transfer should be achieved via channels. It\u2019s important to know whether goroutines are parallel or concurrent because, in general, we need mutexes for parallel goroutines and channels for concurrent ones.
"},{"location":"#not-understanding-race-problems-data-races-vs-race-conditions-and-the-go-memory-model-58","title":"Not understanding race problems (data races vs. race conditions and the Go memory model) (#58)","text":"TL;DR
Being proficient in concurrency also means understanding that data races and race conditions are different concepts. Data races occur when multiple goroutines simultaneously access the same memory location and at least one of them is writing. Meanwhile, being data-race-free doesn\u2019t necessarily mean deterministic execution. When a behavior depends on the sequence or the timing of events that can\u2019t be controlled, this is a race condition.
Race problems can be among the hardest and most insidious bugs a programmer can face. As Go developers, we must understand crucial aspects such as data races and race conditions, their possible impacts, and how to avoid them.
A data race occurs when two or more goroutines simultaneously access the same memory location and at least one is writing. In this case, the result can be hazardous. Even worse, in some situations, the memory location may end up holding a value containing a meaningless combination of bits.
We can prevent a data race from happening using different techniques. For example:
Using the sync/atomic package
In synchronizing the two goroutines with an ad how data structure like a mutex
Using channels to make the two goroutines communicating to ensure that a variable is updated by only one goroutine at a time
Depending on the operation we want to perform, does a data-race-free application necessarily mean a deterministic result? Not necessarily.
A race condition occurs when the behavior depends on the sequence or the timing of events that can\u2019t be controlled. Here, the timing of events is the goroutines\u2019 execution order.
In summary, when we work in concurrent applications, it\u2019s essential to understand that a data race is different from a race condition. A data race occurs when multiple goroutines simultaneously access the same memory location and at least one of them is writing. A data race means unexpected behavior. However, a data-race-free application doesn\u2019t necessarily mean deterministic results. An application can be free of data races but still have behavior that depends on uncontrolled events (such as goroutine execution, how fast a message is published to a channel, or how long a call to a database lasts); this is a race condition. Understanding both concepts is crucial to becoming proficient in designing concurrent applications.
Source code
"},{"location":"#not-understanding-the-concurrency-impacts-of-a-workload-type-59","title":"Not understanding the concurrency impacts of a workload type (#59)","text":"TL;DR
When creating a certain number of goroutines, consider the workload type. Creating CPU-bound goroutines means bounding this number close to the GOMAXPROCS variable (based by default on the number of CPU cores on the host). Creating I/O-bound goroutines depends on other factors, such as the external system.
In programming, the execution time of a workload is limited by one of the following:
The speed of the CPU\u2014For example, running a merge sort algorithm. The workload is called CPU-bound.
The speed of I/O\u2014For example, making a REST call or a database query. The workload is called I/O-bound.
The amount of available memory\u2014The workload is called memory-bound.
Note
The last is the rarest nowadays, given that memory has become very cheap in recent decades. Hence, this section focuses on the two first workload types: CPU- and I/O-bound.
If the workload executed by the workers is I/O-bound, the value mainly depends on the external system. Conversely, if the workload is CPU-bound, the optimal number of goroutines is close to the number of available CPU cores (a best practice can be to use runtime.GOMAXPROCS). Knowing the workload type (I/O or CPU) is crucial when designing concurrent applications.
Source code
"},{"location":"#misunderstanding-go-contexts-60","title":"Misunderstanding Go contexts (#60)","text":"TL;DR
Go contexts are also one of the cornerstones of concurrency in Go. A context allows you to carry a deadline, a cancellation signal, and/or a list of keys-values.
https://pkg.go.dev/context
A Context carries a deadline, a cancellation signal, and other values across API boundaries.
A deadline refers to a specific point in time determined with one of the following:
A time.Duration from now (for example, in 250 ms)
A time.Time (for example, 2023-02-07 00:00:00 UTC)
The semantics of a deadline convey that an ongoing activity should be stopped if this deadline is met. An activity is, for example, an I/O request or a goroutine waiting to receive a message from a channel.
Another use case for Go contexts is to carry a cancellation signal. Let\u2019s imagine that we want to create an application that calls CreateFileWatcher(ctx context.Context, filename string) within another goroutine. This function creates a specific file watcher that keeps reading from a file and catches updates. When the provided context expires or is canceled, this function handles it to close the file descriptor.
The last use case for Go contexts is to carry a key-value list. What\u2019s the point of having a context carrying a key-value list? Because Go contexts are generic and mainstream, there are infinite use cases.
For example, if we use tracing, we may want different subfunctions to share the same correlation ID. Some developers may consider this ID too invasive to be part of the function signature. In this regard, we could also decide to include it as part of the provided context.
"},{"location":"#catching-a-context-cancellation","title":"Catching a context cancellation","text":"
The context.Context type exports a Done method that returns a receive-only notification channel: <-chan struct{}. This channel is closed when the work associated with the context should be canceled. For example,
The Done channel related to a context created with context.WithCancel is closed when the cancel function is called.
The Done channel related to a context created with context.WithDeadline is closed when the deadline has expired.
One thing to note is that the internal channel should be closed when a context is canceled or has met a deadline, instead of when it receives a specific value, because the closure of a channel is the only channel action that all the consumer goroutines will receive. This way, all the consumers will be notified once a context is canceled or a deadline is reached.
In summary, to be a proficient Go developer, we have to understand what a context is and how to use it. In general, a function that users wait for should take a context, as doing so allows upstream callers to decide when calling this function should be aborted.
Source code
"},{"location":"#concurrency-practice","title":"Concurrency: Practice","text":""},{"location":"#propagating-an-inappropriate-context-61","title":"Propagating an inappropriate context (#61)","text":"TL;DR
Understanding the conditions when a context can be canceled should matter when propagating it: for example, an HTTP handler canceling the context when the response has been sent.
In many situations, it is recommended to propagate Go contexts. However, context propagation can sometimes lead to subtle bugs, preventing subfunctions from being correctly executed.
Let\u2019s consider the following example. We expose an HTTP handler that performs some tasks and returns a response. But just before returning the response, we also want to send it to a Kafka topic. We don\u2019t want to penalize the HTTP consumer latency-wise, so we want the publish action to be handled asynchronously within a new goroutine. We assume that we have at our disposal a publish function that accepts a context so the action of publishing a message can be interrupted if the context is canceled, for example. Here is a possible implementation:
func handler(w http.ResponseWriter, r *http.Request) {\nresponse, err := doSomeTask(r.Context(), r)\nif err != nil {\nhttp.Error(w, err.Error(), http.StatusInternalServerError)\nreturn\n}\ngo func() {\nerr := publish(r.Context(), response)\n// Do something with err\n}()\nwriteResponse(response)\n}\n
What\u2019s wrong with this piece of code? We have to know that the context attached to an HTTP request can cancel in different conditions:
When the client\u2019s connection closes
In the case of an HTTP/2 request, when the request is canceled
When the response has been written back to the client
In the first two cases, we probably handle things correctly. For example, if we get a response from doSomeTask but the client has closed the connection, it\u2019s probably OK to call publish with a context already canceled so the message isn\u2019t published. But what about the last case?
When the response has been written to the client, the context associated with the request will be canceled. Therefore, we are facing a race condition:
If the response is written after the Kafka publication, we both return a response and publish a message successfully
However, if the response is written before or during the Kafka publication, the message shouldn\u2019t be published.
In the latter case, calling publish will return an error because we returned the HTTP response quickly.
Note
From Go 1.21, there is a way to create a new context without cancel. context.WithoutCancel returns a copy of parent that is not canceled when parent is canceled.
In summary, propagating a context should be done cautiously.
Source code
"},{"location":"#starting-a-goroutine-without-knowing-when-to-stop-it-62","title":"Starting a goroutine without knowing when to stop it (#62)","text":"TL;DR
Avoiding leaks means being mindful that whenever a goroutine is started, you should have a plan to stop it eventually.
Goroutines are easy and cheap to start\u2014so easy and cheap that we may not necessarily have a plan for when to stop a new goroutine, which can lead to leaks. Not knowing when to stop a goroutine is a design issue and a common concurrency mistake in Go.
Let\u2019s discuss a concrete example. We will design an application that needs to watch some external configuration (for example, using a database connection). Here\u2019s a first implementation:
func main() {\nnewWatcher()\n// Run the application\n}\ntype watcher struct { /* Some resources */ }\nfunc newWatcher() {\nw := watcher{}\ngo w.watch() // Creates a goroutine that watches some external configuration\n}\n
The problem with this code is that when the main goroutine exits (perhaps because of an OS signal or because it has a finite workload), the application is stopped. Hence, the resources created by watcher aren\u2019t closed gracefully. How can we prevent this from happening?
One option could be to pass to newWatcher a context that will be canceled when main returns:
func main() {\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nnewWatcher(ctx)\n// Run the application\n}\nfunc newWatcher(ctx context.Context) {\nw := watcher{}\ngo w.watch(ctx)\n}\n
We propagate the context created to the watch method. When the context is canceled, the watcher struct should close its resources. However, can we guarantee that watch will have time to do so? Absolutely not\u2014and that\u2019s a design flaw.
The problem is that we used signaling to convey that a goroutine had to be stopped. We didn\u2019t block the parent goroutine until the resources had been closed. Let\u2019s make sure we do:
func main() {\nw := newWatcher()\ndefer w.close()\n// Run the application\n}\nfunc newWatcher() watcher {\nw := watcher{}\ngo w.watch()\nreturn w\n}\nfunc (w watcher) close() {\n// Close the resources\n}\n
Instead of signaling watcher that it\u2019s time to close its resources, we now call this close method, using defer to guarantee that the resources are closed before the application exits.
In summary, let\u2019s be mindful that a goroutine is a resource like any other that must eventually be closed to free memory or other resources. Starting a goroutine without knowing when to stop it is a design issue. Whenever a goroutine is started, we should have a clear plan about when it will stop. Last but not least, if a goroutine creates resources and its lifetime is bound to the lifetime of the application, it\u2019s probably safer to wait for this goroutine to complete before exiting the application. This way, we can ensure that the resources can be freed.
Source code
"},{"location":"#not-being-careful-with-goroutines-and-loop-variables-63","title":"Not being careful with goroutines and loop variables (#63)","text":"Warning
This mistake isn't relevant anymore from Go 1.22 (source).
"},{"location":"#expecting-a-deterministic-behavior-using-select-and-channels-64","title":"Expecting a deterministic behavior using select and channels (#64)","text":"TL;DR
Understanding that select with multiple channels chooses the case randomly if multiple options are possible prevents making wrong assumptions that can lead to subtle concurrency bugs.
One common mistake made by Go developers while working with channels is to make wrong assumptions about how select behaves with multiple channels.
For example, let's consider the following case (disconnectCh is a unbuffered channel):
go func() {\nfor i := 0; i < 10; i++ {\nmessageCh <- i\n}\ndisconnectCh <- struct{}{}\n}()\nfor {\nselect {\ncase v := <-messageCh:\nfmt.Println(v)\ncase <-disconnectCh:\nfmt.Println(\"disconnection, return\")\nreturn\n}\n}\n
If we run this example multiple times, the result will be random:
Instead of consuming the 10 messages, we only received a few of them. What\u2019s the reason? It lies in the specification of the select statement with multiple channels (https:// go.dev/ref/spec):
Quote
If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.
Unlike a switch statement, where the first case with a match wins, the select state- ment selects randomly if multiple options are possible.
This behavior might look odd at first, but there\u2019s a good reason for it: to prevent possible starvation. Suppose the first possible communication chosen is based on the source order. In that case, we may fall into a situation where, for example, we only receive from one channel because of a fast sender. To prevent this, the language designers decided to use a random selection.
When using select with multiple channels, we must remember that if multiple options are possible, the first case in the source order does not automatically win. Instead, Go selects randomly, so there\u2019s no guarantee about which option will be chosen. To overcome this behavior, in the case of a single producer goroutine, we can use either unbuffered channels or a single channel.
Source code
"},{"location":"#not-using-notification-channels-65","title":"Not using notification channels (#65)","text":"TL;DR
Send notifications using a chan struct{} type.
Channels are a mechanism for communicating across goroutines via signaling. A signal can be either with or without data.
Let\u2019s look at a concrete example. We will create a channel that will notify us whenever a certain disconnection occurs. One idea is to handle it as a chan bool:
disconnectCh := make(chan bool)\n
Now, let\u2019s say we interact with an API that provides us with such a channel. Because it\u2019s a channel of Booleans, we can receive either true or false messages. It\u2019s probably clear what true conveys. But what does false mean? Does it mean we haven\u2019t been disconnected? And in this case, how frequently will we receive such a signal? Does it mean we have reconnected? Should we even expect to receive false? Perhaps we should only expect to receive true messages.
If that\u2019s the case, meaning we don\u2019t need a specific value to convey some information, we need a channel without data. The idiomatic way to handle it is a channel of empty structs: chan struct{}.
"},{"location":"#not-using-nil-channels-66","title":"Not using nil channels (#66)","text":"TL;DR
Using nil channels should be part of your concurrency toolset because it allows you to remove cases from select statements, for example.
What should this code do?
var ch chan int\n<-ch\n
ch is a chan int type. The zero value of a channel being nil, ch is nil. The goroutine won\u2019t panic; however, it will block forever.
The principle is the same if we send a message to a nil channel. This goroutine blocks forever:
var ch chan int\nch <- 0\n
Then what\u2019s the purpose of Go allowing messages to be received from or sent to a nil channel? For example, we can use nil channels to implement an idiomatic way to merge two channels:
func merge(ch1, ch2 <-chan int) <-chan int {\nch := make(chan int, 1)\ngo func() {\nfor ch1 != nil || ch2 != nil { // Continue if at least one channel isn\u2019t nil\nselect {\ncase v, open := <-ch1:\nif !open {\nch1 = nil // Assign ch1 to a nil channel once closed\nbreak\n}\nch <- v\ncase v, open := <-ch2:\nif !open {\nch2 = nil // Assigns ch2 to a nil channel once closed\nbreak\n}\nch <- v\n}\n}\nclose(ch)\n}()\nreturn ch\n}\n
This elegant solution relies on nil channels to somehow remove one case from the select statement.
Let\u2019s keep this idea in mind: nil channels are useful in some conditions and should be part of the Go developer\u2019s toolset when dealing with concurrent code.
Source code
"},{"location":"#being-puzzled-about-channel-size-67","title":"Being puzzled about channel size (#67)","text":"TL;DR
Carefully decide on the right channel type to use, given a problem. Only unbuffered channels provide strong synchronization guarantees.
You should have a good reason to specify a channel size other than one for buffered channels.
"},{"location":"#forgetting-about-possible-side-effects-with-string-formatting-etcd-data-race-example-and-deadlock-68","title":"Forgetting about possible side effects with string formatting (etcd data race example and deadlock) (#68)","text":"TL;DR
Being aware that string formatting may lead to calling existing functions means watching out for possible deadlocks and other data races.
Source code
"},{"location":"#creating-data-races-with-append-69","title":"Creating data races with append (#69)","text":"TL;DR
Calling append isn\u2019t always data-race-free; hence, it shouldn\u2019t be used concurrently on a shared slice.
Source code
"},{"location":"#using-mutexes-inaccurately-with-slices-and-maps-70","title":"Using mutexes inaccurately with slices and maps (#70)","text":"TL;DR
Remembering that slices and maps are pointers can prevent common data races.
To accurately use sync.WaitGroup, call the Add method before spinning up goroutines.
Source code
"},{"location":"#forgetting-about-synccond-72","title":"Forgetting about sync.Cond (#72)","text":"TL;DR
You can send repeated notifications to multiple goroutines with sync.Cond.
Source code
"},{"location":"#not-using-errgroup-73","title":"Not using errgroup (#73)","text":"TL;DR
You can synchronize a group of goroutines and handle errors and contexts with the errgroup package.
Source code
"},{"location":"#copying-a-sync-type-74","title":"Copying a sync type (#74)","text":"TL;DR
sync types shouldn\u2019t be copied.
Source code
"},{"location":"#standard-library","title":"Standard Library","text":""},{"location":"#providing-a-wrong-time-duration-75","title":"Providing a wrong time duration (#75)","text":"TL;DR
Remain cautious with functions accepting a time.Duration. Even though passing an integer is allowed, strive to use the time API to prevent any possible confusion.
Source code
"},{"location":"#timeafter-and-memory-leaks-76","title":"time.After and memory leaks (#76)","text":"TL;DR
Avoiding calls to time.After in repeated functions (such as loops or HTTP handlers) can avoid peak memory consumption. The resources created by time.After are released only when the timer expires.
Source code
"},{"location":"#json-handling-common-mistakes-77","title":"JSON handling common mistakes (#77)","text":"
Unexpected behavior because of type embedding
Be careful about using embedded fields in Go structs. Doing so may lead to sneaky bugs like an embedded time.Time field implementing the json.Marshaler interface, hence overriding the default marshaling behavior.
Source code
JSON and the monotonic clock
When comparing two time.Time structs, recall that time.Time contains both a wall clock and a monotonic clock, and the comparison using the == operator is done on both clocks.
Source code
Map of any
To avoid wrong assumptions when you provide a map while unmarshaling JSON data, remember that numerics are converted to float64 by default.
Forgetting that sql.Open doesn't necessarily establish connections to a database
Call the Ping or PingContext method if you need to test your configuration and make sure a database is reachable.
Source code
Forgetting about connections pooling
Configure the database connection parameters for production-grade applications.
Not using prepared statements
Using SQL prepared statements makes queries more efficient and more secure.
Source code
Mishandling null values
Deal with nullable columns in tables using pointers or sql.NullXXX types.
Source code
Not handling rows iteration errors
Call the Err method of sql.Rows after row iterations to ensure that you haven\u2019t missed an error while preparing the next row.
Source code
"},{"location":"#not-closing-transient-resources-http-body-sqlrows-and-osfile-79","title":"Not closing transient resources (HTTP body, sql.Rows, and os.File) (#79)","text":"TL;DR
Eventually close all structs implementing io.Closer to avoid possible leaks.
Source code
"},{"location":"#forgetting-the-return-statement-after-replying-to-an-http-request-80","title":"Forgetting the return statement after replying to an HTTP request (#80)","text":"TL;DR
To avoid unexpected behaviors in HTTP handler implementations, make sure you don\u2019t miss the return statement if you want a handler to stop after http.Error.
Source code
"},{"location":"#using-the-default-http-client-and-server-81","title":"Using the default HTTP client and server (#81)","text":"TL;DR
For production-grade applications, don\u2019t use the default HTTP client and server implementations. These implementations are missing timeouts and behaviors that should be mandatory in production.
Source code
"},{"location":"#testing","title":"Testing","text":""},{"location":"#not-categorizing-tests-build-tags-environment-variables-and-short-mode-82","title":"Not categorizing tests (build tags, environment variables, and short mode) (#82)","text":"TL;DR
Categorizing tests using build flags, environment variables, or short mode makes the testing process more efficient. You can create test categories using build flags or environment variables (for example, unit versus integration tests) and differentiate short from long-running tests to decide which kinds of tests to execute.
Source code
"},{"location":"#not-enabling-the-race-flag-83","title":"Not enabling the race flag (#83)","text":"TL;DR
Enabling the -race flag is highly recommended when writing concurrent applications. Doing so allows you to catch potential data races that can lead to software bugs.
"},{"location":"#not-using-test-execution-modes-parallel-and-shuffle-84","title":"Not using test execution modes (parallel and shuffle) (#84)","text":"TL;DR
Using the -parallel flag is an efficient way to speed up tests, especially long-running ones. Use the -shuffle flag to help ensure that a test suite doesn\u2019t rely on wrong assumptions that could hide bugs.
"},{"location":"#not-using-table-driven-tests-85","title":"Not using table-driven tests (#85)","text":"TL;DR
Table-driven tests are an efficient way to group a set of similar tests to prevent code duplication and make future updates easier to handle.
Source code
"},{"location":"#sleeping-in-unit-tests-86","title":"Sleeping in unit tests (#86)","text":"TL;DR
Avoid sleeps using synchronization to make a test less flaky and more robust. If synchronization isn\u2019t possible, consider a retry approach.
Source code
"},{"location":"#not-dealing-with-the-time-api-efficiently-87","title":"Not dealing with the time API efficiently (#87)","text":"TL;DR
Understanding how to deal with functions using the time API is another way to make a test less flaky. You can use standard techniques such as handling the time as part of a hidden dependency or asking clients to provide it.
Source code
"},{"location":"#not-using-testing-utility-packages-httptest-and-iotest-88","title":"Not using testing utility packages (httptest and iotest) (#88)","text":"
The httptest package is helpful for dealing with HTTP applications. It provides a set of utilities to test both clients and servers.
Source code
The iotest package helps write io.Reader and test that an application is tolerant to errors.
Use time methods to preserve the accuracy of a benchmark.
Increasing benchtime or using tools such as benchstat can be helpful when dealing with micro-benchmarks.
Be careful with the results of a micro-benchmark if the system that ends up running the application is different from the one running the micro-bench- mark.
Make sure the function under test leads to a side effect, to prevent compiler optimizations from fooling you about the benchmark results.
To prevent the observer effect, force a benchmark to re-create the data used by a CPU-bound function.
Read the full section here.
Source code
"},{"location":"#not-exploring-all-the-go-testing-features-90","title":"Not exploring all the Go testing features (#90)","text":"
Code coverage
Use code coverage with the -coverprofile flag to quickly see which part of the code needs more attention.
Testing from a different package
Place unit tests in a different package to enforce writing tests that focus on an exposed behavior, not internals.
Source code
Utility functions
Handling errors using the *testing.T variable instead of the classic if err != nil makes code shorter and easier to read.
Source code
Setup and teardown
You can use setup and teardown functions to configure a complex environment, such as in the case of integration tests.
Source code
"},{"location":"#not-using-fuzzing-community-mistake","title":"Not using fuzzing (community mistake)","text":"TL;DR
Fuzzing is an efficient strategy to detect random, unexpected, or malformed inputs to complex functions and methods in order to discover vulnerabilities, bugs, or even potential crashes.
Credits: @jeromedoucet
"},{"location":"#optimizations","title":"Optimizations","text":""},{"location":"#not-understanding-cpu-caches-91","title":"Not understanding CPU caches (#91)","text":"
CPU architecture
Understanding how to use CPU caches is important for optimizing CPU-bound applications because the L1 cache is about 50 to 100 times faster than the main memory.
Cache line
Being conscious of the cache line concept is critical to understanding how to organize data in data-intensive applications. A CPU doesn\u2019t fetch memory word by word; instead, it usually copies a memory block to a 64-byte cache line. To get the most out of each individual cache line, enforce spatial locality.
Source code
Slice of structs vs. struct of slices
Source code
Predictability
Making code predictable for the CPU can also be an efficient way to optimize certain functions. For example, a unit or constant stride is predictable for the CPU, but a non-unit stride (for example, a linked list) isn\u2019t predictable.
Source code
Cache placement policy
To avoid a critical stride, hence utilizing only a tiny portion of the cache, be aware that caches are partitioned.
"},{"location":"#writing-concurrent-code-that-leads-to-false-sharing-92","title":"Writing concurrent code that leads to false sharing (#92)","text":"TL;DR
Knowing that lower levels of CPU caches aren\u2019t shared across all the cores helps avoid performance-degrading patterns such as false sharing while writing concurrency code. Sharing memory is an illusion.
Source code
"},{"location":"#not-taking-into-account-instruction-level-parallelism-93","title":"Not taking into account instruction-level parallelism (#93)","text":"TL;DR
Use instruction-level parallelism (ILP) to optimize specific parts of your code to allow a CPU to execute as many parallel instructions as possible. Identifying data hazards is one of the main steps.
Source code
"},{"location":"#not-being-aware-of-data-alignment-94","title":"Not being aware of data alignment (#94)","text":"TL;DR
You can avoid common mistakes by remembering that in Go, basic types are aligned with their own size. For example, keep in mind that reorganizing the fields of a struct by size in descending order can lead to more compact structs (less memory allocation and potentially a better spatial locality).
Source code
"},{"location":"#not-understanding-stack-vs-heap-95","title":"Not understanding stack vs. heap (#95)","text":"TL;DR
Understanding the fundamental differences between heap and stack should also be part of your core knowledge when optimizing a Go application. Stack allocations are almost free, whereas heap allocations are slower and rely on the GC to clean the memory.
Source code
"},{"location":"#not-knowing-how-to-reduce-allocations-api-change-compiler-optimizations-and-syncpool-96","title":"Not knowing how to reduce allocations (API change, compiler optimizations, and sync.Pool) (#96)","text":"TL;DR
Reducing allocations is also an essential aspect of optimizing a Go application. This can be done in different ways, such as designing the API carefully to prevent sharing up, understanding the common Go compiler optimizations, and using sync.Pool.
Source code
"},{"location":"#not-relying-on-inlining-97","title":"Not relying on inlining (#97)","text":"TL;DR
Use the fast-path inlining technique to efficiently reduce the amortized time to call a function.
"},{"location":"#not-using-go-diagnostics-tooling-98","title":"Not using Go diagnostics tooling (#98)","text":"TL;DR
Rely on profiling and the execution tracer to understand how an application performs and the parts to optimize.
Read the full section here.
"},{"location":"#not-understanding-how-the-gc-works-99","title":"Not understanding how the GC works (#99)","text":"TL;DR
Understanding how to tune the GC can lead to multiple benefits such as handling sudden load increases more efficiently.
"},{"location":"#not-understanding-the-impacts-of-running-go-in-docker-and-kubernetes-100","title":"Not understanding the impacts of running Go in Docker and Kubernetes (#100)","text":"TL;DR
To help avoid CPU throttling when deployed in Docker and Kubernetes, keep in mind that Go isn\u2019t CFS-aware.
"},{"location":"20-slice/","title":"Not understanding slice length and capacity","text":"
It\u2019s pretty common for Go developers to mix slice length and capacity or not understand them thoroughly. Assimilating these two concepts is essential for efficiently handling core operations such as slice initialization and adding elements with append, copying, or slicing. This misunderstanding can lead to using slices suboptimally or even to memory leaks.
In Go, a slice is backed by an array. That means the slice\u2019s data is stored contiguously in an array data structure. A slice also handles the logic of adding an element if the backing array is full or shrinking the backing array if it\u2019s almost empty.
Internally, a slice holds a pointer to the backing array plus a length and a capacity. The length is the number of elements the slice contains, whereas the capacity is the number of elements in the backing array, counting from the first element in the slice. Let\u2019s go through a few examples to make things clearer. First, let\u2019s initialize a slice with a given length and capacity:
s := make([]int, 3, 6) // Three-length, six-capacity slice\n
The first argument, representing the length, is mandatory. However, the second argument representing the capacity is optional. Figure 1 shows the result of this code in memory.
Figure 1: A three-length, six-capacity slice.
In this case, make creates an array of six elements (the capacity). But because the length was set to 3, Go initializes only the first three elements. Also, because the slice is an []int type, the first three elements are initialized to the zeroed value of an int: 0. The grayed elements are allocated but not yet used.
If we print this slice, we get the elements within the range of the length, [0 0 0]. If we set s[1] to 1, the second element of the slice updates without impacting its length or capacity. Figure 2 illustrates this.
Figure 2: Updating the slice\u2019s second element: s[1] = 1.
However, accessing an element outside the length range is forbidden, even though it\u2019s already allocated in memory. For example, s[4] = 0 would lead to the following panic:
panic: runtime error: index out of range [4] with length 3\n
How can we use the remaining space of the slice? By using the append built-in function:
s = append(s, 2)\n
This code appends to the existing s slice a new element. It uses the first grayed element (which was allocated but not yet used) to store element 2, as figure 3 shows.
Figure 3: Appending an element to s.
The length of the slice is updated from 3 to 4 because the slice now contains four elements. Now, what happens if we add three more elements so that the backing array isn\u2019t large enough?
s = append(s, 3, 4, 5)\nfmt.Println(s)\n
If we run this code, we see that the slice was able to cope with our request:
[0 1 0 2 3 4 5]\n
Because an array is a fixed-size structure, it can store the new elements until element 4. When we want to insert element 5, the array is already full: Go internally creates another array by doubling the capacity, copying all the elements, and then inserting element 5. Figure 4 shows this process.
Figure 4: Because the initial backing array is full, Go creates another array and copies all the elements.
The slice now references the new backing array. What will happen to the previous backing array? If it\u2019s no longer referenced, it\u2019s eventually freed by the garbage collector (GC) if allocated on the heap. (We discuss heap memory in mistake #95, \u201cNot understanding stack vs. heap,\u201d and we look at how the GC works in mistake #99, \u201cNot understanding how the GC works.\u201d)
What happens with slicing? Slicing is an operation done on an array or a slice, providing a half-open range; the first index is included, whereas the second is excluded. The following example shows the impact, and figure 5 displays the result in memory:
s1 := make([]int, 3, 6) // Three-length, six-capacity slice\ns2 := s1[1:3] // Slicing from indices 1 to 3\n
Figure 5: The slices s1 and s2 reference the same backing array with different lengths and capacities.
First, s1 is created as a three-length, six-capacity slice. When s2 is created by slicing s1, both slices reference the same backing array. However, s2 starts from a different index, 1. Therefore, its length and capacity (a two-length, five-capacity slice) differ from s1. If we update s1[1] or s2[0], the change is made to the same array, hence, visible in both slices, as figure 6 shows.
Figure 6: Because s1 and s2 are backed by the same array, updating a common element makes the change visible in both slices.
Now, what happens if we append an element to s2? Does the following code change s1 as well?
s2 = append(s2, 2)\n
The shared backing array is modified, but only the length of s2 changes. Figure 7 shows the result of appending an element to s2.
Figure 7: Appending an element to s2.
s1 remains a three-length, six-capacity slice. Therefore, if we print s1 and s2, the added element is only visible for s2:
s1=[0 1 0], s2=[1 0 2]\n
It\u2019s important to understand this behavior so that we don\u2019t make wrong assumptions while using append.
Note
In these examples, the backing array is internal and not available directly to the Go developer. The only exception is when a slice is created from slicing an existing array.
One last thing to note: what if we keep appending elements to s2 until the backing array is full? What will the state be, memory-wise? Let\u2019s add three more elements so that the backing array will not have enough capacity:
s2 = append(s2, 3)\ns2 = append(s2, 4) // At this stage, the backing is already full\ns2 = append(s2, 5)\n
This code leads to creating another backing array. Figure 8 displays the results in memory.
Figure 8: Appending elements to s2 until the backing array is full.
s1 and s2 now reference two different arrays. As s1 is still a three-length, six-capacity slice, it still has some available buffer, so it keeps referencing the initial array. Also, the new backing array was made by copying the initial one from the first index of s2. That\u2019s why the new array starts with element 1, not 0.
To summarize, the slice length is the number of available elements in the slice, whereas the slice capacity is the number of elements in the backing array. Adding an element to a full slice (length == capacity) leads to creating a new backing array with a new capacity, copying all the elements from the previous array, and updating the slice pointer to the new array.
"},{"location":"28-maps-memory-leaks/","title":"Maps and memory leaks","text":"
When working with maps in Go, we need to understand some important characteristics of how a map grows and shrinks. Let\u2019s delve into this to prevent issues that can cause memory leaks.
First, to view a concrete example of this problem, let\u2019s design a scenario where we will work with the following map:
m := make(map[int][128]byte)\n
Each value of m is an array of 128 bytes. We will do the following:
Allocate an empty map.
Add 1 million elements.
Remove all the elements, and run a Garbage Collection (GC).
After each step, we want to print the size of the heap (using a printAlloc utility function). This shows us how this example behaves memory-wise:
func main() {\nn := 1_000_000\nm := make(map[int][128]byte)\nprintAlloc()\nfor i := 0; i < n; i++ { // Adds 1 million elements\nm[i] = [128]byte{}\n}\nprintAlloc()\nfor i := 0; i < n; i++ { // Deletes 1 million elements\ndelete(m, i)\n}\nruntime.GC() // Triggers a manual GC\nprintAlloc()\nruntime.KeepAlive(m) // Keeps a reference to m so that the map isn\u2019t collected\n}\nfunc printAlloc() {\nvar m runtime.MemStats\nruntime.ReadMemStats(&m)\nfmt.Printf(\"%d KB\\n\", m.Alloc/1024)\n}\n
We allocate an empty map, add 1 million elements, remove 1 million elements, and then run a GC. We also make sure to keep a reference to the map using runtime.KeepAlive so that the map isn\u2019t collected as well. Let\u2019s run this example:
0 MB <-- After m is allocated\n461 MB <-- After we add 1 million elements\n293 MB <-- After we remove 1 million elements\n
What can we observe? At first, the heap size is minimal. Then it grows significantly after having added 1 million elements to the map. But if we expected the heap size to decrease after removing all the elements, this isn\u2019t how maps work in Go. In the end, even though the GC has collected all the elements, the heap size is still 293 MB. So the memory shrunk, but not as we might have expected. What\u2019s the rationale? We need to delve into how a map works in Go.
A map provides an unordered collection of key-value pairs in which all the keys are distinct. In Go, a map is based on the hash table data structure: an array where each element is a pointer to a bucket of key-value pairs, as shown in figure 1.
Figure 1: A hash table example with a focus on bucket 0.
Each bucket is a fixed-size array of eight elements. In the case of an insertion into a bucket that is already full (a bucket overflow), Go creates another bucket of eight elements and links the previous one to it. Figure 2 shows an example:
Figure 2: In case of a bucket overflow, Go allocates a new bucket and links the previous bucket to it.
Under the hood, a Go map is a pointer to a runtime.hmap struct. This struct contains multiple fields, including a B field, giving the number of buckets in the map:
type hmap struct {\nB uint8 // log_2 of # of buckets\n// (can hold up to loadFactor * 2^B items)\n// ...\n}\n
After adding 1 million elements, the value of B equals 18, which means 2\u00b9\u2078 = 262,144 buckets. When we remove 1 million elements, what\u2019s the value of B? Still 18. Hence, the map still contains the same number of buckets.
The reason is that the number of buckets in a map cannot shrink. Therefore, removing elements from a map doesn\u2019t impact the number of existing buckets; it just zeroes the slots in the buckets. A map can only grow and have more buckets; it never shrinks.
In the previous example, we went from 461 MB to 293 MB because the elements were collected, but running the GC didn\u2019t impact the map itself. Even the number of extra buckets (the buckets created because of overflows) remains the same.
Let\u2019s take a step back and discuss when the fact that a map cannot shrink can be a problem. Imagine building a cache using a map[int][128]byte. This map holds per customer ID (the int), a sequence of 128 bytes. Now, suppose we want to save the last 1,000 customers. The map size will remain constant, so we shouldn\u2019t worry about the fact that a map cannot shrink.
However, let\u2019s say we want to store one hour of data. Meanwhile, our company has decided to have a big promotion for Black Friday: in one hour, we may have millions of customers connected to our system. But a few days after Black Friday, our map will contain the same number of buckets as during the peak time. This explains why we can experience high memory consumption that doesn\u2019t significantly decrease in such a scenario.
What are the solutions if we don\u2019t want to manually restart our service to clean the amount of memory consumed by the map? One solution could be to re-create a copy of the current map at a regular pace. For example, every hour, we can build a new map, copy all the elements, and release the previous one. The main drawback of this option is that following the copy and until the next garbage collection, we may consume twice the current memory for a short period.
Another solution would be to change the map type to store an array pointer: map[int]*[128]byte. It doesn\u2019t solve the fact that we will have a significant number of buckets; however, each bucket entry will reserve the size of a pointer for the value instead of 128 bytes (8 bytes on 64-bit systems and 4 bytes on 32-bit systems).
Coming back to the original scenario, let\u2019s compare the memory consumption for each map type following each step. The following table shows the comparison.
Step map[int][128]bytemap[int]*[128]byte Allocate an empty map 0 MB 0 MB Add 1 million elements 461 MB 182 MB Remove all the elements and run a GC 293 MB 38 MB Note
If a key or a value is over 128 bytes, Go won\u2019t store it directly in the map bucket. Instead, Go stores a pointer to reference the key or the value.
As we have seen, adding n elements to a map and then deleting all the elements means keeping the same number of buckets in memory. So, we must remember that because a Go map can only grow in size, so does its memory consumption. There is no automated strategy to shrink it. If this leads to high memory consumption, we can try different options such as forcing Go to re-create the map or using pointers to check if it can be optimized.
"},{"location":"56-concurrency-faster/","title":"Thinking concurrency is always faster","text":"
A misconception among many developers is believing that a concurrent solution is always faster than a sequential one. This couldn\u2019t be more wrong. The overall performance of a solution depends on many factors, such as the efficiency of our code structure (concurrency), which parts can be tackled in parallel, and the level of contention among the computation units. This post reminds us about some fundamental knowledge of concurrency in Go; then we will see a concrete example where a concurrent solution isn\u2019t necessarily faster.
A thread is the smallest unit of processing that an OS can perform. If a process wants to execute multiple actions simultaneously, it spins up multiple threads. These threads can be:
Concurrent \u2014 Two or more threads can start, run, and complete in overlapping time periods.
Parallel \u2014 The same task can be executed multiple times at once.
The OS is responsible for scheduling the thread\u2019s processes optimally so that:
All the threads can consume CPU cycles without being starved for too much time.
The workload is distributed as evenly as possible among the different CPU cores.
Note
The word thread can also have a different meaning at a CPU level. Each physical core can be composed of multiple logical cores (the concept of hyper-threading), and a logical core is also called a thread. In this post, when we use the word thread, we mean the unit of processing, not a logical core.
A CPU core executes different threads. When it switches from one thread to another, it executes an operation called context switching. The active thread consuming CPU cycles was in an executing state and moves to a runnable state, meaning it\u2019s ready to be executed pending an available core. Context switching is considered an expensive operation because the OS needs to save the current execution state of a thread before the switch (such as the current register values).
As Go developers, we can\u2019t create threads directly, but we can create goroutines, which can be thought of as application-level threads. However, whereas an OS thread is context-switched on and off a CPU core by the OS, a goroutine is context-switched on and off an OS thread by the Go runtime. Also, compared to an OS thread, a goroutine has a smaller memory footprint: 2 KB for goroutines from Go 1.4. An OS thread depends on the OS, but, for example, on Linux/x86\u201332, the default size is 2 MB (see https://man7.org/linux/man-pages/man3/pthread_create.3.html). Having a smaller size makes context switching faster.
Note
Context switching a goroutine versus a thread is about 80% to 90% faster, depending on the architecture.
Let\u2019s now discuss how the Go scheduler works to overview how goroutines are handled. Internally, the Go scheduler uses the following terminology (see proc.go):
G \u2014 Goroutine
M \u2014 OS thread (stands for machine)
P \u2014 CPU core (stands for processor)
Each OS thread (M) is assigned to a CPU core (P) by the OS scheduler. Then, each goroutine (G) runs on an M. The GOMAXPROCS variable defines the limit of Ms in charge of executing user-level code simultaneously. But if a thread is blocked in a system call (for example, I/O), the scheduler can spin up more Ms. As of Go 1.5, GOMAXPROCS is by default equal to the number of available CPU cores.
A goroutine has a simpler lifecycle than an OS thread. It can be doing one of the following:
Executing \u2014 The goroutine is scheduled on an M and executing its instructions.
Runnable \u2014 The goroutine is waiting to be in an executing state.
Waiting \u2014 The goroutine is stopped and pending something completing, such as a system call or a synchronization operation (such as acquiring a mutex).
There\u2019s one last stage to understand about the implementation of Go scheduling: when a goroutine is created but cannot be executed yet; for example, all the other Ms are already executing a G. In this scenario, what will the Go runtime do about it? The answer is queuing. The Go runtime handles two kinds of queues: one local queue per P and a global queue shared among all the Ps.
Figure 1 shows a given scheduling situation on a four-core machine with GOMAXPROCS equal to 4. The parts are the logical cores (Ps), goroutines (Gs), OS threads (Ms), local queues, and global queue:
Figure 1: An example of the current state of a Go application executed on a four-core machine. Goroutines that aren\u2019t in an executing state are either runnable (pending being executed) or waiting (pending a blocking operation)
First, we can see five Ms, whereas GOMAXPROCS is set to 4. But as we mentioned, if needed, the Go runtime can create more OS threads than the GOMAXPROCS value.
P0, P1, and P3 are currently busy executing Go runtime threads. But P2 is presently idle as M3 is switched off P2, and there\u2019s no goroutine to be executed. This isn\u2019t a good situation because six runnable goroutines are pending being executed, some in the global queue and some in other local queues. How will the Go runtime handle this situation? Here\u2019s the scheduling implementation in pseudocode (see proc.go):
runtime.schedule() {\n // Only 1/61 of the time, check the global runnable queue for a G.\n // If not found, check the local queue.\n // If not found,\n // Try to steal from other Ps.\n // If not, check the global runnable queue.\n // If not found, poll network.\n}\n
Every sixty-first execution, the Go scheduler will check whether goroutines from the global queue are available. If not, it will check its local queue. Meanwhile, if both the global and local queues are empty, the Go scheduler can pick up goroutines from other local queues. This principle in scheduling is called work stealing, and it allows an underutilized processor to actively look for another processor\u2019s goroutines and steal some.
One last important thing to mention: prior to Go 1.14, the scheduler was cooperative, which meant a goroutine could be context-switched off a thread only in specific blocking cases (for example, channel send or receive, I/O, waiting to acquire a mutex). Since Go 1.14, the Go scheduler is now preemptive: when a goroutine is running for a specific amount of time (10 ms), it will be marked preemptible and can be context-switched off to be replaced by another goroutine. This allows a long-running job to be forced to share CPU time.
Now that we understand the fundamentals of scheduling in Go, let\u2019s look at a concrete example: implementing a merge sort in a parallel manner.
First, let\u2019s briefly review how the merge sort algorithm works. Then we will implement a parallel version. Note that the objective isn\u2019t to implement the most efficient version but to support a concrete example showing why concurrency isn\u2019t always faster.
The merge sort algorithm works by breaking a list repeatedly into two sublists until each sublist consists of a single element and then merging these sublists so that the result is a sorted list (see figure 2). Each split operation splits the list into two sublists, whereas the merge operation merges two sublists into a sorted list.
Figure 2: Applying the merge sort algorithm repeatedly breaks each list into two sublists. Then the algorithm uses a merge operation such that the resulting list is sorted
Here is the sequential implementation of this algorithm. We don\u2019t include all of the code as it\u2019s not the main point of this section:
func sequentialMergesort(s []int) {\nif len(s) <= 1 {\nreturn\n}\nmiddle := len(s) / 2\nsequentialMergesort(s[:middle]) // First half\nsequentialMergesort(s[middle:]) // Second half\nmerge(s, middle) // Merges the two halves\n}\nfunc merge(s []int, middle int) {\n// ...\n}\n
This algorithm has a structure that makes it open to concurrency. Indeed, as each sequentialMergesort operation works on an independent set of data that doesn\u2019t need to be fully copied (here, an independent view of the underlying array using slicing), we could distribute this workload among the CPU cores by spinning up each sequentialMergesort operation in a different goroutine. Let\u2019s write a first parallel implementation:
func parallelMergesortV1(s []int) {\nif len(s) <= 1 {\nreturn\n}\nmiddle := len(s) / 2\nvar wg sync.WaitGroup\nwg.Add(2)\ngo func() { // Spins up the first half of the work in a goroutine\ndefer wg.Done()\nparallelMergesortV1(s[:middle])\n}()\ngo func() { // Spins up the second half of the work in a goroutine\ndefer wg.Done()\nparallelMergesortV1(s[middle:])\n}()\nwg.Wait()\nmerge(s, middle) // Merges the halves\n}\n
In this version, each half of the workload is handled in a separate goroutine. The parent goroutine waits for both parts by using sync.WaitGroup. Hence, we call the Wait method before the merge operation.
We now have a parallel version of the merge sort algorithm. Therefore, if we run a benchmark to compare this version against the sequential one, the parallel version should be faster, correct? Let\u2019s run it on a four-core machine with 10,000 elements:
Surprisingly, the parallel version is almost an order of magnitude slower. How can we explain this result? How is it possible that a parallel version that distributes a workload across four cores is slower than a sequential version running on a single machine? Let\u2019s analyze the problem.
If we have a slice of, say, 1,024 elements, the parent goroutine will spin up two goroutines, each in charge of handling a half consisting of 512 elements. Each of these goroutines will spin up two new goroutines in charge of handling 256 elements, then 128, and so on, until we spin up a goroutine to compute a single element.
If the workload that we want to parallelize is too small, meaning we\u2019re going to compute it too fast, the benefit of distributing a job across cores is destroyed: the time it takes to create a goroutine and have the scheduler execute it is much too high compared to directly merging a tiny number of items in the current goroutine. Although goroutines are lightweight and faster to start than threads, we can still face cases where a workload is too small.
So what can we conclude from this result? Does it mean the merge sort algorithm cannot be parallelized? Wait, not so fast.
Let\u2019s try another approach. Because merging a tiny number of elements within a new goroutine isn\u2019t efficient, let\u2019s define a threshold. This threshold will represent how many elements a half should contain in order to be handled in a parallel manner. If the number of elements in the half is fewer than this value, we will handle it sequentially. Here\u2019s a new version:
const max = 2048 // Defines the threshold\nfunc parallelMergesortV2(s []int) {\nif len(s) <= 1 {\nreturn\n}\nif len(s) <= max {\nsequentialMergesort(s) // Calls our initial sequential version\n} else { // If bigger than the threshold, keeps the parallel version\nmiddle := len(s) / 2\nvar wg sync.WaitGroup\nwg.Add(2)\ngo func() {\ndefer wg.Done()\nparallelMergesortV2(s[:middle])\n}()\ngo func() {\ndefer wg.Done()\nparallelMergesortV2(s[middle:])\n}()\nwg.Wait()\nmerge(s, middle)\n}\n}\n
If the number of elements in the s slice is smaller than max, we call the sequential version. Otherwise, we keep calling our parallel implementation. Does this approach impact the result? Yes, it does:
Our v2 parallel implementation is more than 40% faster than the sequential one, thanks to this idea of defining a threshold to indicate when parallel should be more efficient than sequential.
Note
Why did I set the threshold to 2,048? Because it was the optimal value for this specific workload on my machine. In general, such magic values should be defined carefully with benchmarks (running on an execution environment similar to production). It\u2019s also pretty interesting to note that running the same algorithm in a programming language that doesn\u2019t implement the concept of goroutines has an impact on the value. For example, running the same example in Java using threads means an optimal value closer to 8,192. This tends to illustrate how goroutines are more efficient than threads.
We have seen throughout this post the fundamental concepts of scheduling in Go: the differences between a thread and a goroutine and how the Go runtime schedules goroutines. Meanwhile, using the parallel merge sort example, we illustrated that concurrency isn\u2019t always necessarily faster. As we have seen, spinning up goroutines to handle minimal workloads (merging only a small set of elements) demolishes the benefit we could get from parallelism.
So, where should we go from here? We must keep in mind that concurrency isn\u2019t always faster and shouldn\u2019t be considered the default way to go for all problems. First, it makes things more complex. Also, modern CPUs have become incredibly efficient at executing sequential code and predictable code. For example, a superscalar processor can parallelize instruction execution over a single core with high efficiency.
Does this mean we shouldn\u2019t use concurrency? Of course not. However, it\u2019s essential to keep these conclusions in mind. If we\u2019re not sure that a parallel version will be faster, the right approach may be to start with a simple sequential version and build from there using profiling (mistake #98, \u201cNot using Go diagnostics tooling\u201d) and benchmarks (mistake #89, \u201cWriting inaccurate benchmarks\u201d), for example. It can be the only way to ensure that a concurrent implementation is worth it.
In general, we should never guess about performance. When writing optimizations, so many factors may come into play that even if we have a strong opinion about the results, it\u2019s rarely a bad idea to test them. However, writing benchmarks isn\u2019t straightforward. It can be pretty simple to write inaccurate benchmarks and make wrong assumptions based on them. The goal of this post is to examine four common and concrete traps leading to inaccuracy:
Before discussing these traps, let\u2019s briefly review how benchmarks work in Go. The skeleton of a benchmark is as follows:
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nfoo()\n}\n}\n
The function name starts with the Benchmark prefix. The function under test (foo) is called within the for loop. b.N represents a variable number of iterations. When running a benchmark, Go tries to make it match the requested benchmark time. The benchmark time is set by default to 1 second and can be changed with the -benchtime flag. b.N starts at 1; if the benchmark completes in under 1 second, b.N is increased, and the benchmark runs again until b.N roughly matches benchtime:
$ go test -bench=.\ncpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkFoo-4 73 16511228 ns/op\n
Here, the benchmark took about 1 second, and foo was executed 73 times, for an average execution time of 16,511,228 nanoseconds. We can change the benchmark time using -benchtime:
$ go test -bench=. -benchtime=2s\nBenchmarkFoo-4 150 15832169 ns/op\n
foo was executed roughly twice more than during the previous benchmark.
Next, let\u2019s look at some common traps.
"},{"location":"89-benchmarks/#not-resetting-or-pausing-the-timer","title":"Not resetting or pausing the timer","text":"
In some cases, we need to perform operations before the benchmark loop. These operations may take quite a while (for example, generating a large slice of data) and may significantly impact the benchmark results:
func BenchmarkFoo(b *testing.B) {\nexpensiveSetup()\nfor i := 0; i < b.N; i++ {\nfunctionUnderTest()\n}\n}\n
In this case, we can use the ResetTimer method before entering the loop:
func BenchmarkFoo(b *testing.B) {\nexpensiveSetup()\nb.ResetTimer() // Reset the benchmark timer\nfor i := 0; i < b.N; i++ {\nfunctionUnderTest()\n}\n}\n
Calling ResetTimer zeroes the elapsed benchmark time and memory allocation counters since the beginning of the test. This way, an expensive setup can be discarded from the test results.
What if we have to perform an expensive setup not just once but within each loop iteration?
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nexpensiveSetup()\nfunctionUnderTest()\n}\n}\n
We can\u2019t reset the timer, because that would be executed during each loop iteration. But we can stop and resume the benchmark timer, surrounding the call to expensiveSetup:
func BenchmarkFoo(b *testing.B) {\nfor i := 0; i < b.N; i++ {\nb.StopTimer() // Pause the benchmark timer\nexpensiveSetup()\nb.StartTimer() // Resume the benchmark timer\nfunctionUnderTest()\n}\n}\n
Here, we pause the benchmark timer to perform the expensive setup and then resume the timer.
Note
There\u2019s one catch to remember about this approach: if the function under test is too fast to execute compared to the setup function, the benchmark may take too long to complete. The reason is that it would take much longer than 1 second to reach benchtime. Calculating the benchmark time is based solely on the execution time of functionUnderTest. So, if we wait a significant time in each loop iteration, the benchmark will be much slower than 1 second. If we want to keep the benchmark, one possible mitigation is to decrease benchtime.
We must be sure to use the timer methods to preserve the accuracy of a benchmark.
"},{"location":"89-benchmarks/#making-wrong-assumptions-about-micro-benchmarks","title":"Making wrong assumptions about micro-benchmarks","text":"
A micro-benchmark measures a tiny computation unit, and it can be extremely easy to make wrong assumptions about it. Let\u2019s say, for example, that we aren\u2019t sure whether to use atomic.StoreInt32 or atomic.StoreInt64 (assuming that the values we handle will always fit in 32 bits). We want to write a benchmark to compare both functions:
func BenchmarkAtomicStoreInt32(b *testing.B) {\nvar v int32\nfor i := 0; i < b.N; i++ {\natomic.StoreInt32(&v, 1)\n}\n}\nfunc BenchmarkAtomicStoreInt64(b *testing.B) {\nvar v int64\nfor i := 0; i < b.N; i++ {\natomic.StoreInt64(&v, 1)\n}\n}\n
If we run this benchmark, here\u2019s some example output:
We could easily take this benchmark for granted and decide to use atomic.StoreInt64 because it appears to be faster. Now, for the sake of doing a fair benchmark, we reverse the order and test atomic.StoreInt64 first, followed by atomic.StoreInt32. Here is some example output:
This time, atomic.StoreInt32 has better results. What happened?
In the case of micro-benchmarks, many factors can impact the results, such as machine activity while running the benchmarks, power management, thermal scaling, and better cache alignment of a sequence of instructions. We must remember that many factors, even outside the scope of our Go project, can impact the results.
Note
We should make sure the machine executing the benchmark is idle. However, external processes may run in the background, which may affect benchmark results. For that reason, tools such as perflock can limit how much CPU a benchmark can consume. For example, we can run a benchmark with 70% of the total available CPU, giving 30% to the OS and other processes and reducing the impact of the machine activity factor on the results.
One option is to increase the benchmark time using the -benchtime option. Similar to the law of large numbers in probability theory, if we run a benchmark a large number of times, it should tend to approach its expected value (assuming we omit the benefits of instructions caching and similar mechanics).
Another option is to use external tools on top of the classic benchmark tooling. For instance, the benchstat tool, which is part of the golang.org/x repository, allows us to compute and compare statistics about benchmark executions.
Let\u2019s run the benchmark 10 times using the -count option and pipe the output to a specific file:
$ go test -bench=. -count=10 | tee stats.txt\ncpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkAtomicStoreInt32-4 234935682 5.124 ns/op\nBenchmarkAtomicStoreInt32-4 235307204 5.112 ns/op\n// ...\nBenchmarkAtomicStoreInt64-4 235548591 5.107 ns/op\nBenchmarkAtomicStoreInt64-4 235210292 5.090 ns/op\n// ...\n
The results are the same: both functions take on average 5.10 nanoseconds to complete. We also see the percent variation between the executions of a given benchmark: \u00b1 1%. This metric tells us that both benchmarks are stable, giving us more confidence in the computed average results. Therefore, instead of concluding that atomic.StoreInt32 is faster or slower, we can conclude that its execution time is similar to that of atomic.StoreInt64 for the usage we tested (in a specific Go version on a particular machine).
In general, we should be cautious about micro-benchmarks. Many factors can significantly impact the results and potentially lead to wrong assumptions. Increasing the benchmark time or repeating the benchmark executions and computing stats with tools such as benchstat can be an efficient way to limit external factors and get more accurate results, leading to better conclusions.
Let\u2019s also highlight that we should be careful about using the results of a micro-benchmark executed on a given machine if another system ends up running the application. The production system may act quite differently from the one on which we ran the micro-benchmark.
"},{"location":"89-benchmarks/#not-being-careful-about-compiler-optimizations","title":"Not being careful about compiler optimizations","text":"
Another common mistake related to writing benchmarks is being fooled by compiler optimizations, which can also lead to wrong benchmark assumptions. In this section, we look at Go issue 14813 (https://github.com/golang/go/issues/14813, also discussed by Go project member Dave Cheney) with a population count function (a function that counts the number of bits set to 1):
This function takes and returns a uint64. To benchmark this function, we can write the following:
func BenchmarkPopcnt1(b *testing.B) {\nfor i := 0; i < b.N; i++ {\npopcnt(uint64(i))\n}\n}\n
However, if we execute this benchmark, we get a surprisingly low result:
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz\nBenchmarkPopcnt1-4 1000000000 0.2858 ns/op\n
A duration of 0.28 nanoseconds is roughly one clock cycle, so this number is unreasonably low. The problem is that the developer wasn\u2019t careful enough about compiler optimizations. In this case, the function under test is simple enough to be a candidate for inlining: an optimization that replaces a function call with the body of the called function and lets us prevent a function call, which has a small footprint. Once the function is inlined, the compiler notices that the call has no side effects and replaces it with the following benchmark:
func BenchmarkPopcnt1(b *testing.B) {\nfor i := 0; i < b.N; i++ {\n// Empty\n}\n}\n
The benchmark is now empty \u2014 which is why we got a result close to one clock cycle. To prevent this from happening, a best practice is to follow this pattern:
During each loop iteration, assign the result to a local variable (local in the context of the benchmark function).
Assign the latest result to a global variable.
In our case, we write the following benchmark:
var global uint64 // Define a global variable\nfunc BenchmarkPopcnt2(b *testing.B) {\nvar v uint64 // Define a local variable\nfor i := 0; i < b.N; i++ {\nv = popcnt(uint64(i)) // Assign the result to the local variable\n}\nglobal = v // Assign the result to the global variable\n}\n
global is a global variable, whereas v is a local variable whose scope is the benchmark function. During each loop iteration, we assign the result of popcnt to the local variable. Then we assign the latest result to the global variable.
Note
Why not assign the result of the popcnt call directly to global to simplify the test? Writing to a global variable is slower than writing to a local variable (these concepts are discussed in 100 Go Mistakes, mistake #95: \u201cNot understanding stack vs. heap\u201d). Therefore, we should write each result to a local variable to limit the footprint during each loop iteration.
If we run these two benchmarks, we now get a significant difference in the results:
BenchmarkPopcnt2 is the accurate version of the benchmark. It guarantees that we avoid the inlining optimizations, which can artificially lower the execution time or even remove the call to the function under test. Relying on the results of BenchmarkPopcnt1 could have led to wrong assumptions.
Let\u2019s remember the pattern to avoid compiler optimizations fooling benchmark results: assign the result of the function under test to a local variable, and then assign the latest result to a global variable. This best practice also prevents us from making incorrect assumptions.
"},{"location":"89-benchmarks/#being-fooled-by-the-observer-effect","title":"Being fooled by the observer effect","text":"
In physics, the observer effect is the disturbance of an observed system by the act of observation. This effect can also be seen in benchmarks and can lead to wrong assumptions about results. Let\u2019s look at a concrete example and then try to mitigate it.
We want to implement a function receiving a matrix of int64 elements. This matrix has a fixed number of 512 columns, and we want to compute the total sum of the first eight columns, as shown in figure 1.
Figure 1: Computing the sum of the first eight columns.
For the sake of optimizations, we also want to determine whether varying the number of columns has an impact, so we also implement a second function with 513 columns. The implementation is the following:
func calculateSum512(s [][512]int64) int64 {\nvar sum int64\nfor i := 0; i < len(s); i++ { // Iterate over each row\nfor j := 0; j < 8; j++ { // Iterate over the first eight columns\nsum += s[i][j] // Increment sum\n}\n}\nreturn sum\n}\nfunc calculateSum513(s [][513]int64) int64 {\n// Same implementation as calculateSum512\n}\n
We iterate over each row and then over the first eight columns, and we increment a sum variable that we return. The implementation in calculateSum513 remains the same.
We want to benchmark these functions to decide which one is the most performant given a fixed number of rows:
const rows = 1000\nvar res int64\nfunc BenchmarkCalculateSum512(b *testing.B) {\nvar sum int64\ns := createMatrix512(rows) // Create a matrix of 512 columns\nb.ResetTimer()\nfor i := 0; i < b.N; i++ {\nsum = calculateSum512(s) // Create a matrix of 512 columns\n}\nres = sum\n}\nfunc BenchmarkCalculateSum513(b *testing.B) {\nvar sum int64\ns := createMatrix513(rows) // Create a matrix of 513 columns\nb.ResetTimer()\nfor i := 0; i < b.N; i++ {\nsum = calculateSum513(s) // Calculate the sum\n}\nres = sum\n}\n
We want to create the matrix only once, to limit the footprint on the results. Therefore, we call createMatrix512 and createMatrix513 outside of the loop. We may expect the results to be similar as again we only want to iterate on the first eight columns, but this isn\u2019t the case (on my machine):
The second benchmark with 513 columns is about 50% faster. Again, because we iterate only over the first eight columns, this result is quite surprising.
To understand this difference, we need to understand the basics of CPU caches. In a nutshell, a CPU is composed of different caches (usually L1, L2, and L3). These caches reduce the average cost of accessing data from the main memory. In some conditions, the CPU can fetch data from the main memory and copy it to L1. In this case, the CPU tries to fetch into L1 the matrix\u2019s subset that calculateSum is interested in (the first eight columns of each row). However, the matrix fits in memory in one case (513 columns) but not in the other case (512 columns).
Note
This isn\u2019t in the scope of this post to explain why, but we look at this problem in 100 Go Mistakes, mistake #91: \u201cNot understanding CPU caches.\u201d
Coming back to the benchmark, the main issue is that we keep reusing the same matrix in both cases. Because the function is repeated thousands of times, we don\u2019t measure the function\u2019s execution when it receives a plain new matrix. Instead, we measure a function that gets a matrix that already has a subset of the cells present in the cache. Therefore, because calculateSum513 leads to fewer cache misses, it has a better execution time.
This is an example of the observer effect. Because we keep observing a repeatedly called CPU-bound function, CPU caching may come into play and significantly affect the results. In this example, to prevent this effect, we should create a matrix during each test instead of reusing one:
func BenchmarkCalculateSum512(b *testing.B) {\nvar sum int64\nfor i := 0; i < b.N; i++ {\nb.StopTimer()\ns := createMatrix512(rows) // Create a new matrix during each loop iteration\nb.StartTimer()\nsum = calculateSum512(s)\n}\nres = sum\n}\n
A new matrix is now created during each loop iteration. If we run the benchmark again (and adjust benchtime \u2014 otherwise, it takes too long to execute), the results are closer to each other:
Instead of making the incorrect assumption that calculateSum513 is faster, we see that both benchmarks lead to similar results when receiving a new matrix.
As we have seen in this post, because we were reusing the same matrix, CPU caches significantly impacted the results. To prevent this, we had to create a new matrix during each loop iteration. In general, we should remember that observing a function under test may lead to significant differences in results, especially in the context of micro-benchmarks of CPU-bound functions where low-level optimizations matter. Forcing a benchmark to re-create data during each iteration can be a good way to prevent this effect.
"},{"location":"9-generics/","title":"Being confused about when to use generics","text":"
Generics is a fresh addition to the language. In a nutshell, it allows writing code with types that can be specified later and instantiated when needed. However, it can be pretty easy to be confused about when to use generics and when not to. Throughout this post, we will describe the concept of generics in Go and then delve into common use and misuses.
Consider the following function that extracts all the keys from a map[string]int type:
func getKeys(m map[string]int) []string {\nvar keys []string\nfor k := range m {\nkeys = append(keys, k)\n}\nreturn keys\n}\n
What if we would like to use a similar feature for another map type such as a map[int]string? Before generics, Go developers had a couple of options: using code generation, reflection, or duplicating code.
For example, we could write two functions, one for each map type, or even try to extend getKeys to accept different map types:
func getKeys(m any) ([]any, error) {\nswitch t := m.(type) {\ndefault:\nreturn nil, fmt.Errorf(\"unknown type: %T\", t)\ncase map[string]int:\nvar keys []any\nfor k := range t {\nkeys = append(keys, k)\n}\nreturn keys, nil\ncase map[int]string:\n// Copy the extraction logic\n}\n}\n
We can start noticing a couple of issues:
First, it increases boilerplate code. Indeed, whenever we want to add a case, it will require duplicating the range loop.
Meanwhile, the function now accepts an empty interface, which means we are losing some of the benefits of Go being a typed language. Indeed, checking whether a type is supported is done at runtime instead of compile-time. Hence, we also need to return an error if the provided type is unknown.
Last but not least, as the key type can be either int or string, we are obliged to return a slice of empty interfaces to factor out key types. This approach increases the effort on the caller-side as the client may also have to perform a type check of the keys or extra conversion.
Thanks to generics, we can now refactor this code using type parameters.
Type parameters are generic types we can use with functions and types. For example, the following function accepts a type parameter:
func foo[T any](t T) {\n// ...\n}\n
When calling foo, we will pass a type argument of any type. Passing a type argument is called instantiation because the work is done at compile time which keeps type safety as part of the core language features and avoids runtime overheads.
Let\u2019s get back to the getKeys function and use type parameters to write a generic version that would accept any kind of map:
func getKeys[K comparable, V any](m map[K]V) []K { <1>\nvar keys []K <2>\nfor k := range m {\nkeys = append(keys, k)\n}\nreturn keys\n}\n
To handle the map, we defined two kinds of type parameters. First, the values can be of any type: V any. However, in Go, the map keys can\u2019t be of any type. For example, we cannot use slices:
var m map[[]byte]int\n
This code leads to a compilation error: invalid map key type []byte. Therefore, instead of accepting any key type, we are obliged to restrict type arguments so that the key type meets specific requirements. Here, being comparable (we can use == or !=). Hence, we defined K as comparable instead of any.
Restricting type arguments to match specific requirements is called a constraint. A constraint is an interface type that can contain:
A set of behaviors (methods)
But also arbitrary type
Let\u2019s see a concrete example for the latter. Imagine we don\u2019t want to accept any comparable type for map key type. For instance, we would like to restrict it to either int or string types. We can define a custom constraint this way:
type customConstraint interface {\n~int | ~string // Define a custom type that will restrict types to int and string\n}\n// Change the type parameter K to be custom\nfunc getKeys[K customConstraint, V any](m map[K]V) []K {\n// Same implementation\n}\n
First, we define a customConstraint interface to restrict the types to be either int or string using the union operator | (we will discuss the use of ~ a bit later). Then, K is now a customConstraint instead of a comparable as before.
Now, the signature of getKeys enforces that we can call it with a map of any value type, but the key type has to be an int or a string. For example, on the caller-side:
m = map[string]int{\n\"one\": 1,\n\"two\": 2,\n\"three\": 3,\n}\nkeys := getKeys(m)\n
Note that Go can infer that getKeys is called with a string type argument. The previous call was similar to this:
keys := getKeys[string](m)\n
Note
What\u2019s the difference between a constraint using ~int or int? Using int restricts it to that type, whereas ~int restricts all the types whose underlying type is an int.
To illustrate it, let\u2019s imagine a constraint where we would like to restrict a type to any int type implementing the String() string method:
type customConstraint interface {\n~int\nString() string\n}\n
Using this constraint will restrict type arguments to custom types like this one:
type customInt int\nfunc (i customInt) String() string {\nreturn strconv.Itoa(int(i))\n}\n
As customInt is an int and implements the String() string method, the customInt type satisfies the constraint defined.
However, if we change the constraint to contain an int instead of an ~int, using customInt would lead to a compilation error because the int type doesn\u2019t implement String() string.
Let\u2019s also note the constraints package contains a set of common constraints such as Signed that includes all the signed integer types. Let\u2019s ensure that a constraint doesn\u2019t already exist in this package before creating a new one.
So far, we have discussed examples using generics for functions. However, we can also use generics with data structures.
For example, we will create a linked list containing values of any type. Meanwhile, we will write an Add method to append a node:
type Node[T any] struct { // Use type parameter\nVal T\nnext *Node[T]\n}\nfunc (n *Node[T]) Add(next *Node[T]) { // Instantiate type receiver\nn.next = next\n}\n
We use type parameters to define T and use both fields in Node. Regarding the method, the receiver is instantiated. Indeed, because Node is generic, it has to follow also the type parameter defined.
One last thing to note about type parameters: they can\u2019t be used on methods, only on functions. For example, the following method wouldn\u2019t compile:
type Foo struct {}\nfunc (Foo) bar[T any](t T) {}\n
./main.go:29:15: methods cannot have type parameters\n
Now, let\u2019s delve into concrete cases where we should and shouldn\u2019t use generics.
"},{"location":"9-generics/#common-uses-and-misuses","title":"Common uses and misuses","text":"
So when are generics useful? Let\u2019s discuss a couple of common uses where generics are recommended:
Data structures. For example, we can use generics to factor out the element type if we implement a binary tree, a linked list, or a heap.
Functions working with slices, maps, and channels of any type. For example, a function to merge two channels would work with any channel type. Hence, we could use type parameters to factor out the channel type:
func merge[T any](ch1, ch2 <-chan T) <-chan T {\n// ...\n}\n
Meanwhile, instead of factoring out a type, we can factor out behaviors. For example, the sort package contains functions to sort different slice types such as sort.Ints or sort.Float64s. Using type parameters, we can factor out the sorting behaviors that rely on three methods, Len, Less, and Swap:
type sliceFn[T any] struct { // Use type parameter\ns []T\ncompare func(T, T) bool // Compare two T elements\n}\nfunc (s sliceFn[T]) Len() int { return len(s.s) }\nfunc (s sliceFn[T]) Less(i, j int) bool { return s.compare(s.s[i], s.s[j]) }\nfunc (s sliceFn[T]) Swap(i, j int) { s.s[i], s.s[j] = s.s[j], s.s[i] }\n
Conversely, when is it recommended not to use generics?
When just calling a method of the type argument. For example, consider a function that receives an io.Writer and call the Write method:
When it makes our code more complex. Generics are never mandatory, and as Go developers, we have been able to live without them for more than a decade. If writing generic functions or structures we figure out that it doesn\u2019t make our code clearer, we should probably reconsider our decision for this particular use case.
Though generics can be very helpful in particular conditions, we should be cautious about when to use them and not use them.
In general, when we want to answer when not to use generics, we can find similarities with when not to use interfaces. Indeed, generics introduce a form of abstraction, and we have to remember that unnecessary abstractions introduce complexity.
Let\u2019s not pollute our code with needless abstractions, and let\u2019s focus on solving concrete problems for now. It means that we shouldn\u2019t use type parameters prematurely. Let\u2019s wait until we are about to write boilerplate code to consider using generics.
"},{"location":"98-profiling-execution-tracing/","title":"Not using Go diagnostics tooling","text":"
Go offers a few excellent diagnostics tools to help us get insights into how an application performs. This post focuses on the most important ones: profiling and the execution tracer. Both tools are so important that they should be part of the core toolset of any Go developer who is interested in optimization. First, let\u2019s discuss profiling.
Profiling provides insights into the execution of an application. It allows us to resolve performance issues, detect contention, locate memory leaks, and more. These insights can be collected via several profiles:
CPU\u2014 Determines where an application spends its time
Goroutine\u2014 Reports the stack traces of the ongoing goroutines
Heap\u2014 Reports heap memory allocation to monitor current memory usage and check for possible memory leaks
Mutex\u2014 Reports lock contentions to see the behaviors of the mutexes used in our code and whether an application spends too much time in locking calls
Block\u2014 Shows where goroutines block waiting on synchronization primitives
Profiling is achieved via instrumentation using a tool called a profiler, in Go: pprof. First, let\u2019s understand how and when to enable pprof; then, we discuss the most critical profile types.
There are several ways to enable pprof. For example, we can use the net/http/pprof package to serve the profiling data via HTTP:
package main\nimport (\n\"fmt\"\n\"log\"\n\"net/http\"\n_ \"net/http/pprof\" // Blank import to pprof\n)\nfunc main() {\n// Exposes an HTTP endpoint\nhttp.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\nfmt.Fprintf(w, \"\")\n})\nlog.Fatal(http.ListenAndServe(\":80\", nil))\n}\n
Importing net/http/pprof leads to a side effect that allows us to reach the pprof URL: http://host/debug/pprof. Note that enabling pprof is safe even in production (https://go.dev/doc/diagnostics#profiling). The profiles that impact performance, such as CPU profiling, aren\u2019t enabled by default, nor do they run continuously: they are activated only for a specific period.
Now that we have seen how to expose a pprof endpoint, let\u2019s discuss the most common profiles.
The CPU profiler relies on the OS and signaling. When it is activated, the application asks the OS to interrupt it every 10 ms by default via a SIGPROF signal. When the application receives a SIGPROF, it suspends the current activity and transfers the execution to the profiler. The profiler collects data such as the current goroutine activity and aggregates execution statistics that we can retrieve. Then it stops, and the execution resumes until the next SIGPROF.
We can access the /debug/pprof/profile endpoint to activate CPU profiling. Accessing this endpoint executes CPU profiling for 30 seconds by default. For 30 seconds, our application is interrupted every 10 ms. Note that we can change these two default values: we can use the seconds parameter to pass to the endpoint how long the profiling should last (for example, /debug/pprof/profile?seconds=15), and we can change the interruption rate (even to less than 10 ms). But in most cases, 10 ms should be enough, and in decreasing this value (meaning increasing the rate), we should be careful not to harm performance. After 30 seconds, we download the results of the CPU profiler.
Note
We can also enable the CPU profiler using the -cpuprofile flag, such as when running a benchmark. For example, the following command produces the same type of file that can be downloaded via /debug/ pprof/profile.
$ go test -bench=. -cpuprofile profile.out\n
From this file, we can navigate to the results using go tool:
$ go tool pprof -http=:8080 <file>\n
This command opens a web UI showing the call graph. The next figure shows an example taken from an application. The larger the arrow, the more it was a hot path. We can then navigate into this graph and get execution insights.
Figure 1: The call graph of an application during 30 seconds.
For example, the graph in the next figure tells us that during 30 seconds, 0.06 seconds were spent in the decode method (*FetchResponse receiver). Of these 0.06 seconds, 0.02 were spent in RecordBatch.decode and 0.01 in makemap (creating a map).
Figure 2: Example call graph.
We can also access this kind of information from the web UI with different representations. For example, the Top view sorts the functions per execution time, and Flame Graph visualizes the execution time hierarchy. The UI can even display the expensive parts of the source code line by line.
Note
We can also delve into profiling data via a command line. However, we focus on the web UI in this post.
Thanks to this data, we can get a general idea of how an application behaves:
Too many calls to runtime.mallogc can mean an excessive number of small heap allocations that we can try to minimize.
Too much time spent in channel operations or mutex locks can indicate excessive contention that is harming the application\u2019s performance.
Too much time spent on syscall.Read or syscall.Write means the application spends a significant amount of time in Kernel mode. Working on I/O buffering may be an avenue for improvement.
These are the kinds of insights we can get from the CPU profiler. It\u2019s valuable to understand the hottest code path and identify bottlenecks. But it won\u2019t determine more than the configured rate because the CPU profiler is executed at a fixed pace (by default, 10 ms). To get finer-grained insights, we should use tracing, which we discuss later in this post.
Note
We can also attach labels to the different functions. For example, imagine a common function called from different clients. To track the time spent for both clients, we can use pprof.Labels.
Heap profiling allows us to get statistics about the current heap usage. Like CPU profiling, heap profiling is sample-based. We can change this rate, but we shouldn\u2019t be too granular because the more we decrease the rate, the more effort heap profiling will require to collect data. By default, samples are profiled at one allocation for every 512 KB of heap allocation.
If we reach /debug/pprof/heap/, we get raw data that can be hard to read. However, we can download a heap profile using /debug/pprof/heap/?debug=0 and then open it with go tool (the same command as in the previous section) to navigate into the data using the web UI.
The next figure shows an example of a heap graph. Calling the MetadataResponse.decode method leads to allocating 1536 KB of heap data (which represents 6.32% of the total heap). However, 0 out of these 1536 KB were allocated by this function directly, so we need to inspect the second call. The TopicMetadata.decode method allocated 512 KB out of the 1536 KB; the rest \u2014 1024 KB \u2014 were allocated in another method.
Figure 3: A heap graph.
This is how we can navigate the call chain to understand what part of an application is responsible for most of the heap allocations. We can also look at different sample types:
alloc_objects\u2014 Total number of objects allocated
alloc_space\u2014 Total amount of memory allocated
inuse_objects \u2014 Number of objects allocated and not yet released
inuse_space\u2014 Amount of memory allocated and not yet released
Another very helpful capability with heap profiling is tracking memory leaks. With a GC-based language, the usual procedure is the following:
Trigger a GC.
Download heap data.
Wait for a few seconds/minutes.
Trigger another GC.
Download another heap data.
Compare.
Forcing a GC before downloading data is a way to prevent false assumptions. For example, if we see a peak of retained objects without running a GC first, we cannot be sure whether it\u2019s a leak or objects that the next GC will collect.
Using pprof, we can download a heap profile and force a GC in the meantime. The procedure in Go is the following:
Go to /debug/pprof/heap?gc=1 (trigger the GC and download the heap profile).
Wait for a few seconds/minutes.
Go to /debug/pprof/heap?gc=1 again.
Use go tool to compare both heap profiles:
$ go tool pprof -http=:8080 -diff_base <file2> <file1>\n
The next figure shows the kind of data we can access. For example, the amount of heap memory held by the newTopicProducer method (top left) has decreased (\u2013513 KB). In contrast, the amount held by updateMetadata (bottom right) has increased (+512 KB). Slow increases are normal. The second heap profile may have been calculated in the middle of a service call, for example. We can repeat this process or wait longer; the important part is to track steady increases in allocations of a specific object.
Figure 4: The differences between the two heap profiles. Note
Another type of profiling related to the heap is allocs, which reports allocations. Heap profiling shows the current state of the heap memory. To get insights about past memory allocations since the application started, we can use allocations profiling. As discussed, because stack allocations are cheap, they aren\u2019t part of this profiling, which only focuses on the heap.
The goroutine profile reports the stack trace of all the current goroutines in an application. We can download a file using /debug/pprof/goroutine/?debug=0 and use go tool again. The next figure shows the kind of information we can get.
Figure 5: Goroutine graph.
We can see the current state of the application and how many goroutines were created per function. In this case, withRecover has created 296 ongoing goroutines (63%), and 29 were related to a call to responseFeeder.
This kind of information is also beneficial if we suspect goroutine leaks. We can look at goroutine profiler data to know which part of a system is the suspect.
The block profile reports where ongoing goroutines block waiting on synchronization primitives. Possibilities include
Sending or receiving on an unbuffered channel
Sending to a full channel
Receiving from an empty channel
Mutex contention
Network or filesystem waits
Block profiling also records the amount of time a goroutine has been waiting and is accessible via /debug/pprof/block. This profile can be extremely helpful if we suspect that performance is being harmed by blocking calls.
The block profile isn\u2019t enabled by default: we have to call runtime.SetBlockProfileRate to enable it. This function controls the fraction of goroutine blocking events that are reported. Once enabled, the profiler will keep collecting data in the background even if we don\u2019t call the /debug/pprof/block endpoint. Let\u2019s be cautious if we want to set a high rate so we don\u2019t harm performance.
Note
If we face a deadlock or suspect that goroutines are in a blocked state, the full goroutine stack dump (/debug/pprof/goroutine/?debug=2) creates a dump of all the current goroutine stack traces. This can be helpful as a first analysis step. For example, the following dump shows a Sarama goroutine blocked for 1,420 minutes on a channel-receive operation:
The last profile type is related to blocking but only regarding mutexes. If we suspect that our application spends significant time waiting for locking mutexes, thus harming execution, we can use mutex profiling. It\u2019s accessible via /debug/pprof/mutex.
This profile works in a manner similar to that for blocking. It\u2019s disabled by default: we have to enable it using runtime.SetMutexProfileFraction, which controls the fraction of mutex contention events reported.
Following are a few additional notes about profiling:
We haven\u2019t mentioned the threadcreate profile because it\u2019s been broken since 2013 (https://github.com/golang/go/issues/6104).
Be sure to enable only one profiler at a time: for example, do not enable CPU and heap profiling simultaneously. Doing so can lead to erroneous observations.
pprof is extensible, and we can create our own custom profiles using pprof.Profile.
We have seen the most important profiles that we can enable to help us understand how an application performs and possible avenues for optimization. In general, enabling pprof is recommended, even in production, because in most cases it offers an excellent balance between its footprint and the amount of insight we can get from it. Some profiles, such as the CPU profile, lead to performance penalties but only during the time they are enabled.
The execution tracer is a tool that captures a wide range of runtime events with go tool to make them available for visualization. It is helpful for the following:
Understanding runtime events such as how the GC performs
Understanding how goroutines execute
Identifying poorly parallelized execution
Let\u2019s try it with an example given the Concurrency isn\u2019t Always Faster in Go section. We discussed two parallel versions of the merge sort algorithm. The issue with the first version was poor parallelization, leading to the creation of too many goroutines. Let\u2019s see how the tracer can help us in validating this statement.
We will write a benchmark for the first version and execute it with the -trace flag to enable the execution tracer:
$ go test -bench=. -v -trace=trace.out\n
Note
We can also download a remote trace file using the /debug/pprof/ trace?debug=0 pprof endpoint.
This command creates a trace.out file that we can open using go tool:
$ go tool trace trace.out\n2021/11/26 21:36:03 Parsing trace...\n2021/11/26 21:36:31 Splitting trace...\n2021/11/26 21:37:00 Opening browser. Trace viewer is listening on\n http://127.0.0.1:54518\n
The web browser opens, and we can click View Trace to see all the traces during a specific timeframe, as shown in the next figure. This figure represents about 150 ms. We can see multiple helpful metrics, such as the goroutine count and the heap size. The heap size grows steadily until a GC is triggered. We can also observe the activity of the Go application per CPU core. The timeframe starts with user-level code; then a \u201cstop the world\u201d is executed, which occupies the four CPU cores for approximately 40 ms.
Figure 6: Showing goroutine activity and runtime events such as a GC phase.
Regarding concurrency, we can see that this version uses all the available CPU cores on the machine. However, the next figure zooms in on a portion of 1 ms. Each bar corresponds to a single goroutine execution. Having too many small bars doesn\u2019t look right: it means execution that is poorly parallelized.
Figure 7: Too many small bars mean poorly parallelized execution.
The next figure zooms even closer to see how these goroutines are orchestrated. Roughly 50% of the CPU time isn\u2019t spent executing application code. The white spaces represent the time the Go runtime takes to spin up and orchestrate new goroutines.
Figure 8: About 50% of CPU time is spent handling goroutine switches.
Let\u2019s compare this with the second parallel implementation, which was about an order of magnitude faster. The next figure again zooms to a 1 ms timeframe.
Figure 9: The number of white spaces has been significantly reduced, proving that the CPU is more fully occupied.
Each goroutine takes more time to execute, and the number of white spaces has been significantly reduced. Hence, the CPU is much more occupied executing application code than it was in the first version. Each millisecond of CPU time is spent more efficiently, explaining the benchmark differences.
Note that the granularity of the traces is per goroutine, not per function like CPU profiling. However, it\u2019s possible to define user-level tasks to get insights per function or group of functions using the runtime/trace package.
For example, imagine a function that computes a Fibonacci number and then writes it to a global variable using atomic. We can define two different tasks:
var v int64\n// Creates a fibonacci task\nctx, fibTask := trace.NewTask(context.Background(), \"fibonacci\")\ntrace.WithRegion(ctx, \"main\", func() {\nv = fibonacci(10)\n})\nfibTask.End()\n// Creates a store task\nctx, fibStore := trace.NewTask(ctx, \"store\")\ntrace.WithRegion(ctx, \"main\", func() {\natomic.StoreInt64(&result, v)\n})\nfibStore.End()\n
Using go tool, we can get more precise information about how these two tasks perform. In the previous trace UI, we can see the boundaries for each task per goroutine. In User-Defined Tasks, we can follow the duration distribution:
Figure 10: Distribution of user-level tasks.
We see that in most cases, the fibonacci task is executed in less than 15 microseconds, whereas the store task takes less than 6309 nanoseconds.
In the previous section, we discussed the kinds of information we can get from CPU profiling. What are the main differences compared to the data we can get from user-level traces?
CPU profiling:
Sample-based
Per function
Doesn\u2019t go below the sampling rate (10 ms by default)
User-level traces:
Not sample-based
Per-goroutine execution (unless we use the runtime/trace package)
Time executions aren\u2019t bound by any rate
In summary, the execution tracer is a powerful tool for understanding how an application performs. As we have seen with the merge sort example, we can identify poorly parallelized execution. However, the tracer\u2019s granularity remains per goroutine unless we manually use runtime/trace compared to a CPU profile, for example. We can use both profiling and the execution tracer to get the most out of the standard Go diagnostics tools when optimizing an application.
"},{"location":"book/","title":"100 Go Mistakes and How to Avoid Them","text":"
Community space of \ud83d\udcd6 100 Go Mistakes and How to Avoid Them, published by Manning in 2022.
If you're a Go developer looking to improve your skills, this book is for you. With a focus on practical examples, 100 Go Mistakes and How to Avoid Them covers a wide range of topics from concurrency and error handling to testing and code organization. You'll learn to write more idiomatic, efficient, and maintainable code and become a proficient Go developer.
Read a summary of the 100 mistakes here.
"},{"location":"book/#quotes-and-ratings","title":"Quotes and Ratings","text":"
Krystian (Goodreads user)
This is an exceptional book. Usually, if a book contains either high-quality explanations or is written succinctly, I consider myself lucky to have found it. This one combines these two characteristics, which is super rare. It's another Go book for me and I still had quite a lot of \"a-ha!\" moments while reading it, and all of that without the unnecessary fluff, just straight to the point.
Akash Chetty
The book is completely exceptional, especially the examples carved out for each topic are really great. There is one topic that I struggled to understand is Concurrency but the way it is explained in this book is truly an art of genius.
Neeraj Shah
This should be the required reading for all Golang developers before they touch code in Production... It's the Golang equivalent of the legendary 'Effective Java' by Joshua Bloch.
Anupam Sengupta
Not having this will be the 101st mistake a Go programmer could make.
Manning, Goodreads, and Amazon reviews: 4.7/5 avg rating"},{"location":"book/#where-to-buy","title":"Where to Buy?","text":"
100 Go Mistakes and How to Avoid Them (\ud83c\uddec\ud83c\udde7 edition: paper, digital, or audiobook)
Manning (make sure to use my personal discount code for -35%: au35har)
Go\u8a00\u8a9e100Tips \u958b\u767a\u8005\u306b\u3042\u308a\u304c\u3061\u306a\u9593\u9055\u3044\u3078\u306e\u5bfe\u51e6\u6cd5 (\ud83c\uddef\ud83c\uddf5 edition: paper or digital)
Amazon: .co.jp
English and Japanese covers"},{"location":"book/#about-the-author","title":"About the Author","text":"
Teiva Harsanyi is a senior software engineer at Google. He has worked in various domains, including insurance, transportation, and safety-critical industries like air traffic management. He is passionate about Go and how to design and implement reliable systems.
"},{"location":"chapter-1/","title":"Go: Simple to learn but hard to master","text":"
This chapter covers
What makes Go an efficient, scalable, and productive language
Exploring why Go is simple to learn but hard to master
Presenting the common types of mistakes made by developers
Making mistakes is part of everyone\u2019s life. As Albert Einstein once said,
Albert Einstein
A person who never made a mistake never tried anything new.
What matters in the end isn\u2019t the number of mistakes we make, but our capacity to learn from them. This assertion also applies to programming. The seniority we acquire in a language isn\u2019t a magical process; it involves making many mistakes and learning from them. The purpose of this book is centered around this idea. It will help you, the reader, become a more proficient Go developer by looking at and learning from 100 common mistakes people make in many areas of the language.
This chapter presents a quick refresher as to why Go has become mainstream over the years. We\u2019ll discuss why, despite Go being considered simple to learn, mastering its nuances can be challenging. Finally, we\u2019ll introduce the concepts this book covers.
If you are reading this book, it\u2019s likely that you\u2019re already sold on Go. Therefore, this section provides a brief reminder about what makes Go such a powerful language.
Software engineering has evolved considerably during the past decades. Most modern systems are no longer written by a single person but by teams consisting of multiple programmers\u2014sometimes even hundreds, if not thousands. Nowadays, code must be readable, expressive, and maintainable to guarantee a system\u2019s durability over the years. Meanwhile, in our fast-moving world, maximizing agility and reducing the time to market is critical for most organizations. Programming should also follow this trend, and companies strive to ensure that software engineers are as productive as possible when reading, writing, and maintaining code.
In response to these challenges, Google created the Go programming language in 2007. Since then, many organizations have adopted the language to support various use cases: APIs, automation, databases, CLIs (command-line interfaces), and so on. Many today consider Go the language of the cloud.
Feature-wise, Go has no type inheritance, no exceptions, no macros, no partial functions, no support for lazy variable evaluation or immutability, no operator overloading, no pattern matching, and on and on. Why are these features missing from the language? The official Go FAQ gives us some insight:
Go FAQ
Why does Go not have feature X? Your favorite feature may be missing because it doesn\u2019t fit, because it affects compilation speed or clarity of design, or because it would make the fundamental system model too difficult.
Judging the quality of a programming language via its number of features is probably not an accurate metric. At least, it\u2019s not an objective of Go. Instead, Go utilizes a few essential characteristics when adopting a language at scale for an organization. These include the following:
Stability\u2014Even though Go receives frequent updates (including improvements and security patches), it remains a stable language. Some may even consider this one of the best features of the language.
Expressivity\u2014We can define expressivity in a programming language by how naturally and intuitively we can write and read code. A reduced number of keywords and limited ways to solve common problems make Go an expressive language for large codebases.
Compilation\u2014As developers, what can be more exasperating than having to wait for a build to test our application? Targeting fast compilation times has always been a conscious goal for the language designers. This, in turn, enables productivity.
Safety\u2014Go is a strong, statically typed language. Hence, it has strict compiletime rules, which ensure the code is type-safe in most cases.
Go was built from the ground up with solid features such as outstanding concurrency primitives with goroutines and channels. There\u2019s not a strong need to rely on external libraries to build efficient concurrent applications. Observing how important concurrency is these days also demonstrates why Go is such a suitable language for the present and probably for the foreseeable future.
Some also consider Go a simple language. And, in a sense, this isn\u2019t necessarily wrong. For example, a newcomer can learn the language\u2019s main features in less than a day. So why read a book centered on the concept of mistakes if Go is simple?
"},{"location":"chapter-1/#simple-doesnt-mean-easy","title":"Simple doesn\u2019t mean easy","text":"
There is a subtle difference between simple and easy. Simple, applied to a technology, means not complicated to learn or understand. However, easy means that we can achieve anything without much effort. Go is simple to learn but not necessarily easy to master.
Let\u2019s take concurrency, for example. In 2019, a study focusing on concurrency bugs was published: Understanding Real-World Concurrency Bugs in Go. This study was the first systematic analysis of concurrency bugs. It focused on multiple popular Go repositories such as Docker, gRPC, and Kubernetes. One of the most important takeaways from this study is that most of the blocking bugs are caused by inaccurate use of the message-passing paradigm via channels, despite the belief that message passing is easier to handle and less error-prone than sharing memory.
What should be an appropriate reaction to such a takeaway? Should we consider that the language designers were wrong about message passing? Should we reconsider how we deal with concurrency in our project? Of course not.
It\u2019s not a question of confronting message passing versus sharing memory and determining the winner. However, it\u2019s up to us as Go developers to thoroughly understand how to use concurrency, its implications on modern processors, when to favor one approach over the other, and how to avoid common traps. This example highlights that although a concept such as channels and goroutines can be simple to learn, it isn\u2019t an easy topic in practice.
This leitmotif\u2014simple doesn\u2019t mean easy\u2014can be generalized to many aspects of Go, not only concurrency. Hence, to be proficient Go developers, we must have a thorough understanding of many aspects of the language, which requires time, effort, and mistakes.
This book aims to help accelerate our journey toward proficiency by delving into 100 Go mistakes.
"},{"location":"chapter-1/#100-go-mistakes","title":"100 Go mistakes","text":"
Why should we read a book about common Go mistakes? Why not deepen our knowledge with an ordinary book that would dig into different topics?
In a 2011 article, neuroscientists proved that the best time for brain growth is when we\u2019re facing mistakes. 1 Haven\u2019t we all experienced the process of learning from a mistake and recalling that occasion after months or even years, when some context related to it? As presented in another article, by Janet Metcalfe, this happens because mistakes have a facilitative effect. 2 The main idea is that we can remember not only the error but also the context surrounding the mistake. This is one of the reasons why learning from mistakes is so efficient.
To strengthen this facilitative effect, this book accompanies each mistake as much as possible with real-world examples. This book isn\u2019t only about theory; it also helps us get better at avoiding mistakes and making more well-informed, conscious decisions because we now understand the rationale behind them.
Unknown
Tell me and I forget. Teach me and I remember. Involve me and I learn.
This book presents seven main categories of mistakes. Overall, the mistakes can be classified as
The first type of mistake and probably the most obvious is software bugs. In 2020, a study conducted by Synopsys estimated the cost of software bugs in the U.S. alone to be over $2 trillion. 3
Furthermore, bugs can also lead to tragic impacts. We can, for example, mention cases such as Therac-25, a radiation therapy machine produced by Atomic Energy of Canada Limited (AECL). Because of a race condition, the machine gave its patients radiation doses that were hundreds of times greater than expected, leading to the death of three patients. Hence, software bugs aren\u2019t only about money. As developers, we should remember how impactful our jobs are.
This book covers plenty of cases that could lead to various software bugs, including data races, leaks, logic errors, and other defects. Although accurate tests should be a way to discover such bugs as early as possible, we may sometimes miss cases because of different factors such as time constraints or complexity. Therefore, as a Go developer, it\u2019s essential to make sure we avoid common bugs.
The next category of mistakes is related to unnecessary complexity. A significant part of software complexity comes from the fact that, as developers, we strive to think about imaginary futures. Instead of solving concrete problems right now, it can be tempting to build evolutionary software that could tackle whatever future use case arises. However, this leads to more drawbacks than benefits in most cases because it can make a codebase more complex to understand and reason about.
Getting back to Go, we can think of plenty of use cases where developers might be tempted to design abstractions for future needs, such as interfaces or generics. This book discusses topics where we should remain careful not to harm a codebase with needless complexity.
Another kind of mistake is to weaken readability. As Robert C. Martin wrote in his book Clean Code: A Handbook of Agile Software Craftsmanship, the ratio of time spent reading versus writing is well over 10 to 1. Most of us started to program on solo projects where readability wasn\u2019t that important. However, today\u2019s software engineering is programming with a time dimension: making sure we can still work with and maintain an application months, years, or perhaps even decades later.
When programming in Go, we can make many mistakes that can harm readability. These mistakes may include nested code, data type representations, or not using named result parameters in some cases. Throughout this book, we will learn how to write readable code and care for future readers (including our future selves).
"},{"location":"chapter-1/#suboptimal-or-unidiomatic-organization","title":"Suboptimal or unidiomatic organization","text":"
Be it while working on a new project or because we acquire inaccurate reflexes, another type of mistake is organizing our code and a project suboptimally and unidiomatically. Such issues can make a project harder to reason about and maintain. This book covers some of these common mistakes in Go. For example, we\u2019ll look at how to structure a project and deal with utility packages or init functions. All in all, looking at these mistakes should help us organize our code and projects more efficiently and idiomatically.
"},{"location":"chapter-1/#lack-of-api-convenience","title":"Lack of API convenience","text":"
Making common mistakes that weaken how convenient an API is for our clients is another type of mistake. If an API isn\u2019t user-friendly, it will be less expressive and, hence, harder to understand and more error-prone.
We can think about many situations such as overusing any types, using the wrong creational pattern to deal with options, or blindly applying standard practices from object-oriented programming that affect the usability of our APIs. This book covers common mistakes that prevent us from exposing convenient APIs for our users.
Under-optimized code is another type of mistake made by developers. It can happen for various reasons, such as not understanding language features or even a lack of fundamental knowledge. Performance is one of the most obvious impacts of this mistake, but not the only one.
We can think about optimizing code for other goals, such as accuracy. For example, this book provides some common techniques to ensure that floating-point operations are accurate. Meanwhile, we will cover plenty of cases that can negatively impact performance code because of poorly parallelized executions, not knowing how to reduce allocations, or the impacts of data alignment, for example. We will tackle optimization via different prisms.
"},{"location":"chapter-1/#lack-of-productivity","title":"Lack of productivity","text":"
In most cases, what\u2019s the best language we can choose when working on a new project? The one we\u2019re the most productive with. Being comfortable with how a language works and exploiting it to get the best out of it is crucial to reach proficiency.
In this book, we will cover many cases and concrete examples that will help us to be more productive while working in Go. For instance, we\u2019ll look at writing efficient tests to ensure that our code works, relying on the standard library to be more effective, and getting the best out of the profiling tools and linters. Now, it\u2019s time to delve into those 100 common Go mistakes.
Go is a modern programming language that enables developer productivity, which is crucial for most companies today.
Go is simple to learn but not easy to master. This is why we need to deepen our knowledge to make the most effective use of the language.
Learning via mistakes and concrete examples is a powerful way to be proficient in a language. This book will accelerate our path to proficiency by exploring 100 common mistakes.
J. S. Moser, H. S. Schroder, et al., \u201cMind Your Errors: Evidence for a Neural Mechanism Linking Growth Mindset to Adaptive Posterror Adjustments,\u201d Psychological Science, vol. 22, no. 12, pp. 1484\u20131489, Dec. 2011.\u00a0\u21a9
J. Metcalfe, \u201cLearning from Errors,\u201d Annual Review of Psychology, vol. 68, pp. 465\u2013489, Jan. 2017.\u00a0\u21a9
Synopsys, \u201cThe Cost of Poor Software Quality in the US: A 2020 Report.\u201d 2020. https://news.synopsys.com/2021-01-06-Synopsys-Sponsored-CISQ-Research-Estimates-Cost-of-Poor-Software-Quality-in-the-US-2-08-Trillion-in-2020.\u00a0\u21a9
\u3053\u306e\u30da\u30fc\u30b8\u306f\u300e100 Go Mistakes\u300f\u306e\u5185\u5bb9\u3092\u307e\u3068\u3081\u305f\u3082\u306e\u3067\u3059\u3002\u4e00\u65b9\u3067\u3001\u30b3\u30df\u30e5\u30cb\u30c6\u30a3\u306b\u958b\u304b\u308c\u305f\u30da\u30fc\u30b8\u3067\u3082\u3042\u308a\u307e\u3059\u3002\u300c\u3042\u308a\u304c\u3061\u306a\u9593\u9055\u3044\u300d\u304c\u65b0\u305f\u306b\u8ffd\u52a0\u3055\u308c\u308b\u3079\u304d\u3060\u3068\u304a\u8003\u3048\u3067\u3057\u305f\u3089 community mistake issue \u3092\u4f5c\u6210\u3057\u3066\u304f\u3060\u3055\u3044\u3002
if \u30d6\u30ed\u30c3\u30af\u304c\u8fd4\u3055\u308c\u308b\u3068\u304d\u3001\u3059\u3079\u3066\u306e\u5834\u5408\u306b\u304a\u3044\u3066 else \u30d6\u30ed\u30c3\u30af\u3092\u7701\u7565\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002 \u305f\u3068\u3048\u3070\u3001\u6b21\u306e\u3088\u3046\u306b\u66f8\u304f\u3079\u304d\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3002
if foo() {\n// ...\nreturn true\n} else {\n// ...\n}\n
if s != \"\" {\n// ...\n} else {\nreturn errors.New(\"empty string\")\n}\n
\u3053\u3053\u3067\u306f\u3001\u7a7a\u306e s \u304c\u30ce\u30f3\u30cf\u30c3\u30d4\u30fc\u30d1\u30b9\u3092\u8868\u3057\u307e\u3059\u3002\u3057\u305f\u304c\u3063\u3066\u3001\u6b21\u306e\u3088\u3046\u306b\u6761\u4ef6\u3092\u3072\u3063\u304f\u308a\u8fd4\u3059\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002
if s == \"\" {\nreturn errors.New(\"empty string\")\n}\n// ...\n
\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u306f\u3001\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u306e\u52d5\u4f5c\u3092\u6307\u5b9a\u3059\u308b\u65b9\u6cd5\u3092\u63d0\u4f9b\u3057\u307e\u3059\u3002\u8907\u6570\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u304c\u5b9f\u88c5\u3067\u304d\u308b\u5171\u901a\u9805\u3092\u62bd\u51fa\u3059\u308b\u305f\u3081\u306b\u3001\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u306f\u4f7f\u7528\u3055\u308c\u307e\u3059\u3002Go\u8a00\u8a9e\u306e\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u304c\u5927\u304d\u304f\u7570\u306a\u308b\u306e\u306f\u3001\u6697\u9ed9\u7684\u306b\u6e80\u305f\u3055\u308c\u308b\u3053\u3068\u3067\u3059\u3002\u30aa\u30d6\u30b8\u30a7\u30af\u30c8 X \u304c\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9 Y \u3092\u5b9f\u88c5\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u793a\u3059 implements \u306e\u3088\u3046\u306a\u660e\u793a\u7684\u306a\u30ad\u30fc\u30ef\u30fc\u30c9\u306f\u3042\u308a\u307e\u305b\u3093\u3002
Is your company hiring? Sponsor this repository and let a significant audience of Go developers know about your opportunities in this section (book sales: +8k, website traffic: +2000 views and +400 unique visitors per week).
\u53ea\u6709\u5728\u9700\u8981\u63a5\u53d7\u6216\u8fd4\u56de\u4efb\u610f\u7c7b\u578b\u65f6\uff0c\u624d\u4f7f\u7528 any\uff0c\u4f8b\u5982 json.Marshal\u3002\u5176\u4ed6\u60c5\u51b5\u4e0b\uff0c\u56e0\u4e3a any \u4e0d\u63d0\u4f9b\u6709\u610f\u4e49\u7684\u4fe1\u606f\uff0c\u53ef\u80fd\u4f1a\u5bfc\u81f4\u7f16\u8bd1\u65f6\u95ee\u9898\uff0c\u5982\u5141\u8bb8\u8c03\u7528\u8005\u8c03\u7528\u65b9\u6cd5\u5904\u7406\u4efb\u610f\u7c7b\u578b\u6570\u636e\u3002
\u5728 Go \u4e2d\u6574\u6570\u4e0a\u6ea2\u51fa\u548c\u4e0b\u6ea2\u662f\u9759\u9ed8\u5904\u7406\u7684\uff0c\u6240\u4ee5\u4f60\u53ef\u4ee5\u5b9e\u73b0\u81ea\u5df1\u7684\u51fd\u6570\u6765\u6355\u83b7\u5b83\u4eec\u3002
"},{"location":"zh/#_3","title":"\u63a7\u5236\u7ed3\u6784","text":""},{"location":"zh/#range-30","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u53d8\u91cf\u662f\u4e00\u4e2a\u62f7\u8d1d (#30)","text":"
range \u5faa\u73af\u4e2d\u7684\u5faa\u73af\u53d8\u91cf\u662f\u904d\u5386\u5bb9\u5668\u4e2d\u5143\u7d20\u503c\u7684\u4e00\u4e2a\u62f7\u8d1d\u3002\u56e0\u6b64\uff0c\u5982\u679c\u5143\u7d20\u503c\u662f\u4e00\u4e2astruct\u5e76\u4e14\u60f3\u5728 range \u4e2d\u4fee\u6539\u5b83\uff0c\u53ef\u4ee5\u901a\u8fc7\u7d22\u5f15\u503c\u6765\u8bbf\u95ee\u5e76\u4fee\u6539\u5b83\uff0c\u6216\u8005\u4f7f\u7528\u7ecf\u5178\u7684for\u5faa\u73af+\u7d22\u5f15\u503c\u7684\u5199\u6cd5\uff08\u9664\u975e\u904d\u5386\u7684\u5143\u7d20\u662f\u4e00\u4e2a\u6307\u9488\uff09\u3002
"},{"location":"zh/#range-channels-arrays-31","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u4e2d\u8fed\u4ee3\u76ee\u6807\u503c\u7684\u8ba1\u7b97\u65b9\u5f0f (channels \u548c arrays) (#31)","text":"
\u4f20\u9012\u7ed9 range \u64cd\u4f5c\u7684\u8fed\u4ee3\u76ee\u6807\u5bf9\u5e94\u7684\u8868\u8fbe\u5f0f\u7684\u503c\uff0c\u53ea\u4f1a\u5728\u5faa\u73af\u6267\u884c\u524d\u88ab\u8ba1\u7b97\u4e00\u6b21\uff0c\u7406\u89e3\u8fd9\u4e2a\u6709\u52a9\u4e8e\u907f\u514d\u72af\u4e00\u4e9b\u5e38\u89c1\u7684\u9519\u8bef\uff0c\u4f8b\u5982\u4e0d\u9ad8\u6548\u7684channel\u8d4b\u503c\u64cd\u4f5c\u3001slice\u8fed\u4ee3\u64cd\u4f5c\u3002
"},{"location":"zh/#range-range-loops-32","title":"\u5ffd\u7565\u4e86 range \u5faa\u73af\u4e2d\u6307\u9488\u5143\u7d20\u7684\u5f71\u54cd range loops (#32)","text":"
\u8fd9\u91cc\u5176\u5b9e\u5f3a\u8c03\u7684\u662f range \u8fed\u4ee3\u8fc7\u7a0b\u4e2d\uff0c\u8fed\u4ee3\u53d8\u91cf\u5b9e\u9645\u4e0a\u662f\u4e00\u4e2a\u62f7\u8d1d\uff0c\u5047\u8bbe\u7ed9\u53e6\u5916\u4e00\u4e2a\u5bb9\u5668\u5143\u7d20\uff08\u6307\u9488\u7c7b\u578b\uff09\u8d4b\u503c\uff0c\u4e14\u9700\u8981\u5bf9\u8fed\u4ee3\u53d8\u91cf\u53d6\u5730\u5740\u8f6c\u6362\u6210\u6307\u9488\u518d\u8d4b\u503c\u7684\u8bdd\uff0c\u8fd9\u91cc\u6f5c\u85cf\u7740\u4e00\u4e2a\u9519\u8bef\uff0c\u5c31\u662ffor\u5faa\u73af\u8fed\u4ee3\u53d8\u91cf\u662f per-variable-per-loop \u800c\u4e0d\u662f per-variable-per-iteration\u3002\u5982\u679c\u662f\u901a\u8fc7\u5c40\u90e8\u53d8\u91cf\uff08\u7528\u8fed\u4ee3\u53d8\u91cf\u6765\u521d\u59cb\u5316\uff09\u6216\u8005\u4f7f\u7528\u7d22\u5f15\u503c\u6765\u76f4\u63a5\u5f15\u7528\u8fed\u4ee3\u7684\u5143\u7d20\uff0c\u5c06\u6709\u52a9\u4e8e\u907f\u514d\u62f7\u8d1d\u6307\u9488(\u8fed\u4ee3\u53d8\u91cf\u7684\u5730\u5740)\u4e4b\u7c7b\u7684bug\u3002
ps\uff1a\u6570\u636e\u7ade\u4e89\u662f\u7ade\u6001\u6761\u4ef6\u7684\u5b50\u96c6\uff0c\u7ade\u6001\u6761\u4ef6\u4e0d\u4ec5\u5c40\u9650\u4e8e\u8bbf\u5b58\u672a\u540c\u6b65\uff0c\u5b83\u53ef\u4ee5\u53d1\u751f\u5728\u66f4\u9ad8\u7684\u5c42\u9762\u3002go test -race \u68c0\u6d4b\u7684\u662f\u6570\u636e\u7ade\u4e89\uff0c\u9700\u8981\u540c\u6b65\u6765\u89e3\u51b3\uff0c\u800c\u5f00\u53d1\u8005\u8fd8\u9700\u8981\u5173\u6ce8\u9762\u66f4\u5e7f\u7684\u7ade\u6001\u6761\u4ef6\uff0c\u5b83\u9700\u8981\u5bf9\u591a\u4e2agoroutines\u7684\u6267\u884c\u8fdb\u884c\u7f16\u6392\u3002
\u7406\u89e3 Go \u7684\u5185\u5b58\u6a21\u578b\u4ee5\u53ca\u6709\u5173\u987a\u5e8f\u548c\u540c\u6b65\u7684\u5e95\u5c42\u4fdd\u8bc1\u662f\u9632\u6b62\u53ef\u80fd\u7684\u6570\u636e\u7ade\u4e89\u548c\u7ade\u6001\u6761\u4ef6\u7684\u5173\u952e\u3002
Go \u7684\u4e0a\u4e0b\u6587\uff08context\uff09\u4e5f\u662f Go \u5e76\u53d1\u7f16\u7a0b\u7684\u57fa\u77f3\u4e4b\u4e00\u3002\u4e0a\u4e0b\u6587\u5141\u8bb8\u60a8\u643a\u5e26\u622a\u6b62\u65f6\u95f4\u3001\u53d6\u6d88\u4fe1\u53f7\u548c\u952e\u503c\u5217\u8868\u3002
\u8981\u660e\u767d\uff0cselect \u591a\u4e2achannels\u65f6\uff0c\u5982\u679c\u591a\u4e2achannels\u4e0a\u7684\u64cd\u4f5c\u90fd\u5c31\u7eea\uff0c\u90a3\u4e48\u4f1a\u968f\u673a\u9009\u62e9\u4e00\u4e2a case \u5206\u652f\u6765\u6267\u884c\uff0c\u56e0\u6b64\u8981\u907f\u514d\u6709\u5206\u652f\u9009\u62e9\u987a\u5e8f\u662f\u4ece\u4e0a\u5230\u4e0b\u7684\u8fd9\u79cd\u9519\u8bef\u9884\u8bbe\uff0c\u8fd9\u53ef\u80fd\u4f1a\u5bfc\u81f4\u8bbe\u8ba1\u4e0a\u7684bug\u3002
\u6ce8\u610f\u6709\u4e9b\u51fd\u6570\u63a5\u6536\u4e00\u4e2a time.Duration \u7c7b\u578b\u7684\u53c2\u6570\u65f6\uff0c\u5c3d\u7ba1\u76f4\u63a5\u4f20\u9012\u4e00\u4e2a\u6574\u6570\u662f\u53ef\u4ee5\u7684\uff0c\u4f46\u6700\u597d\u8fd8\u662f\u4f7f\u7528 time API \u4e2d\u7684\u65b9\u6cd5\u6765\u4f20\u9012 duration\uff0c\u4ee5\u907f\u514d\u53ef\u80fd\u9020\u6210\u7684\u56f0\u60d1\u3001bug\u3002
\u5f53\u63d0\u4f9b\u4e00\u4e2amap\u7528\u6765unmarshal JSON\u6570\u636e\u65f6\uff0c\u4e3a\u4e86\u907f\u514d\u4e0d\u786e\u5b9a\u7684value\u7ed3\u6784\u6211\u4eec\u4f1a\u4f7f\u7528 any \u6765\u4f5c\u4e3avalue\u7684\u7c7b\u578b\u800c\u4e0d\u662f\u5b9a\u4e49\u4e00\u4e2astruct\uff0c\u8fd9\u79cd\u60c5\u51b5\u4e0b\u9700\u8981\u8bb0\u5f97\u6570\u503c\u9ed8\u8ba4\u4f1a\u88ab\u8f6c\u6362\u4e3a float64\u3002
"},{"location":"zh/#time-api-87","title":"\u6ca1\u6709\u9ad8\u6548\u5730\u5904\u7406 time API (#87)","text":"
\u7406\u89e3\u5982\u4f55\u5904\u7406\u4f7f\u7528 time API \u7684\u51fd\u6570\uff0c\u662f\u4f7f\u6d4b\u8bd5\u66f4\u52a0\u7a33\u5b9a\u7684\u53e6\u4e00\u79cd\u65b9\u5f0f\u3002\u60a8\u53ef\u4ee5\u4f7f\u7528\u6807\u51c6\u6280\u672f\uff0c\u4f8b\u5982\u5c06\u65f6\u95f4\u4f5c\u4e3a\u9690\u85cf\u4f9d\u8d56\u9879\u7684\u4e00\u90e8\u5206\u6765\u5904\u7406\uff0c\u6216\u8005\u8981\u6c42\u5ba2\u6237\u7aef\u63d0\u4f9b\u65f6\u95f4\u3002
