spec: less error-prone loop variable scoping #60078

rsc · 2023-05-09T15:36:18Z

We propose to change for loop variables declared with := from one-instance-per-loop to one-instance-per-iteration. This change would apply only to packages in modules that explicitly declare a new enough Go version in the go.mod file, so all existing code is unaffected. Changing the loop semantics would prevent unintended sharing in per-iteration closures and goroutines (see this entry in the Go FAQ for one explanation). We expect this change to fix many subtly broken for loops in new code, as well as in old code as it is updated to the newer Go version. There was an earlier pre-proposal discussion of this idea at #56010.

For example, consider a loop like the one in this test:

func TestAllEven(t *testing.T) {
	testCases := []int{0, 2, 4, 6}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
}

This test aims to check that all the test cases are even, but it doesn't check them all. The problem is that t.Parallel stops the closure and lets the loop continue, and then it runs all the closures in parallel when the loop is over and TestAllEven has returned. By the time the if statement in the closure executes, the loop is done, and v has its final iteration value, 6. All four subtests now continue executing in parallel, and they all check that 6 is even, instead of checking each of the test cases.

Most Go developers are familiar with this mistake and know the answer: add v := v to the loop body:

func TestAllEven(t *testing.T) {
	testCases := []int{0, 2, 4, 6}
	for _, v := range testCases {
		v := v // MAKE ITERATION VALUE SHARING BUG GO AWAY
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
}

Changing the loop semantics would in essence insert this kind of v := v statement for every for loop variable declared with :=. It would fix this loop and many others to do what the author clearly intends.

A subtler variation is when the code says testCases := []int{1, 2, 4, 6} (note the non-even test case 1):

func TestAllEvenBuggy(t *testing.T) {
	testCases := []int{1, 2, 4, 6}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
}

TestAllEvenBuggy passes today, because the test is only checking 6, four times. Changing the loop semantics would still fix the loop to do what the author clearly intended, but it would break the test, because the test is passing incorrectly. So there is the potential to cause problems for users.

Because of this potential for problems, the new loop semantics would only apply in Go modules that have opted in to the release with the new loops. If that was Go 1.22, then only packages in a module with a go.mod that says go 1.22 would get the new loop semantics. Packages in other modules, including packages in dependencies, would get the old semantics. This would guarantee that all existing Go code keeps working the same as it does today, even when compiling with a new toolchain. People only need to debug loop changes when they opt in to the new semantics in go.mod. This approach is in keeping with our backwards and forwards compatibility work, #56986 and #57001, specifically the principle that toolchain upgrades preserve the behavior of old code, and compatibility is based on the go line.

If this proposal is accepted, users will need additional tooling support for a successful transition. That would come primarily in two forms: a compiler mode that reports affected loops, and a debugging tool that identifies the specific loops whose changed compilation is responsible for causing a test failure. There is a demo of the tooling support at the end of this comment.

The tooling support is important and necessary, but we expect it to be needed only rarely. Analysis and conversion of Google's own Go source code found that only about 1 package test per 8,000 started failing due to the new semantics, and the bug was essentially always in the test itself, like in TestAllEvenBuggy. In contrast, the new loopclosure vet analysis that shipped in Go 1.20 flagged definite bugs in 1 test per 400. The rest stayed passing with their bugs fixed by the new semantics. That is, in Google's code base, about 5% of the tests that contain this kind of sharing mistake were like TestAllEvenBuggy, exposed as buggy by the new semantics. The other 95% of the tests with this kind of mistake still passed when they started testing what they intended to. Of all the test failures, only one was caused by a loop variable semantic test change in non-test code. That code was very low-level and could not tolerate a new allocation in the loop. The design document has details. This evidence suggests that changing the semantics is usually a no-op, and when it’s not, it fixes buggy code far more often than it breaks correct code.

Based on the preliminary discussion the further work summarized here, @dr2chase and I propose that we make the change in an appropriate future Go version, perhaps Go 1.22 if the stars align, and otherwise a later version.

More details can be found in the design document.

Update, May 10 2023: Note that the change will not break the common idiom of changing a loop variable in the body of a 3-clause for loops. See this comment for details and links before commenting about 3-clause for loops. Thanks.

Tooling Demonstration

To demonstrate the tooling we would provide to support a successful transition, here is a complete example of a test that passes today but fails with the new loop semantics:

% cat x_test.go
package x

import "testing"

func Test(t *testing.T) {
	testCases := []int{1, 2, 4, 6}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Log(v)
		})
	}
	for _, v := range testCases {
		t.Run("sub", func(t *testing.T) {
			t.Parallel()
			if v&1 != 0 {
				t.Fatal("odd v", v)
			}
		})
	}
}
%

Building the program with -d=loopvar=2 reports all the affected loops:

% GOEXPERIMENT=loopvar go test -gcflags=all=-d=loopvar=2 x_test.go
# runtime
go/src/runtime/proc.go:1815:7: loop variable freem now per-iteration, stack-allocated
go/src/runtime/proc.go:3149:7: loop variable enum now per-iteration, stack-allocated
go/src/runtime/mgcmark.go:810:6: loop variable d now per-iteration, stack-allocated
go/src/runtime/traceback.go:623:7: loop variable iu now per-iteration, stack-allocated
go/src/runtime/traceback.go:943:7: loop variable iu now per-iteration, stack-allocated
# runtime/pprof
go/src/runtime/pprof/pprof.go:386:9: loop variable r now per-iteration, heap-allocated
go/src/runtime/pprof/proto.go:363:6: loop variable e now per-iteration, stack-allocated
go/src/runtime/pprof/proto.go:612:9: loop variable frame now per-iteration, heap-allocated
go/src/runtime/pprof/protomem.go:29:9: loop variable r now per-iteration, heap-allocated
# command-line-arguments [command-line-arguments.test]
./x_test.go:7:9: loop variable v now per-iteration, stack-allocated
./x_test.go:15:9: loop variable v now per-iteration, stack-allocated
./x_test.go:20:9: loop variable v now per-iteration, stack-allocated
# internal/fuzz
go/src/internal/fuzz/fuzz.go:432:9: loop variable e now per-iteration, stack-allocated
--- FAIL: Test (0.00s)
    --- FAIL: Test/sub (0.00s)
        x_test.go:11: odd v 1
    --- FAIL: Test/sub#08 (0.00s)
        x_test.go:24: odd v 1
FAIL
FAIL	command-line-arguments	0.199s
FAIL
%

Note that some loops are in the Go standard library. Those are unlikely to be the cause, so we can limit the diagnostics to the current package by dropping all=:

% GOEXPERIMENT=loopvar go test -gcflags=-d=loopvar=2 x_test.go
# command-line-arguments [command-line-arguments.test]
./x_test.go:7:9: loop variable v now per-iteration, stack-allocated
./x_test.go:15:9: loop variable v now per-iteration, stack-allocated
./x_test.go:20:9: loop variable v now per-iteration, stack-allocated
--- FAIL: Test (0.00s)
    --- FAIL: Test/sub (0.00s)
        x_test.go:11: odd v 1
    --- FAIL: Test/sub#08 (0.00s)
        x_test.go:24: odd v 1
FAIL
FAIL	command-line-arguments	0.119s
FAIL
%

That trims the diagnostic output, but it still leaves us to check all our loops. In this trivial example, 2 of 3 are buggy, but in a real program there are fewer needles and more haystack. To solve that problem, we can use a dynamic tool that reruns a test, varying which loops compile with the new semantics, to identify the specific loops that cause the failure:

% bisect -loopvar go test x_test.go
bisect: checking target with all changes disabled
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=n x_test.go... ok (13 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=n x_test.go... ok (13 matches)
bisect: checking target with all changes enabled
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=y x_test.go... FAIL (13 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=y x_test.go... FAIL (13 matches)
bisect: target succeeds with no changes, fails with all changes
bisect: searching for minimal set of changes to enable to cause failure
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+0 x_test.go... ok (7 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+0 x_test.go... ok (7 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+1 x_test.go... FAIL (6 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+1 x_test.go... FAIL (6 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+01 x_test.go... FAIL (3 matches)
bisect: run: go test -gcflags=all=-d=loopvar=1,loopvarhash=+01 x_test.go... FAIL (3 matches)
...
bisect: FOUND failing change set
--- change set #1 (enabling changes causes failure)
./x_test.go:7:9
---
...
bisect: FOUND failing change set
--- change set #2 (enabling changes causes failure)
./x_test.go:20:9
---
...
bisect: target succeeds with all remaining changes enabled
%

Now we know to spend our attention on the loops at lines 7 and 20, which are in fact the buggy ones. (The loop at line 15 has been cleared of suspicion.) Bisect uses an adaptation of binary search that can identify the relevant loop in a relatively small number of trials, making it useful even on very large, very complicated tests that run for a long time.

The text was updated successfully, but these errors were encountered:

seebs · 2023-05-09T16:02:07Z

The biggest concern I could suggest would be the compatibility promise, and the go.mod stuff addresses that.

When I started learning Go, I found a page which purported to be a list of common Go pitfalls. It said "pitfalls", plural, but in fact, this was the only one. That has been my experience of using the language for the last few years; this is the pitfall in Go programming.

Do it. Doooo eeeeet.

ulikunitz · 2023-05-10T06:01:32Z

It may be pedantic, but packages in modules with go.mod files that specify go 1.22 or a higher go version should get the new semantics.

go101 · 2023-05-10T06:09:56Z

Personally, I strongly oppose to apply this proposal to for ...; ...; ... {...} loops.

The proposal docs mentions that the semantic-equivalent form of

for i := 0; i < 3; i++ {
    prints = append(prints, func() { println(i) })
}

is like

{
	i_outer := 0
	first := true
	for {
		i := i_outer
		if first {
			first = false
		} else {
			i++
		}
		if !(i < 3) {
			break
		}
		prints = append(prints, func() { println(i) })
		i_outer = i
	}
}

Sorry, this adds too much magical implicitness and makes it is hard to make a clear explanation for the new semantic.
And it breaks the try-to-be-explicit principle in Go design.
Worse, this breaks most people's expectation. The iteration variable declaration statement is only executed once, but there will be multiple instances of it created at run time.

Obeying users' intuitions is important. This is why I support applying this proposal to for-range loops.

[update]: I don't think the so-called "accidental" bugs in using for ...; ...; ... {...} loops mentioned in the proposal docs are caused by mis-understanding the semantic of for ...; ...; ... {...} loops. IMHO, the cause of the bug in

var ids []*int
for i := 0; i < 10; i++ {
	ids = append(ids, &i)
}

is totally the same as the bug in

var ids []*int
var i int
for i = 0; i < 10; i++ {
	ids = append(ids, &i)
}

In both, I think most people think there is only one instance of the variable i during the whole loop execution.

zikaeroh · 2023-05-10T06:15:47Z

@go101 That is not the code as mentioned in the proposal; you have added an extra outer loop where the proposal only had a block.

go101 · 2023-05-10T06:18:38Z

@zikaeroh corrected now. It doesn't affect the opinion.

Merovius · 2023-05-10T06:22:00Z

@go101 to be clear, you're only showing the "after" part of that kind of expansion, as far as this proposal is concerned. Currently, the equivalent expansion would be

{
	i := 0 // with proposal this is i_outer
	first := true
	for {
                // with proposal, there's an i := i_outer here
		if first {
			first = false
		} else {
			i++
		}
		if !(i < 3) {
			break
		}
		prints = append(prints, func() { println(i) })
                // with proposal, there's an i_outer = i here
	}
}

i.e. the proposal is to add i := i_outer; … ; i_outer = i, which is significantly less "added magic" than your comment makes it appear.

go101 · 2023-05-10T06:52:21Z

@Merovius Yes, that was a small mistake which has been corrected. It doesn't affect the whole opinion.

davidmdm · 2023-05-10T14:10:50Z

Overall I am very much for this proposal. However @go101 did bring up a valid point about for ; ; {} loops as opposed to range loops.

I would just want somebody to confirm the behavior of the following loop under the new semantics:

for i := 0; i < 5; i++ {
  fmt.Println(i)
  i += 1
}

Under the new semantics I believe it would print: 0, 1, 2, 3, 4 instead of: 0, 2, 4.

This might be a little unexpected to users who do not know this history, and the bugs associated with capturing the loop variable.

To reiterate (unintentional pun), I think the proposal is beneficial to Go but this is my only worry.

go101 · 2023-05-10T14:16:00Z

@davidmdm
The semantic change doesn't affect the behavior of your example. It will still print 0 2 4.

The change does affect the behaviors of some use cases.

Merovius · 2023-05-10T14:34:14Z

@davidmdm I haven't seen any sort of convincing argument, why the three-clause for loop would be different than a range loop. Any argument I've seen is just as applicable to both.

Meanwhile "having two loop constructs which behave extremely differently in relation to short variable declarations is confusing and hard to explain" seems obvious and hard to refute to me.

YMMV, of course.

davidmdm · 2023-05-10T15:29:30Z

@Merovius
It's different than most other imperative programming languages. If we take JS for example:

for (let i = 0; i < 5; i++) {
  console.log(i)
  i++
}

This will print the sequence 0, 2, 4. The same goes for python, and most other high level languages.

Go would be unique (maybe, I won't pretend to know most languages...) in this regard.

I am still for it, because it solves subtle bugs with regards to closures and addressability that I have fallen prey to before.
What I am looking for, perhaps, is an acknowledgement that Go would adopt a slightly different for-loop semantic than other languages and that this may be confusing to new-comers. Beyond that, is Go going to provide go vet checks or something to help users not write poor code in the new semantics?

For example:

for i := 0; i < 5; i++ {
  fmt.Println(i)
  i += 1 // go vet: ineffective assignment -> iteration variable moving out of scope
}

zikaeroh · 2023-05-10T15:34:38Z

@davidmdm Maybe you missed it, but the proposal has the "transformed" code copy the final state of the loop variable to the beginning of the next loop, meaning that the above should also print ”0, 2, 4", and that += 1 is not actually ineffective.

{
	i_outer := 0
	first := true
	for {
		i := i_outer
		if first {
			first = false
		} else {
			i++
		}
		if !(i < 5) {
			break
		}
		fmt.Println(i)
		i += 1
		i_outer = i
	}
}

The only semantic change this proposal should have is to do with capturing the loop variable by address or by a closure.

rsc · 2023-05-10T15:35:38Z

@davidmdm You misunderstand the new semantics. They are basically the same as the Javascript let form you mentioned. The loop you showed does not change meaning. This kind of loop is explicitly called out at the end of the https://go.googlesource.com/proposal/+/master/design/60078-loopvar.md#language-specification section. ^F for "Note that in both 3-clause..."

I would encourage you to try any programs you like on your own computer, using gotip with GOEXPERIMENT=loopvar.

% go install golang.org/dl/gotip@latest
go: downloading golang.org/dl v0.0.0-20230502172222-5216546bad51
% gotip download
Updating the go development tree...
...
Building Go cmd/dist using /Users/rsc/sdk/go1.17.13. (go1.17.13 darwin/amd64)
Building Go toolchain1 using /Users/rsc/sdk/go1.17.13.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /Users/rsc/sdk/gotip
Installed commands in /Users/rsc/sdk/gotip/bin
*** You need to add /Users/rsc/sdk/gotip/bin to your PATH.
Success. You may now run 'gotip'!
% cat x.go
package main

import "fmt"

func main() {
	for i := 0; i < 5; i++ {
		fmt.Println(i)
		i += 1
	}
}
% gotip run x.go
0
2
4
% GOEXPERIMENT=loopvar gotip run x.go
0
2
4
%

rsc · 2023-05-10T15:46:06Z

Regarding 3-clause loops, three points:

If we don't fix 3-clause for loops, the footgun remains.

If we leave 3-clause loops alone, then every time you change a loop from 3-clause to range or back, you have to stop and reason about the possible capture difference you might be introducing. When you are doing that kind of conversion, you should have to think about whether the iteration visits the same things and then be able to stop there.

Real-world evidence all points to this change fixing some 3-clause for loops and breaking none.

I freely admit that it is possible to write 3-clause for loops that behave differently in the new semantics, just as it is possible to write range loops that behave differently, such as this one by Tim King:

func sum(list []int) int {
	m := make(map[*int]int)
	for _, x := range list {
		m[&x] += x
	}
	for _, sum := range m {
		return sum
	}
	return 0
}

As Merovius put it so well, if you saw this in real life, the surprise would be not that the code behaves differently with the new loop semantics but that it was written this way at all.

Focusing on artificial examples misses the point, which is that in actual programs, when per-iteration vs per-loop does matter, 99% of the time users expect per-iteration. Here are three real examples of 3-clause loop mistakes that made it into a repo and had to be corrected (I found these by searching git histories for diff hunks consisting only of added x := x lines):

(Of course, searching git histories misses all the buggy loops that were debugged before being checked in.)

Having just gone through all the test failures inside Google caused by enabling the new semantics globally, I can tell you that 3-clause loops were a minority of root causes of test failures, but they did exist, and all of them clearly expected per-iteration semantics. The test failures happened because the test was incorrectly passing, like TestAllEvenBuggy above. In no case was the loop change breaking a good loop. That is, I saw no 3-clause for loops that correctly depended on the current per-loop semantics. Nor did I see any range loops, except the low-level one I mentioned that was prohibited from allocating.

That's some of the real-world evidence in favor of changing 3-clause loops. I have looked, and I found no evidence in favor of not changing them. If you want to make the case for not changing them, the way to do that would be to provide real-world examples of code that breaks with the new semantics. You might start by running your own tests against gotip with the loop variable change enabled (instructions here).

Evidence outweighs gut reaction.

I sympathize with the gut reaction from @go101 and @davidmdm here. I remember the first time I saw 3-clause for loops with per-iteration variable semantics, in an early talk about Dart around 2009 or so. My own gut reaction was that I was certain it was a terrible idea that couldn't possibly work, too strange to merit serious consideration. But as they say, certainty is just an emotion. The evidence we've built up over more than a decade of experience with Go seems to prove conclusively that Dart was right (as is JavaScript's for-let) and Go was wrong – I was wrong. Live and learn.

davidmdm · 2023-05-10T15:48:03Z

@zikaeroh @rsc

I am embarrassed to have not caught on that the iteration loop variable would inform the next loop variable.
I hope my confusion was at least instructive to others, and with that I am 100% behind this proposal.

🚀 💥 🚀 💯

mcandre · 2023-05-10T16:08:37Z

Number one frustration with how lambdas interact with loops in Go and Node.js. This is actually a safety concern.

On the one hand, declaring lambdas inside of a loop is usually a performance loss, compared to declaring the lambda before or after the loop. On the other hand, when you need to curry some specific iteration state, then it is annoying to have to use additional boilerplate there.

100% code coverage is not guaranteed to catch this glitch, by the way. This is a rather subtle quirk dating back to how C does its dirty work. If the developer has insufficient tests, then many downstream problems can occur.

i can think of no useful program that desires only the last iteration state to be curried into all the lambdas.

Let's adopt this fix ASAP.

cep21 · 2023-05-10T20:40:13Z

This is 100% a good idea and worth doing, but I worry we're down playing the idea "only packages in a module with a go.mod that says go 1.22 would get the new loop semantics". I was trying to find more about this in @rsc 's video and link (may have missed it there's a lot to catch up on).

Tools like renovatebot and dependabot are almost required for open source projects. Many configure these tools to auto update minor (and patch) version dependencies (of course after tests run) and even auto merge (there could be dozens of these packages to keep up to date). My mental model is to never expect the line

-go 1.21
+go 1.22

to be something that could break my existing code since it is a minor change. These tools by default auto update this line.

What is the correct mental model for seeing the above diff? Is there an assumption beyond optimizations (sort ordering changing). Maybe the bar is "beyond reasonable doubt" for behavior changes, while I thought the previous bar was much more deterministic. Maybe it's off topic or overblown, but I don't know what I should think when I review renovatebot diffs in the future that do patch changes to the "go" line in the mod file.

Maybe there should be an official-ish recommendation of Only change go.mod go versions if you explicitly need a new feature or library.

rsc · 2023-05-10T21:05:42Z

What is the correct mental model for seeing the above diff?

The correct mental model should be that you are running a newer version of a dependency and you should test that your software continues to work as expected with that newer dependency. This is true for any dependency, not just the Go line. If you are using foo/bar v1.2.3 and dependabot bumps it to v1.2.4 or v1.3.0, the fact that it's still v1 (not a different major version) means the author intends that there are no breaking changes. But maybe the author fixed a bug, and your code accidentally depends on the buggy behavior. Or maybe the author introduced a bug by accident, and your program is the one that discovers the bug. Intending not to break you is different from not breaking you.

The same is true for the go line. If you see the go version change, then the Go team intends that there are no breaking changes for you. But maybe we fixed a bug you depend on, or maybe we introduced a bug by accident. In literally every instance of a loopvar-induced failure I've seen, "fixed a bug you depend on" is an accurate description. We absolutely understand that intending not to break you is different from not breaking you. When we do break you, we want you to have the right tools to make that as painless as possible. That's why the loopvar change is keyed on the go line, and it's also why other compatible-but-possibly-breaking changes will have GODEBUGs with defaults keyed on the go line, as of Go 1.21 (see this doc).

Circling back to dependabot, if dependabot proposes a change, any change at all, it would be a mistake to accept it solely because "it's only a version number increment". The only way dependabot is safe is if (1) the change passes all your tests, and (2) you have very good test coverage. If that's already true for your module dependencies, then you'll have no problem with go line bumps either.

rsc · 2023-05-10T21:55:58Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

go101 · 2023-05-11T16:42:29Z

Let me re-state my opinion:

I support making the change for for-range loops. What I oppose to is making the change for traditional for;; loops.

The current semantic of for-range loops is counter-intuitive, the change will make it intuitive.

The current semantic of traditional for;; loops is intuitive, the change will make it counter-intuitive.

It is natural for me to think the iteration variables k and vin the following for-range loop are declared in each loop step.

func foo(f func(*K, *V)) {
	for k, v := range c {
		// If we think the iteration variables k and v are declared in
		// each loop step, then the following k and v are exactly
		// the iteration variables k and v.
		f(&k, &v)
	}
}

On the other hand, it is natural for me to think the iteration variables k and v in the following traditional for;; loop are only declared once during the whole loop execution, because the statement k, v := next() is only executed once during the whole loop execution. And I think this is also most people's intuitions (proof).

func bar(f func(*K, *V)) {
	for k, v := next(); ; k, v = next() {
		// By the proposed change, the following k and v are implicitly
		// declared and they are not the above iteration variables k and v.
		// This is too counter-intuitive.
		f(&k, &v)
	}
}

That is all. Obeying intuition is so important for a language. A thousand arguments to break the rule are still pale. Adding magic implicitness is a big no-no (esp. for Go, a language promoting explicitness). Doing this will break many people's expectations.

In fact, after I read all the arguments in related threads to break the rule, I found none of them are ~~reasonable~~convincing.

If we don't fix 3-clause for loops, the footgun remains.

Sorry, if the change is made on traditional for;; loops, it might remove a footgun (so-called, which is actually caused by unprofession and carelessness), but it will also create new footguns.

Currently, I have found two new footguns caused by this:

The first:

// The loop prints true now. The semantic change will make it print false.
	for i, p := 0, (*int)(nil); p == nil; print(p == &i) {
		p = &i
	}

The second:

// Now the value "a" will not duplicated at each loop step.
// The semantic change will make "a" be duplicated (twice)
// at each loop step.
func foo(constFunc func(*[100000]int)) {
	for a, i := [100000]int{}, 0; i < len(a); i++ {
		constFunc(&a)
	}
}

That is two surprises for many people. Several people said the two cases are artificial or "intentionally confusing programs". I'm speechless on such non-serious attitudes. I can't believe that the bar to keep backward-compatibility is down to such a low level. The two code pieces are just two demonstrations. They are valid code, it is hard to say no code like these are used in practice.

The evidence we've built up over more than a decade of experience with Go seems to prove conclusively that Dart was right (as is JavaScript's for-let) and Go was wrong

It is unfair to compare Go with JS (I'm not familar with Dart, so I'm not sure if this is also true for Dart). Go is quite different from JS, Go supports pointers and large-size types/values, which makes the use cases using Go loops much more diverse. This is evident in the above two new footgun examples.

If we leave 3-clause loops alone, then every time you change a loop from 3-clause to range or back, you have to stop and reason about the possible capture difference you might be introducing. When you are doing that kind of conversion, you should have to think about whether the iteration visits the same things and then be able to stop there.

I don't think this is not a problem for a qualified Go programmer. It is unnecessary to make the pointless consistency. The two kinds of loops are different in nature and a qualified Go programmer should realize the result. In fact, making them behave differently is a great feature so that Go programmers have more choices for different scenarios.

Even if the argument is valid, then similar situations will happen if the proposed change is applied to traditional for;; loops. For example, in practice, sometimes, for some reason, a programmer rewrites the first loop to the second one. She/he expects that the re-writing will not change the execution of loop, but her/his expectation might be broken. If the execution change is a performance degradation, it might be not detected in time, depending on the degree of the performance degradation.

	// 1
	for v := f(); g(); h(&v) {
		... // no continue here
	}

	// 2
	for v := f(); g();  {
		... // no continue here
		h(&v)
	}

That's some of the real-world evidence in favor of changing 3-clause loops. I have looked, and I found no evidence in favor of not changing them.

Firstly, I think maybe what you have looked are still not enough. Second, I also don't see universal and strong desires in favor of changing them.

And we need to make it clear whether or not the causes of the following two bugs are the same.

var ids []*int
for i := 0; i < 10; i++ {
	ids = append(ids, &i)
}

and

var ids []*int
var i int
for i = 0; i < 10; i++ {
	ids = append(ids, &i)
}

Finally, in my opinion, mixing the changes to the two kinds of loops is not a good idea. I suggest to split this proposal into two, one for for-range loops and the other for traditional for;; loops.

I have noticed that this proposal has been added to the active column and will be reviewed. I'm curious that why do this in such a hurry. Personally, I think the current analysis work is still not sufficient. Shouldn't it be done after the experimental phase is almost over?

Merovius · 2023-05-11T16:58:22Z

@go101 I think most participants in the conversation understand your opinion. That doesn't mean they can't disagree with it.

And I think this is also most people's intuitions (proof).

To me, the fact that your arguments are based on extremely contrived code to show anything about "intuition" makes them very weak. I still, to this day, have not formed an opinion on what any of your examples "should" do, because I think the only correct response to encountering them in the wild isn't to try to puzzle through and explain their behavior, but to remove them.

They are bad code. "This change would make extremely bad and obtuse code counter-intuitive" is a non-starter of an argument to me. That code is counter-intuitive, no matter what it does.

That is two surprises for many people. Several people said the two cases are artificial or "intentionally confusing programs". I'm speechless on such non-serious attitudes. I can't believe that the bar to keep backward-compatibility is down to such a low level.

We are accepting breaking backwards compatibility with this proposal. Even with the part you explicitly endorse. So, clearly this is not the issue.

You are making an argument based on supposed "counter-intuitive semantics". And for that, yes, the fact that your examples are contrived is extremely salient.

Merovius · 2023-05-11T17:19:40Z

I have noticed that this proposal has been added to the active column and will be reviewed. I'm curious that why do this in such a hurry. Personally, I think the current analysis work is still not sufficient. Shouldn't it be done after the experimental phase is almost over?

This proposal had a discussion and a separate issue discussing adding the experiment flag. Both of which you are perfectly aware of, as you've participated in both. So, pretending that there hasn't been extensive conversation about this so far is simply disingenuous.

Meanwhile, being added to the active column simply means that this proposal is in the active discussion phase. Nothing more. That is, it's not a "likely accept", or anything. There are proposals which spent many months in the active phase. So, again, pretending that there is any "hurry" implied by adding it to the list of active proposals is wrong.

That being said, the response to both the discussion and this proposal have been overwhelmingly positive. Note that this issue accumulated 230 👍 in two days, and only 6 👎. The emoji-votes on the original discussion are even more unanimously and overwhelmingly positive. There also have been few proposals in the past so thoroughly evidenced and backed by actual data. So, yes, just because you don't like it, doesn't mean we shouldn't do this soon, based on the long discussion phase and extremely positive feedback we had so far.

rsc · 2023-05-11T21:37:56Z

@go101, your follow-up post repeatedly uses words like “intuitive” and “natural” and “reasonable” and “expectations” as if they are universal absolutes. To me, the new semantics are intuitive and natural and reasonable and match my expectations better than the old semantics. We are going to have to agree to disagree about how the new semantics make us feel.

I want to be clear: instead of focusing on feelings, we are going to make this decision based on evidence.

The evidence we have presented in favor of making the change is our experience running the new semantics in a code base with many million lines of Go code. The evidence you have presented is a few hypothetical programs that look nothing like any Go program I have ever seen in actual practice. The evidence in favor of the change is – objectively and scientifically – far stronger.

About your hypotheticals, you wrote:

The two code pieces are just two demonstrations. They are valid code, it is hard to say no code like these are used in practice.

On the contrary, it is easy for me to say that no code like those is used in practice, or at least that it is exceedingly rare. Because, again, we are already running many millions of lines of Go code using the new semantics and have encountered zero instances of code that broke due to the new semantics of 3-clause for loops. The single instance where correct code arguably broke was a range loop. The bar is not “never anywhere”. The bar is “exceedingly rare”, and the evidence we have gathered absolutely establishes that.

I have invited you to look for real-world evidence supporting your position and explained how to go about doing that. If you would like to gather real evidence and share it with us, please do. But otherwise, please stop posting the same arguments over and over. Thank you.

mbyio · 2023-05-11T22:38:25Z

I think making this change is definitely the right thing to do. It always takes me out of my flow when I need to think about variable scoping in loops, and I agree that it is almost always a bug to rely on the old behavior.

I'm only concerned about the upgrade process. I'm specifically looking at this section of the proposal https://go.googlesource.com/proposal/+/master/design/60078-loopvar.md#mechanical-rewriting-to-preserve-old-semantics-is-possible-but-mostly-unnecessary-churn

I often work with code that all lives in a single Go module and lacks trustworthy tests. The code is still worked on, on a weekly to monthly basis. We might dive in, make a small edit somewhere, maybe add a new if statement, upgrade a module, etc. No major changes, just maintenance work. I worry that there will be a strong tendency to just leave code like that on go 1.21 (or whatever version it is) forever.

My understanding is, with the static compiler tooling being proposed, engineers will have to do some manual work to check any loops that were flagged. Then they'll have to test the program to make sure it works as expected after their changes. I think even the suggestion that there is some indeterminate amount of manual work required will cause some people to never upgrade. For some other people, especially those who are less confident in their understanding of the language and this change, they may have trouble using the tools you're proposing.

Maybe I am still scarred by the Python 2 -> Python 3 upgrade process. One of the biggest annoyances was having to come back to some codebase still on Python 2, where the language was very different from Python 3, and make small edits like I described above. It made the whole process more error prone because of the context switching and unclear set of supported language features. But I didn't really have time to go through and do the actual upgrade, because I was supposed to just make a small edit. So I had to tough it out.

Obviously, this change is tiny compared to Python 2 -> Python 3. But I'm thinking of the future, go 1.50, when maybe there are a bunch of these tiny language fixes that add up over time. There might still be old projects stuck on 1.21 years from now because of this loop change.

So, I would like there to be some opt-in tooling that automatically edits code to preserve the old behavior in suspicious loops. That way, in cases where they otherwise might not be able to upgrade, someone can just run this upgrade tool, and engineers who fully understand the change and have some time to spare can go back and make gradual edits to fix old code over weeks/months/years, and we won't have to deal with a divide between go 1.21 and go 1.22 codebases. I know the community could always write a tool like that, but once you have to rely on a community tool, that adds another obstacle to the upgrade since you now have to research to find out if the tool is trustworthy or not.

Sorry for the long message, I hope that makes sense. Thanks for all the work on Go.

ghost · 2023-05-12T02:11:14Z

I sympathize with the gut reaction from @go101 and @davidmdm here. I remember the first time I saw 3-clause for loops with per-iteration variable semantics, in an early talk about Dart around 2009 or so. My own gut reaction was that I was certain it was a terrible idea that couldn't possibly work, too strange to merit serious consideration. But as they say, certainty is just an emotion. The evidence we've built up over more than a decade of experience with Go seems to prove conclusively that Dart was right (as is JavaScript's for-let) and Go was wrong – I was wrong. Live and learn.

@rsc you seem to be mis-representing what JavaScript actually does. From the docs:

This also happens if you use a var statement as the initialization, because variables declared with var are only function-scoped, but not lexically scoped (i.e. they can't be scoped to the loop body).

https://developer.mozilla.org/docs/Web/JavaScript/Reference/Statements/for#lexical_declarations_in_the_initialization_block

In this context, var variables are function-scoped, even when declared in a loop clause. that is not what Go does. Current Go scopes to the loop, not the function, and "new" Go scopes to the iteration. Neither Go behavior matches JavaScript. As others have said and demonstrated with code examples, this change is essentially a "macro" that expands into some surprising code. Saving one line of explicit code that is currently needed, does not justify that added ambiguity introduced with this change I feel.

zikaeroh · 2023-05-12T02:36:49Z

@4cq2 The examples given in the link show that the scoping behavior of let matching what this Go proposal is implementing. Namely, it says:

for (let i = 0; i < 3; i++) {
  setTimeout(() => {
    console.log(i);
  }, 1000);
}

Prints 0, 1, 2. So, I don't find the JS comparison a misrepresentation at all. Additionally, the current Go behavior is exactly what the linked page describes for a let pulled out of the for loop, whose equivalent is also described in the Go proposal as the new way to write the code using the old semantics.

Sure, var doesn't behave that way, but var has scoping unlike any variable in Go and I think it's very accurate to say that nearly all developers avoid var as much as possible when writing JavaScript/TypeScript so it's not representative of "what JavaScript actually does" on its own.

gopherbot · 2023-08-11T15:01:07Z

Change https://go.dev/cl/518675 mentions this issue: [release-branch.go1.20] cmd/go: refuse to build Go 1.22 code

With #60078 accepted, we expect Go 1.22 will have different for loop semantics than Go 1.20 did. Once Go 1.22 is released, Go 1.20 will be unsupported, but add a check anyway, just to help catch some mistakes and usage of old Go toolchains beyond their end-of-support. Note that Go 1.20 can keep being used indefinitely with pre-Go 1.22 code. This change only makes it refuse to build code that says it needs Go 1.22 semantics, because Go 1.20 does not provide those. For #60078. Change-Id: I75118d6fbd0cc08a6bc309aca54c389a255ba7dc Reviewed-on: https://go-review.googlesource.com/c/go/+/518675 Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2023-08-11T17:38:07Z

Change https://go.dev/cl/518815 mentions this issue: [release-branch.go1.19] cmd/go: refuse to build Go 1.22 code

With #60078 accepted, we expect Go 1.22 will have different for loop semantics than Go 1.19 did. Go 1.19 is already unsupported, but add a check anyway, just to help catch some mistakes and usage of old Go toolchains beyond their end-of-support. Note that Go 1.19 can keep being used indefinitely with pre-Go 1.22 code. This change only makes it refuse to build code that says it needs Go 1.22 semantics, because Go 1.19 does not provide those. Cherry-pick of the change from the Go 1.20 branch. For #60078. Change-Id: I75118d6fbd0cc08a6bc309aca54c389a255ba7dc Reviewed-on: https://go-review.googlesource.com/c/go/+/518675 Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-on: https://go-review.googlesource.com/c/go/+/518815 Auto-Submit: Russ Cox <rsc@golang.org> TryBot-Bypass: Russ Cox <rsc@golang.org>

iBug · 2023-08-12T10:06:31Z

I must say @go101's argument makes total sense. I have the following information to defend for 3-clause for loops (hereinafter "for;; loops").

Similarity in other languages is an important source for intuition. To the best of my knowledge, the only other languages that share similarity with this proposal (if applied to for;; loops) is JavaScript ES6 for-loops-with-let-declaration. Most compared languages, like C++ and Rust, both have explicit distinction between values and references, which makes capturing loopvars unambiguous. Go, however, has no concept of "reference", which is commonly replaced by pointers. This is concrete, objective evidence on "intuition".

Additional points about JavaScript:
- It has no concept of either "pointers" or "references", making it an inappropriate comparison with Go.
- It maintains the behavior of old code: As long as you don't add let to your loops, your code behaves consistently, be it in ES3 or ES2023.
On the contrary, other languages with for-range or for-each loop all behave similarly, in that the loopvar in different iterations refer to distinct objects, excluding languages where values and references are explicit concepts (like C++).
Using for-range loops in all examples in the initial comment, mentioning nothing about for;; loops, and then adding that for;; loops are affected as well, is a fundamentally flawed process to push a proposal forward. That the original proposal gained popularity as it was written does not mean changing for;; loops would be equally received. It is therefore important to split this proposal into two, the other regarding for;; loops, and gather votes and opinions separately.

domdom82 · 2023-08-14T09:56:21Z

It's funny. I got totally caught by surprise with this issue. Thankfully, I never actually ran into it but my assumption has always been that loop variables are closure-safe and retain their value on each iteration. IMHO if your program breaks by this change, it was buggy to begin with. So by all means, do go ahead @rsc

gopherbot · 2023-10-04T22:20:33Z

Change https://go.dev/cl/532580 mentions this issue: go/ssa: new range var semantics

entonio · 2023-10-13T18:22:33Z

As someone whose codebase contains lots and lots of for-range loops that cannot all be individually unit-tested, my concern isn't with 'contrived' code. Contrived examples are there so that effects can de demonstrated; in real life, it's usually abstraction and the escaping of variables into downstream code that produce the same problems, and are much harder to detect. And they're types of domain interaction that aren't as likely to exist within the base sdk, nor within the scope of a very focused codebase such as Encrypt's. The risk of breakage doesn't seem especially high, it's the risk of unnoticed breakage. And it's not just a matter of unit tests, just think of all the sdk unit tests that you say were shown to be wrong when adopting the change.

At least as for-range loops go, it seems it would have been trivial to just keep the old code running as is by tying the new behavior to a new keyword, e.g. for k, v := loop m { ... }. Or even loop k, v := range m { ... }, loop i := 0; ; i++ { ... }. Or make the new keyword use the old behaviour, that way wary folk could replace all their for instances and be calm. Or create a new assignment operator for this case, since this seems to be about the semantics of assignment.

Otoh, I fully understand not wanting to multiply operators or keywords, lest we end up in C++ turmoil.

One thing I think could help would be -as someone suggested- a migration tool that would not only show which loops would be affected by the change, but also provide a rewriting that would keep the old behaviour (by declaring vars outside the for, is that it? how does that work with for-range?). For me, that would solve the problem entirely, or almost. The last time I used the flag, I don't recall seeing anything towards that.

zephyrtronium · 2023-10-13T18:42:19Z

@entonio Adding a new keyword like loop is never an option in Go because any code using loop as a variable name would cease to compile, violating the Go 1 compatibility promise. Beyond that, the design document linked from the initial post explains why changing loop syntax or creating a migration tool are poor choices.

thediveo · 2023-10-13T18:46:38Z

@entonio Last time I looked at it, the k8s code base consisted of huge quantities of for range loops. They aren't individually unit-tested either. Yet IIRC so far running the exisiting tests revealed no detectable problem. If you have followed the discussion then you might have noticed mentioning of code scans for misuse of for range variables. The following discussion should also have sufficiently answered your own argument: only contrived examples were presented, but no real code. I really don't get your writing about abstraction and skds here. To me, this doesn't bring any new reasoning here. In your logic, even the slightest change in code generation anywhere else must be dangerous, with all those abstractions and sdks?

timothy-king · 2023-10-13T23:26:54Z

One thing I think could help would be -as someone suggested- a migration tool that would not only show which loops would be affected by the change

@entonio You may wish to read the section Transitional Support Tooling in the design doc. This describes how to get the data one needs from the compiler to create such a tool. It would be a decent amount of work, but that is the starting point.

dgoodine · 2023-11-19T20:53:26Z

Not weighing in one way or another on this specific proposal, but I have an observation that may be useful.

IIRC, most implementations of closures in Lisp implementations treated values of outer lexically scoped variables and references to them differently.

That is to say, referencing a loop variable by value stores its current value in the closure environment when the closure is created. Referencing a loop variable by reference (which is what the current implementation seems to be doing in all cases) is storing a pointer to the variable in the closure environment.

Clearly, doing the latter in a loop would make this a race-condition factory, but in cases where the closure only needs the values of an outer lexical context, it would avoid this issue completely.

I don't know anything about how they are implemented in Go, nor whether such an approach is relevant or feasible. Perhaps if someone here has some insight or knows someone on the go team with more information, it might be worth considering figuring out if that might be a better and more general solution. The compiler should be able to handle the distinction between the two in the generated code, without wondering if it would break things.

(Edit: It also is completely insensitive to the looping context (e.g. for/range).)

seankhliao · 2024-02-11T13:33:56Z

Now that Go 1.22 is released, I think we can close this as done.

rsc added the Proposal label May 9, 2023

gopherbot added this to the Proposal milestone May 9, 2023

rsc mentioned this issue May 9, 2023

proposal: spec: redefine range loop variables in each iteration #20733

Closed

Deleplace mentioned this issue May 11, 2023

Bug: loop variable capture, causes data races and probably wrong behavior syke99/gworker#2

Closed

go101 mentioned this issue Aug 10, 2023

cmd/compile: GOEXPERIMENT=loopvar bug #61906

Closed

AlexanderYastrebov mentioned this issue Aug 18, 2023

gosec: fix G601 rule zalando/skipper#2522

Merged

griesemer mentioned this issue Sep 13, 2023

go/types: Export Info.FileVersions for access to file-specific version information #62605

Closed

thehowl mentioned this issue Sep 16, 2023

for loops maintain the same block on iteration, which is referenced in any closures generated within gnolang/gno#1135

Open

qiulaidongfeng mentioned this issue Oct 15, 2023

proposal: change the definition of zero value, and the semantics of reflect.Value.IsZero are used to compare any type of zero value #63552

Closed

dolmen mentioned this issue Oct 28, 2023

proposal: sync: add Go method to WaitGroup to launch a tracked goroutine #63796

Open

griesemer mentioned this issue Nov 2, 2023

cmd/vet: don't report "loop variable captured" error for versions >= go1.22 #63891

Closed

pmahindrakar-oss mentioned this issue Nov 22, 2023

Fix the loop variable scheduler issue flyteorg/flyte#4468

Merged

3 tasks

cheesesashimi mentioned this issue Nov 28, 2023

MCO-795: MCO-813: MCO-473: Implement Upgrade-Monitor, FeatureGate, and MachineConfigNode types openshift/machine-config-operator#4012

Merged

tynes mentioned this issue Dec 28, 2023

refactor: move op-preimage tests to separate package ethereum-optimism/optimism#8776

Closed

rynowak mentioned this issue Jan 26, 2024

Support routes.*.destination of gateway for App Graph radius-project/radius#7079

Merged

enrichman mentioned this issue Jan 30, 2024

Add status field for global roles rancher/rancher#44246

Merged

seankhliao closed this as completed Feb 11, 2024

dmitshur modified the milestones: Backlog, Go1.22 Feb 11, 2024

dmitris mentioned this issue Feb 29, 2024

cmd/vet: warn about capturing loop iterator variables #16520

Closed

yanskun mentioned this issue Mar 18, 2024

Upgrade to Go 1.22 cli/cli#8836

Merged

spec: less error-prone loop variable scoping #60078

spec: less error-prone loop variable scoping #60078

Comments

rsc commented May 9, 2023 • edited

Tooling Demonstration

seebs commented May 9, 2023

ulikunitz commented May 10, 2023

go101 commented May 10, 2023 • edited

zikaeroh commented May 10, 2023

go101 commented May 10, 2023

Merovius commented May 10, 2023 • edited

go101 commented May 10, 2023 • edited

davidmdm commented May 10, 2023 • edited

go101 commented May 10, 2023 • edited

Merovius commented May 10, 2023

davidmdm commented May 10, 2023

zikaeroh commented May 10, 2023 • edited

rsc commented May 10, 2023

rsc commented May 10, 2023 • edited

davidmdm commented May 10, 2023

mcandre commented May 10, 2023 • edited

cep21 commented May 10, 2023

rsc commented May 10, 2023

rsc commented May 10, 2023

go101 commented May 11, 2023 • edited

Merovius commented May 11, 2023 • edited

Merovius commented May 11, 2023 • edited

rsc commented May 11, 2023

mbyio commented May 11, 2023

ghost commented May 12, 2023 • edited by ghost

zikaeroh commented May 12, 2023 • edited

gopherbot commented Aug 11, 2023

gopherbot commented Aug 11, 2023

iBug commented Aug 12, 2023

domdom82 commented Aug 14, 2023

gopherbot commented Oct 4, 2023

entonio commented Oct 13, 2023

zephyrtronium commented Oct 13, 2023

thediveo commented Oct 13, 2023 • edited

timothy-king commented Oct 13, 2023

dgoodine commented Nov 19, 2023 • edited

seankhliao commented Feb 11, 2024

rsc commented May 9, 2023 •

edited

go101 commented May 10, 2023 •

edited

Merovius commented May 10, 2023 •

edited

go101 commented May 10, 2023 •

edited

davidmdm commented May 10, 2023 •

edited

go101 commented May 10, 2023 •

edited

zikaeroh commented May 10, 2023 •

edited

rsc commented May 10, 2023 •

edited

mcandre commented May 10, 2023 •

edited

go101 commented May 11, 2023 •

edited

Merovius commented May 11, 2023 •

edited

Merovius commented May 11, 2023 •

edited

ghost commented May 12, 2023 •

edited by ghost

zikaeroh commented May 12, 2023 •

edited

thediveo commented Oct 13, 2023 •

edited

dgoodine commented Nov 19, 2023 •

edited