testing/synctest: bubble not terminating

[alecthomas]

Go version

go version go1.26-devel_a4d9977 Wed Jul 30 14:00:16 2025 -0700 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/alec/dev/zero/.hermit/go/bin'
GOCACHE='/Users/alec/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/alec/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='synctest'
GOFIPS140='off'
GOFLAGS='--tags=mysql,postgres,sqlite'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/xk/0xh6fbnx3wx2_rkxr4w9vx140000gn/T/go-build3867150125=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/alec/dev/zero/go.mod'
GOMODCACHE='/Users/alec/go/pkg/mod'
GONOPROXY='*.sqcorp.co,github.com/squareup'
GONOSUMDB='*.sqcorp.co,github.com/squareup'
GOOS='darwin'
GOPATH='/Users/alec/go'
GOPRIVATE='*.sqcorp.co,github.com/squareup'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/alec/Library/Caches/hermit/pkg/go@tip'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/Users/alec/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/Users/alec/Library/Caches/hermit/pkg/go@tip/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.26-devel_a4d9977 Wed Jul 30 14:00:16 2025 -0700'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I've written a test using testing/synctest that I believe should exit, but doesn't. The code to reproduce this is in this draft branch, but the stack dump is also attached.

Cloning that branch down and running GOEXPERIMENT=synctest go test -v ./providers/cron should reproduce the issue.

This occurs with latest stable Go and Go tip from a4d9977.

stack.txt

What did you see happen?

I believe it should exit because when I SIGQUIT the process, the only goroutines running are from the stdlib. It's also entirely possible I'm doing something spectacularly stupid, but if I am I can't figure it out.

As an aside, I think it would be very useful if SIGQUITting inside a bubble marked the goroutines preventing the bubble from terminating.

What did you expect to see?

I expect to either have synctest.Run() terminate, or for a goroutine to be in the stack trace that is obviously causing the issue.

[gabyhelp]

Related Issues

• testing: Example test with runtime.Goexit hangs #41084 (closed)
• testing/synctest: bug with httptest #71345 (closed)
• testing/synctest: Repeated sync.WaitGroup.Add appears flaky under synctest #74386 (closed)
• testing: stack unavailable when a parallel test times out #18042 (closed)
• testing: inconsistent behaviors between running tests directly and after compiling the code first #59879 (closed)
• cmd/go: test gets hung #27720 (closed)
• cmd/go: go test sometimes hangs if interrupted #60220 (closed)
• cmd/go: trim build cache incrementally #38939
• testing: "panic: Log in goroutine after Test..." is unreliable due to lack of synchronization on t.done #67701

Related Code Changes

• runtime: print stack traces for bubbled goroutines on synctest deadlock

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mrkfrmn]

CC @golang/runtime, @neild from owners.

[neild]

I'm on vacation for the next week, so I probably won't have a chance to look at this in detail until I'm back.

The stack traces don't show any goroutines in a synctest bubble, so I believe the bubble has exited at the time the SIGQUIT happened. I'm not sure what's going on from a cursory look, but goroutine 10 in the attached stack traces appears to be stuck trying to lock a context.Context mutex and there aren't any other goroutines that I see that would be holding that lock. That's pretty suspicious.

As an aside, I think it would be very useful if SIGQUITting inside a bubble marked the goroutines preventing the bubble from terminating.

On Go tip, the stack traces should include which bubble (if any) each goroutine is part of, and bubbled goroutines are marked with an indication of whether they're durably blocking or not.

[alecthomas]

On Go tip, the stack traces should include which bubble (if any) each goroutine is part of, and bubbled goroutines are marked with an indication of whether they're durably blocking or not.

Aha! That's fantastic news, thanks.

[alecthomas]

The stack traces don't show any goroutines in a synctest bubble, so I believe the bubble has exited at the time the SIGQUIT happened. I'm not sure what's going on from a cursory look, but goroutine 10 in the attached stack traces appears to be stuck trying to lock a context.Context mutex and there aren't any other goroutines that I see that would be holding that lock. That's pretty suspicious.

Based on this observation, I added a log line after the bubble in the main test and saw the output. So it looks like the bubble is terminating, but the test itself is not.

[neild]

The problem here is quite subtle. The good news is that you're much less likely to run into it with the final version of the synctest API in Go 1.25.

We can reduce the reproduction case to:

func Test(t *testing.T) {
        synctest.Run(func() {
                _, cancel := context.WithCancel(t.Context())
                defer cancel()
        })
}

Contexts have a "done" channel, which is created lazily on demand.

The context.WithCancel call creates the done channel for t.Context. This channel is created inside the synctest bubble. The test then completes, and as part of the test cleanup the t.Context context is canceled, which closes the done channel.

This results in a panic, since it a channel created inside a bubble is being closed from outside the bubble. (The bubble has exited by this point, but that's not relevant.)

The panic gets recovered by the testing package, which (I think, I haven't traced this) tries to cancel the context again, which hangs because the first panic left the context mutex locked. This is all very subtle and complicated and I wouldn't expect anyone to figure out what's gone wrong easily.

The good news is that with the revised synctest API in Go 1.25, you'd write this as the following, which will work as expected since the synctest bubble uses a context scoped to the bubble:

func Test(t *testing.T) {
        synctest.Test(t, func(t *testing.T) {
                _, cancel := context.WithCancel(t.Context())
                defer cancel()
        })
}

The bad news is that it's still possible to get into a confusing situation by lazily creating a context channel within a synctest bubble. If nothing else, we should try to avoid the confusing deadlock observed here.

[prattmic]

This results in a panic, since it a channel created inside a bubble is being closed from outside the bubble. (The bubble has exited by this point, but that's not relevant.)

The panic gets recovered by the testing package, which (I think, I haven't traced this) tries to cancel the context again, which hangs because the first panic left the context mutex locked.

Context cancellation leaving the context mutex locked after a recovered panic sounds like a bug(ish) in context (context.(*cancelCtx).cancel specifically).

Prior to synctest, I think a panic in close(d) would be impossible since context has complete ownership over the channel. It seems like we should do something about this in the context package, though I'm not sure what:

• Recover the panic and convert it to fatal throw?
• Pretend the context is cancelled an set the error?

[neild]

I can think of two things we could do. Not sure which is better.

We can change the context package to defer mutex unlocks so that any panics in cross-bubble channel operations will leave the context mutex unlocked. This is a fairly small change (there are only a few places where the mutex unlock isn't already deferred). It's a bit unfortunate to be making changes to previously-working code to work around new behavior introduced by synctest, though.

Or we can change the runtime's panic for cross-bubble channel operations into a fatal throw. We already use fatal throws for some other cross-bubble operations; perhaps we should just consistently use them in all cases. Then the offending context operation just immediately terminates the program.

Laying it out like that, perhaps the fatal-throw option is the better one? This panic can only happen in tests, and it's a surprising new way for channel operations to fail, so ensuring that it's always visible and unrecoverable seems reasonable.

[alecthomas]

That's good news that my use case will work fine in 1.25, thanks for the update

[prattmic]

Fatal throw does seem better at first glance. I have a hard time imagining what a test might want to do after recovering from this panic.

That said, testing itself recovers panic to add some special handling like running cleanups. I'm not sure if that is important/desirable in this case.

[gopherbot]

Change https://go.dev/cl/696195 mentions this issue: runtime: make all synctest bubble violations fatal panics