runtime/metrics: add goroutine state counts, total goroutines created, total threads

[deft-code]

Update, Jun 7 2023: runtime/metrics now exists, but there are a few metrics in the draft CL here that aren't yet exposed. See #15490 (comment).

---

MemStats provides a way to monitor allocation and garbage collection.

We need a similar facility to monitor the Scheduler.

Briefly:

• Total goroutines create
• Current number of of goroutines
• Total number of goroutines scheduled
• Current number of goroutines scheduled
• Total thread starts
• Current number of threads.
• Metrics on the delay between a goroutine being ready and running on a proc.

[minux]

You can get all of these statistics by parsing the tracing output from runtime/trace.

[aclements]

@minux, while true, runtime/trace seems like a pretty high overhead way to collect what amounts to a fairly small amount of information. It's certainly low overhead for what it does, but what it does is much more than what's needed here. The metrics @deft-code wants are primarily intended for continuous monitoring (based on offline conversations), so it needs to be cheap.

[aclements]

Here are the notes on the desired metrics I had from our meeting a while ago:

Ring buffer of sampled duration between entering and exiting runnable state

• With some probability, when a goroutine enters runnable, tag it, and when it exits runnable, add the runnable duration to a ring
• Consumers can do what they want with these samples, including just averaging them, or building distributions.

Four global stats

• Current number of goroutines
• Total number of goroutines ever created
• Current number of runnable goroutines
• Total number of runnable-to-running transitions

Maybe current number of running goroutines

[bradfitz]

Assigning to @aclements to decide what we're willing to support long-term.

[adg]

Ping @aclements

[rsc]

Still up to @aclements.

[rsc]

Ping @aclements. Can you look at this during the release candidate quiet?

[aclements]

@deft-code, do the specific stats I suggested in #15490 (comment) address your needs?

[aclements]

Sorry, I'd lost track of the fact that there was a concrete proposal doc for this: https://github.com/deft-code/proposal/blob/master/design/15490-schedstats.md

@deft-code, could you mail a CL to add this to the go-proposal repository and, once submitted, edit your first post to link to it? Thanks.