First, note that Go, itself, doesn’t always shrink its own memory space:
https://groups.google.com/forum/#!topic/Golang-Nuts/vfmd6zaRQVs
The heap is freed, you can check this using runtime.ReadMemStats(),
but the processes virtual address space does not shrink — ie, your
program will not return memory to the operating system. On Unix based
platforms we use a system call to tell the operating system that it
can reclaim unused parts of the heap, this facility is not available
on Windows platforms.
But you’re not on Windows, right?
Well, this thread is less definitive, but it says:
https://groups.google.com/forum/#!topic/golang-nuts/MC2hWpuT7Xc
As I understand, memory is returned to the OS about 5 minutes after is has been marked
as free by the GC. And the GC runs every two minutes top, if not
triggered by an increase in memory use. So worst-case would be 7
minutes to be freed.In this case, I think that the slice is not marked as freed, but in
use, so it would never be returned to the OS.
It’s possible you weren’t waiting long enough for the GC sweep followed by the OS return sweep, which could be up to 7 minutes after the final “big” pulse. You can explicitly force this with runtime.FreeOSMemory, but keep in mind that it won’t do anything unless the GC has been run.
(Edit: Note that you can force garbage collection with runtime.GC() though obviously you need to be careful how often you use it; you may be able to sync it with sudden downward spikes in connections).
As a slight aside, I can’t find an explicit source for this (other than the second thread I posted where someone mentions the same thing), but I recall it being mentioned several times that not all of the memory Go uses is “real” memory. If it’s allocated by the runtime but not actually in use by the program, the OS actually has use of the memory regardless of what top or MemStats says, so the amount of memory the program is “really” using is often very overreported.
Edit: As Kostix notex in the comments and supports JimB’s answer, this question was crossposted on Golang-nuts and we got a rather definitive answer from Dmitri Vyukov:
https://groups.google.com/forum/#!topic/golang-nuts/0WSOKnHGBZE/discussion
I don’t there is a solution today.
Most of the memory seems to be occupied by goroutine stacks, and we don’t release that memory to OS.
It will be somewhat better in the next release.
So what I outlines only applies to heap variables, memory on a Goroutine stack will never be released. How exactly this interacts with my last “not all shown allocated system memory is ‘real memory'” point remains to be seen.