You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memory leak on executables embedded with 3.13 #113055
Since 67807cf, _testembed.exe (x64 Release) increases the memory usage in the cycle of Py_Initialize and Py_Finalize.
I monitored the Task Manager (taskmgr.exe) on Windows10/11, with a change against Programs/_testembed.c:
-#defineINIT_LOOPS 4
+#defineINIT_LOOPS 200
Commands:
_testembed.exe test_repeated_simple_init or
python -m test test_embed -m test_simple_initialization_api (no log)
With RADIX TREE (as-is)
Loop
67807cf
main
3.11.7
100
67MB
210MB
4MB
200
135MB
406MB
4MB
No RADIX TREE (as-is)
Loop
67807cf
main
3.11.7
100
10MB
168MB
4MB
200
16MB
336MB
4MB
Recent obmalloc.c looks ready for finalization. Just adding my rough (invalid?) experiment made the leaks even. So, I hope that they share the same issue. Otherwise, ea2c001 or 15d4c9f has another leak. #98359 (comment)
That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free.
changed the title [-]Memory leak on embedded Windows executable with 3.12 and later[/-][+]Memory leak on Windows executable embedded with 3.12 and later[/+]on Dec 15, 2023
I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to 67807cfc it didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.
A possible way to fix this: when free_interpreter() gets called, walk through obmalloc arenas and free them. The pymalloc_print_stats() function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.
If you free the arenas and there is allocated blocks inside of them, you can't call pymalloc_free() on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters.
Building gh-113218 with the address sanitizer and then running ./Programs _testembed test_repeated_simple_init shows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this.
changed the title [-]Memory leak on Windows executable embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.12 and later[/+]on Dec 16, 2023
Your test_repeated_simple_init case leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:
==1275635== 502,840 bytes in 9,800 blocks are definitely lost in loss record 421 of 422
==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635== by 0x2FBAC3: unicode_decode_utf8 (unicodeobject.c:4701)
==1275635== by 0x2FC017: PyUnicode_DecodeUTF8Stateful (unicodeobject.c:4834)
==1275635== by 0x2F0E38: PyUnicode_FromString (unicodeobject.c:1893)
==1275635== by 0x3135D9: PyUnicode_InternFromString (unicodeobject.c:14943)
==1275635== by 0x289EBA: PyObject_SetAttrString (object.c:1062)
==1275635== by 0x28504F: _add_methods_to_object (moduleobject.c:174)
==1275635== by 0x2857D1: PyModule_FromDefAndSpec2 (moduleobject.c:384)
==1275635== by 0x3FB5C6: create_builtin (import.c:1396)
==1275635== by 0x4002F6: _imp_create_builtin (import.c:3400)
==1275635== by 0x284483: cfunction_vectorcall_O (methodobject.c:512)
==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635== by 0x21B94C: _PyObject_Call (call.c:348)
==1275635== by 0x21BA23: PyObject_Call (call.c:373)
==1275635== by 0x3856F4: _PyEval_EvalFrameDefault (generated_cases.c.h:1250)
==1275635== by 0x37E2B1: _PyEval_EvalFrame (pycore_ceval.h:115)
==1275635== by 0x3AA774: _PyEval_Vector (ceval.c:1771)
And
==1275635== 339,080 bytes in 6,920 blocks are definitely lost in loss record 420 of 422
==1275635== at 0x48407B4: malloc (vg_replace_malloc.c:381)
==1275635== by 0x2A0F00: _PyMem_RawMalloc (obmalloc.c:56)
==1275635== by 0x2A2568: PyObject_Malloc (obmalloc.c:917)
==1275635== by 0x2EF536: PyUnicode_New (unicodeobject.c:1238)
==1275635== by 0x2F11FE: _PyUnicode_FromUCS1 (unicodeobject.c:2025)
==1275635== by 0x2F16F9: PyUnicode_FromKindAndData (unicodeobject.c:2096)
==1275635== by 0x414B4E: r_object (marshal.c:1157)
==1275635== by 0x414DC0: r_object (marshal.c:1222)
==1275635== by 0x415AC6: r_object (marshal.c:1399)
==1275635== by 0x414DC0: r_object (marshal.c:1222)
==1275635== by 0x415A8A: r_object (marshal.c:1393)
==1275635== by 0x416028: read_object (marshal.c:1524)
==1275635== by 0x416339: PyMarshal_ReadObjectFromString (marshal.c:1641)
==1275635== by 0x3FCDF0: unmarshal_frozen_code (import.c:2062)
==1275635== by 0x4008B1: _imp_get_frozen_object_impl (import.c:3575)
==1275635== by 0x3F8E6B: _imp_get_frozen_object (import.c.h:282)
==1275635== by 0x284098: cfunction_vectorcall_FASTCALL (methodobject.c:425)
==1275635== by 0x21B736: _PyVectorcall_Call (call.c:273)
==1275635== by 0x21B94C: _PyObject_Call (call.c:348)
==1275635== by 0x21BA23: PyObject_Call (call.c:373)
My previous comment about disabling the INTERNED_STRINGS global hash table was incorrect. I must have built with Py_DEBUG enabled and so I erroneously thought it fixed the leak. You seem to be correct about _PyUnicode_ClearInterned(). There is this comment:
/* TODO:
* Currently, the runtime is not able to guarantee that it can exit without
* allocations that carry over to a future initialization of Python within
* the same process. i.e:
* ./python -X showrefcount -c 'import itertools'
* [237 refs, 237 blocks]
*
* Therefore, this should remain disabled for until there is a strict guarantee
* that no memory will be left after `Py_Finalize`.
If Py_DEBUG is enabled, then that function will free interned strings with the flag SSTATE_INTERNED_IMMORTAL. That comment was from Eddie's immortal refcnt change (gh-19474, ea2c001).
GH-118618 would fix the most serious memory leak in 3.12. For the last bugfix release of 3.12, @Yhg1s decided it was too large/disruptive to merge a backport and so that PR is closed. Assuming we are not changing our minds on merging that, I think the memory leak due to the obmalloc state will remain unfixed in 3.12. In theory there might be a less invasive and simpler fix for 3.12, I'd have to research that though.
It's getting confusing to keep track of these embedding bugs and what fixes have been backported. There are two classes of them: memory leaks and crashers. The crashers are the more serious ones. I believe we have fixes the known crashers (see GH-124865, GH-124536, GH-125314). For the memory leaks, the obmalloc state one is the worst. The leaks due to interned strings also depend on the mortal string change (49f6beb). Running the 3.12 head with ASAN enabled would show what's still leaking.
In the long term, i think it would be nice to have CPython running with no ASAN warnings. Then we have a build-bot to ensure we don't regress on that.
Activity
nascheme commentedon Dec 13, 2023
That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free.
neonene commentedon Dec 13, 2023
Only the Task Manager on Windows 10 and 11 for me.
The microsoft.com offers VMMap as a small alternative.
[-]Memory leak on embedded Windows executable with 3.12 and later[/-][+]Memory leak on Windows executable embedded with 3.12 and later[/+]nascheme commentedon Dec 16, 2023
@ericsnowcurrently
I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to
67807cfcit didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.A possible way to fix this: when
free_interpreter()gets called, walk through obmalloc arenas and free them. Thepymalloc_print_stats()function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.If you free the arenas and there is allocated blocks inside of them, you can't call
pymalloc_free()on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters.nascheme commentedon Dec 16, 2023
Building gh-113218 with the address sanitizer and then running
./Programs _testembed test_repeated_simple_initshows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this.[-]Memory leak on Windows executable embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.12 and later[/+]neonene commentedon Dec 17, 2023
I can also personally confirm that on Windows, including the 32-bit version of Python.
nascheme commentedon Dec 22, 2023
I think I've come up with a better fix. See gh-113412.
neonene commentedon Dec 23, 2023
ea2c001 has a memory leak to be fixed, which is found at gh-113412.
I've seen a few reports about likely suspects: #110411 (comment), #113190 (comment)
nascheme commentedon Dec 23, 2023
Your
test_repeated_simple_initcase leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_simple_initTrace log:
https://gist.github.com/nascheme/fac2dd566f09fe3d75c98134f18c6170
E.g.
And
nascheme commentedon Dec 23, 2023
The following change seems to greatly reduce the leaks. The
_Py_hashtable_set(INTERNED_STRINGS, s, s)call looks like a problem to me.neonene commentedon Dec 24, 2023
My memory leak seems to come from
_PyUnicode_ClearInterned()when thePy_DEBUGflag is not specified. I'm not sure of the root cause, though.cpython/Objects/unicodeobject.c
Lines 14952 to 14973 in 0c57454
neonene commentedon Dec 29, 2023
For the record, my memory gain reduced from 2M bytes to 200K bytes per cycle with gh-113412 (non-debug).
Regarding the crash after re-initialization due to the full cleanup by
_PyUnicode_ClearInterned():(leaks)
A dangling pointer in a non-isolable (small) extension could be worked around by making the interned string statically allocated, if any.
Original issue: #100911
nascheme commentedon Dec 29, 2023
My previous comment about disabling the
INTERNED_STRINGSglobal hash table was incorrect. I must have built with Py_DEBUG enabled and so I erroneously thought it fixed the leak. You seem to be correct about_PyUnicode_ClearInterned(). There is this comment:If Py_DEBUG is enabled, then that function will free interned strings with the flag
SSTATE_INTERNED_IMMORTAL. That comment was from Eddie's immortal refcnt change (gh-19474, ea2c001).3 remaining items
gh-113055: Use pointer for interp->obmalloc state (gh-113412)
pythongh-113055: Use pointer for interp->obmalloc state (pythongh-113412
pythongh-113055: Use pointer for interp->obmalloc state (pythongh-113412
[-]Memory leak on executables embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.13[/+]neonene commentedon Oct 15, 2024
I think this can be closed after 3203a74 is backported to 3.13. #113190 is opened for 3.12.
erlend-aasland commentedon Oct 15, 2024
What about #113218 then, @nascheme?
nascheme commentedon Oct 15, 2024
GH-118618 would fix the most serious memory leak in 3.12. For the last bugfix release of 3.12, @Yhg1s decided it was too large/disruptive to merge a backport and so that PR is closed. Assuming we are not changing our minds on merging that, I think the memory leak due to the obmalloc state will remain unfixed in 3.12. In theory there might be a less invasive and simpler fix for 3.12, I'd have to research that though.
It's getting confusing to keep track of these embedding bugs and what fixes have been backported. There are two classes of them: memory leaks and crashers. The crashers are the more serious ones. I believe we have fixes the known crashers (see GH-124865, GH-124536, GH-125314). For the memory leaks, the obmalloc state one is the worst. The leaks due to interned strings also depend on the mortal string change (49f6beb). Running the 3.12 head with ASAN enabled would show what's still leaking.
In the long term, i think it would be nice to have CPython running with no ASAN warnings. Then we have a build-bot to ensure we don't regress on that.
Yhg1s commentedon Oct 21, 2024
FYI, 3.12.7 was not the last bugfix release of 3.12. We have scheduled bugfix releases of 3.12 until April 2025.
vstinner commentedon Nov 27, 2024
I cannot reproduce this issue anymore on the main branch:
Output:
Since 3.12 branch is not going to be fixed, can we close the issue?