-
-
Notifications
You must be signed in to change notification settings - Fork 32.4k
Description
Bug report
Every test class in test_httpservers starts a thread to run a server. CGIHTTPServerTestCase tests trigger htt.server.CGIHTTPRequestHandler.run_cgi() which executes a CGI script in a separate process. Using os.fork()
if possible. But os.fork()
now emits a DeprecationWarning if used in multi-threaded environment, and in future it will be an error.
$ ./python -Wa -m test -v test_httpservers -m CGIHTTPServerTestCase
...
test_accept (test.test_httpservers.CGIHTTPServerTestCase.test_accept) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_authorization (test.test_httpservers.CGIHTTPServerTestCase.test_authorization) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
test_cgi_path_in_sub_directories (test.test_httpservers.CGIHTTPServerTestCase.test_cgi_path_in_sub_directories) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
pid = os.fork()
ok
...
We should do something with this. First, the tests are polluted with presumably false alerts. Second, these alerts may be not false. Third, these tests will be broken in future, when the warning is turned into error.
Also, what actions are expected from users of http.server
when they see a warning? How they can solve the problem on their side? If http.server
is deprecated, it should be documented. If it remains, it should not produce scary warnings.
cc @gpshead
Activity
gpshead commentedon Sep 12, 2023
At a minimum we should at least silence this particular warning in this particular
CGIHTTPServerTestCase
class. If this part of the test suite starts hanging on particular platform configurations due to the technically not allowed by posix issue, we can add explicit skips for those. To fix the underlying issue and not silence it:The real bug exposed is that
http.server.CGIHTTPRequestHandler
has never been safe to use on a multithreaded server. This code - https://github.com/python/cpython/blob/main/Lib/http/server.py#L1172-L1192 - where the warning comes from needs to be rewritten to usesubprocess
.But given that
http.server
is not considered a good idea to use in serious applications (aka "production") per the warning at the top of its docs, nobody has been paying attention to that kind of maintenance on it. It does not see much use. https://docs.python.org/3/library/http.server.htmlserhiy-storchaka commentedon Sep 13, 2023
We should not simply silence deprecation warnings. We should make
os.fork()
only be used if it is safe (only in single-threaded program) or allow the user to control this by an option. We should rewrite tests and test the fork-using code without threads, and subprocess-using code with threads. We should document the limitations of using CGI in http.server.gpshead commentedon Sep 13, 2023
I wonder if we need CGIHTTPRequestHandler at all? It isn't the '90s any more, CGI is... extremely dated and often problematic. Deprecating it could be another option.
http.server.CGIHTTPRequestHandler
#109387gpshead commentedon Sep 13, 2023
the
cgi
andcgitb
modules were removed in 3.13 for https://peps.python.org/pep-0594/#deprecated-modules.Searching internally (which covers a large PyPI/github third_party corpus as well) I only found a single use of
CGIHTTPRequestHandler
in actual code in an internal demo test server. It was a misunderstanding, they didn't need to use that class and should've usedSimpleHTTPRequestHandler
as they implemented their owndo_POST
method.[-]test_httpservers produces DeprecationWarning[/-][+]test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler?[/+]gpshead commentedon Sep 13, 2023
If anyone can find actual uses of CGIHTTPRequestHandler in real world code, it'd be good to see what still uses it and why.
gh-109096: Deprecate `http.server.CGIHTTPRequestHandler` (#109387)
gpshead commentedon Sep 15, 2023
The deprecation is in for 3.13. I'll work up a small 3.12 specific test-only change to silence the warning on the CGIHTTPRequestHandler test there.
6 remaining items