Skip to content

test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler? #109096

@serhiy-storchaka

Description

@serhiy-storchaka

Bug report

Every test class in test_httpservers starts a thread to run a server. CGIHTTPServerTestCase tests trigger htt.server.CGIHTTPRequestHandler.run_cgi() which executes a CGI script in a separate process. Using os.fork() if possible. But os.fork() now emits a DeprecationWarning if used in multi-threaded environment, and in future it will be an error.

$ ./python -Wa -m test -v test_httpservers -m CGIHTTPServerTestCase
...
test_accept (test.test_httpservers.CGIHTTPServerTestCase.test_accept) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
/home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
test_authorization (test.test_httpservers.CGIHTTPServerTestCase.test_authorization) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
test_cgi_path_in_sub_directories (test.test_httpservers.CGIHTTPServerTestCase.test_cgi_path_in_sub_directories) ... /home/serhiy/py/cpython/Lib/http/server.py:1172: DeprecationWarning: This process (pid=2768812) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
ok
...

We should do something with this. First, the tests are polluted with presumably false alerts. Second, these alerts may be not false. Third, these tests will be broken in future, when the warning is turned into error.

Also, what actions are expected from users of http.server when they see a warning? How they can solve the problem on their side? If http.server is deprecated, it should be documented. If it remains, it should not produce scary warnings.

cc @gpshead

Linked PRs

Activity

added
type-bugAn unexpected behavior, bug, or error
testsTests in the Lib/test dir
stdlibPython modules in the Lib dir
3.12only security fixes
3.13bugs and security fixes
on Sep 7, 2023
gpshead

gpshead commented on Sep 12, 2023

@gpshead
Member

At a minimum we should at least silence this particular warning in this particular CGIHTTPServerTestCase class. If this part of the test suite starts hanging on particular platform configurations due to the technically not allowed by posix issue, we can add explicit skips for those. To fix the underlying issue and not silence it:

The real bug exposed is that http.server.CGIHTTPRequestHandler has never been safe to use on a multithreaded server. This code - https://github.com/python/cpython/blob/main/Lib/http/server.py#L1172-L1192 - where the warning comes from needs to be rewritten to use subprocess.

But given that http.server is not considered a good idea to use in serious applications (aka "production") per the warning at the top of its docs, nobody has been paying attention to that kind of maintenance on it. It does not see much use. https://docs.python.org/3/library/http.server.html

serhiy-storchaka

serhiy-storchaka commented on Sep 13, 2023

@serhiy-storchaka
MemberAuthor

We should not simply silence deprecation warnings. We should make os.fork() only be used if it is safe (only in single-threaded program) or allow the user to control this by an option. We should rewrite tests and test the fork-using code without threads, and subprocess-using code with threads. We should document the limitations of using CGI in http.server.

gpshead

gpshead commented on Sep 13, 2023

@gpshead
Member

I wonder if we need CGIHTTPRequestHandler at all? It isn't the '90s any more, CGI is... extremely dated and often problematic. Deprecating it could be another option.

gpshead

gpshead commented on Sep 13, 2023

@gpshead
Member

the cgi and cgitb modules were removed in 3.13 for https://peps.python.org/pep-0594/#deprecated-modules.

Searching internally (which covers a large PyPI/github third_party corpus as well) I only found a single use of CGIHTTPRequestHandler in actual code in an internal demo test server. It was a misunderstanding, they didn't need to use that class and should've used SimpleHTTPRequestHandler as they implemented their own do_POST method.

changed the title [-]test_httpservers produces DeprecationWarning[/-] [+]test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler?[/+] on Sep 13, 2023
gpshead

gpshead commented on Sep 13, 2023

@gpshead
Member

If anyone can find actual uses of CGIHTTPRequestHandler in real world code, it'd be good to see what still uses it and why.

added a commit that references this issue on Sep 15, 2023
59073c9
gpshead

gpshead commented on Sep 15, 2023

@gpshead
Member

The deprecation is in for 3.13. I'll work up a small 3.12 specific test-only change to silence the warning on the CGIHTTPRequestHandler test there.

6 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13bugs and security fixesstdlibPython modules in the Lib dirtestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      test_httpservers causes os.fork DeprecationWarning, can we deprecate CGIHTTPRequestHandler? · Issue #109096 · python/cpython