Skip to content

Breaking backward compatibility between ctypes and metaclasses in Python 3.13. #124520

@junkmd

Description

@junkmd

Documentation

Background

I am one of the maintainers of comtypes. comtypes is based on ctypes and uses metaclasses to implement IUnknown.

It was reported to the comtypes community that an error occurs when attempting to use conventional metaclasses with Python 3.13.

A similar error was encountered in pyglet, which also uses metaclasses to implement COM interfaces, when running on Python 3.13.

By referring to pyglet/pyglet#1196 and pyglet/pyglet#1199, I made several modifications to the code through trial and error, and now comtypes works in both Python 3.13 and earlier versions without problems:

  • https://github.com/enthought/comtypes/compare/a3a8733..04b766a
    • It is necessary to pass arguments directly to the metaclass instead of using __new__ in places where the metaclass is instantiated (i.e., where the class is dynamically defined). This also works in versions prior to Python 3.13.
    • After calling type.__new__, any remaining initialization would be handled in __init__ instead of in __new__. Since this results in an error in versions prior to Python 3.13, a bridge using sys.version_info is necessary.

I think these changes are likely related to #114314 and #117142 and were introduced by the PRs linked to those issues.

Since this change to ctypes breaks compatibility, I think it should be mentioned in the What’s New In Python 3.13 and/or in the ctypes documentation.
There are likely other projects besides comtypes and pyglet that rely on the combination of ctypes and metaclasses, and I want to prevent confusion for those maintainers when they try to support Python 3.13.

(Additionally, I would like to ask with the ctypes maintainers to confirm whether the changes for the metaclasses in comtypes (and pyglet) are appropriate.)

Linked PRs

Activity

AA-Turner

AA-Turner commented on Sep 25, 2024

@AA-Turner
Member
encukou

encukou commented on Sep 25, 2024

@encukou
Member

Thanks for reporting this! If I knew about this earlier I'd try to make your life easier, but at this point it looks like a docs change is the best way to go.

I'm afraid that by using type(c_void_p) to get to ctypes.PyCSimpleType internals, and by importing _pointer_type_cache, you're getting in the realm of things that can change.
Unfortunately, ctypes has been... stable? neglected? ... for a long time, and these workarounds are common in the wild.
If you can, please do plan to test with 3.14 as the alphas/betas come out, and ping me on any other incompatibilities you might find.

For the diff you linked:

Unfortunately I don't have a Windows box to test on. What error are you getting if you put all of the logic in __init__? (Is it __init__() should return None? Don't return self from __init__.)

If the if/else is needed, I recommend putting the logic in common methods like _new and _init to remove the duplication, so the branch becomes something like:

if sys.version_info >= (3, 13):
    def __new__(cls, ...):
       return cls._new(...)
    def __init__(self, ...):
       self._init(...)
else:
    def __new__(cls, ...):
       self = cls._new(cls, ...)
       self.__init__(...)
       return self

or even:

if sys.version_info >= (3, 13):
    __new__ = _new
    __init__ = _init
else:
    def __new__(cls, ...):
       self = cls._new(cls, ...)
       self._init(...)
       return self
encukou

encukou commented on Sep 25, 2024

@encukou
Member

It is necessary to pass arguments directly to the metaclass instead of using new in places where the metaclass is instantiated (i.e., where the class is dynamically defined).

Hm, I don't understand this point. Do you have an example?

added a commit that references this issue on Sep 25, 2024
junkmd

junkmd commented on Sep 25, 2024

@junkmd
ContributorAuthor

It is necessary to pass arguments directly to the metaclass instead of using new in places where the metaclass is instantiated (i.e., where the class is dynamically defined).

Hm, I don't understand this point. Do you have an example?

In comtypes, the following part corresponds to this.

https://github.com/enthought/comtypes/compare/c631f97..6f036d4

    meta = type(_safearray.tagSAFEARRAY)
-   sa_type = meta.__new__(
-       meta, "SAFEARRAY_%s" % itemtype.__name__, (_safearray.tagSAFEARRAY,), {}
-   )
+   sa_type = meta(f"SAFEARRAY_{itemtype.__name__}", (_safearray.tagSAFEARRAY,), {})

enthought/comtypes#618 (comment)

junkmd

junkmd commented on Sep 25, 2024

@junkmd
ContributorAuthor

Unfortunately I don't have a Windows box to test on. What error are you getting if you put all of the logic in __init__?

--- a/comtypes/_post_coinit/unknwn.py
+++ b/comtypes/_post_coinit/unknwn.py
@@ -75,7 +75,9 @@ class _cominterface_meta(type):
             new_cls._methods_ = methods
         if dispmethods is not None:
             new_cls._disp_methods_ = dispmethods
+        return new_cls

+    def __init__(self, name, bases, namespace):
         # If we sublass a COM interface, for example:
         #
         # class IDispatch(IUnknown):
@@ -85,23 +87,22 @@ class _cominterface_meta(type):
         # subclass of POINTER(IUnknown) because of the way ctypes
         # typechecks work.
         if bases == (object,):
-            _ptr_bases = (new_cls, _compointer_base)
+            _ptr_bases = (self, _compointer_base)
         else:
-            _ptr_bases = (new_cls, POINTER(bases[0]))
+            _ptr_bases = (self, POINTER(bases[0]))

         # The interface 'new_cls' is used as a mixin.
         p = type(_compointer_base)(
-            "POINTER(%s)" % new_cls.__name__,
+            "POINTER(%s)" % self.__name__,
             _ptr_bases,
-            {"__com_interface__": new_cls, "_needs_com_addref_": None},
+            {"__com_interface__": self, "_needs_com_addref_": None},
         )

         from ctypes import _pointer_type_cache

-        _pointer_type_cache[new_cls] = p
-
-        if new_cls._case_insensitive_:
+        _pointer_type_cache[self] = p

+        if self._case_insensitive_:
             @patcher.Patch(p)
             class CaseInsensitive(object):
                 # case insensitive attributes for COM methods and properties
@@ -155,8 +156,6 @@ class _cominterface_meta(type):

                 CopyComPointer(value, self)

-        return new_cls
-

The following error occurs in Python 3.11 when making the changes mentioned above.

Traceback (most recent call last):
  File "...\Python311\Lib\unittest\__main__.py", line 18, in <module>
    main(module=None)
  File "...\Python311\Lib\unittest\main.py", line 101, in __init__
    self.parseArgs(argv)
  File "...\Python311\Lib\unittest\main.py", line 150, in parseArgs
    self.createTests()
  File "...\Python311\Lib\unittest\main.py", line 161, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\unittest\loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\unittest\loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Python311\Lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^
    from comtypes._post_coinit import _shutdown
  File "...\comtypes\_post_coinit\__init__.py", line 15, in <module>
    from comtypes._post_coinit.unknwn import _shutdown  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\comtypes\_post_coinit\unknwn.py", line 370, in <module>
    class _compointer_base(c_void_p, metaclass=_compointer_meta):
  File "...\comtypes\_post_coinit\unknwn.py", line 95, in __init__
    p = type(_compointer_base)(
             ^^^^^^^^^^^^^^^^
NameError: name '_compointer_base' is not defined. Did you mean: '_compointer_meta'?
encukou

encukou commented on Sep 25, 2024

@encukou
Member

Thanks!
I won't be able to debug that by reading the code. I'll follow up next week when I get back to a Windows PC.

junkmd

junkmd commented on Sep 25, 2024

@junkmd
ContributorAuthor

Thank you.

If I knew about this earlier I'd try to make your life easier, but at this point it looks like a docs change is the best way to go.

Fortunately, libraries like comtypes and pyglet were able to adapt with changes to the codebase.
However, if there is a codebase that performs more complex pre-processing before calling super().__new__ within __new__, it might need more complex workarounds.

If it is still possible to modify things so that performing all initialization in __new__ does not result in an error, that would be great.
Alternatively, documenting a workaround for such cases would be helpful for maintainers of those kinds of packages.

31 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13bugs and security fixesdocsDocumentation in the Doc dirtopic-ctypes

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Breaking backward compatibility between `ctypes` and metaclasses in Python 3.13. · Issue #124520 · python/cpython