In 2010, the Python core team wrote PEP 3107, which introduced function annotations for Python 3.x.
Nearly 4 years ago, I wrote this response to the PEP, but I published it to a discussion site that ended up becoming defunct (Clusterify). I saw that recently, interest in function annotations for type-checking was revived by GvR, and thought I might resurrect this discussion.
Background
There is a huge flaw with the creation of Python annotations, IMO. Lack of composability.
The problem only arises when you consider that at some point in the future, there may be more than one use case for function annotations (as the PEP suggests). For example, let’s say that in my code, I use function annotations both for documentation and for optional run-time type checking. If I have a framework that expects all the annotations on my function definition to be docstrings, and another framework that expects all the annotations to be classes, how do I annotate my function with both documentation and type checks?
This amounts to lack of a standard for layering function annotations. Is this really a problem?
It’s true that some standard for this could organically form in the community. For example, one could imagine tuples being used for this. If an annotation expression is a tuple, then every framework should iterate through the items of the tuple until they find an item of the matching type. However, this won’t always work: what if two frameworks are both expecting strings, or two frameworks are both expecting classes, with different semantics?
If we used dicts, this could be avoided since you could do something like this:
def foo( *args: {"doc": "arguments", "type": list}, **kwargs: {"doc": "keyword arguments", "type": dict}): \ -> {"doc": "a bar instance", "type": Bar} |
but isn’t this getting very ugly, very quickly?
Function Annotation Syntax
An aside: my personal opinion is that the function annotation syntax is very, very ugly, and will quickly clutter function definitions so as to make them totally unreadable. If you compare how annotations would look using the PEP 3107 syntax to the kinds of ‘annotation decorators’ you find in the Pyanno project, you can see exactly what I mean.
My Proposal
What I’m proposing for this project is to work on a PEP 3107 “meta-framework”. That is, a module that provides a set of functions, classes, and possibly decorators that make using the function annotations support provided in PEP 3107 much easier, and making it possible for people to write function annotation processors that will follow conventions for annotation layering, working around the problem described above by following some basic conventions.
Here one idea, but I’d like to hear others. Still very back of the envelope:
We could work around PEP 3107’s syntax by mostly ignoring it. For example, let’s say we created a class called “Annotation” and a corresponding “ann” function. Annotations could be declared thusly:
foo_args = Annotation(std_type_check=list, doc="arguments") foo_kwargs = Annotation(std_type_check=dict, doc="keyword arguments") foo_return = Annotation(std_type_check=Bar, doc="a bar instance") def foo(*args: foo_args, **kwargs: foo_kwargs) -> foo_return: del foo_args, foo_kwargs, foo_return |
… or …
def foo(*args: ann(std_type_check=list, doc="arguments"), **kwargs: ann(std_type_check=list, doc="keyword arguments")) \ -> ann(std_type_check=Bar, doc="a Bar instance"): |
The keyword arguments that the Annotation class could take could be registered dynamically by frameworks. For example, if you import foo.stdtypecheck, then std_type_check becomes available. This is due to a registration mechanism. Of course, these aren’t very heavily namespaced, but they could be if we were to tweak the design a bit.
We could also maintain a Wiki where we describe the standard annotation names that are already taken.
An alternative to this is to make Annotation an ABC; see PEP 3119.
Annotation implementors could then implement their own Annotation types, which could then be taken as arguments to the ann function. e.g.
from typechecks import StdTypeCheck from runtimedocumentation import Documentation def foo(*args: ann(StdTypeCheck(list), Documentation("arguments"), **kwargs: ann(StdTypeCheck(list), Documentation("arguments")) \ -> ann(StdTypeCheck(Bar), Documentation("a Bar instance")): |
A final option would be to really ignore function annotation syntax and simply write decorator versions of these that modify the func_annotations dict specified in the PEP.
Reflecting on my writing, 4 years later
I still think function annotations are a “meh” feature of Python 3. Without a clear definition in the PEP of how to use them, we have a lot of experimental libraries using them in non-composable ways. Composability is important — it’s what made function and class decorators so successful as an abstraction.
But my other problem is that even if the community adopted a composition layer atop the syntax, I don’t think it would actually lead to code clarity.
What GvR is now proposing
So, that was my idea. Some sort of composability of annotations. But GvR is going in a different direction altogether.
He is saying, perhaps the non-committal approach to annotations simply led to confusion. (I thought so then, and I still think so now, so I agree.)
Perhaps instead a PEP should actually propose that annotations be used exclusively for type checking, and to actually adopt the syntax of the mypy project.
I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them.
–Guido van Rossum, Aug 13, 2014
Alex Gaynor wrote a forceful -1 against this proposal.
Python’s type system isn’t very good. It lacks many features of more powerful systems such as algebraic data types, interfaces, and parametric polymorphism. Despite this, it works pretty well because of Python’s dynamic typing. I strongly believe that attempting to enforce the existing type system would be a real shame.
PS: You’re right. None of this would provide any value for PyPy.
I imagine Gaynor’s words are felt by a lot of Python programmers. We tend not to like type checking; we even consider eager type checking to be an anti-pattern of Python code. So, why endorse weak type checking support in the language in any way?
GvR’s argument for this is mostly tooling and documentation support. Making linters a little better, making editors help with refactorings, etc. But I can’t help feeling that perhaps he’s just searching for a reasonable use case for a language syntax that is now irreversibly part of the core language and hasn’t found a strong use case in the open source community.
I might suggest that the best we could do is let Python annotations support type annotations not for a type checking / linting / IDE use case, but merely for a documentation use case. It would actually be a helpful improvement if specifying the expected type of a return value in library code made it so that this information showed up inside help() calls and obj? checks in ipython. At least programmers would get a real benefit out of that.
But I think that’s probably the best we can hope for in terms of salvaging this dangerous syntax for something useful and simple. GvR is right that mypy’s syntax is at least concise and expressive, and so it may provide a good starting point.
But let’s remember: simple is better than complex — and practicality beats purity!
One thought on “Python annotations and type-checking”