Properly define what a `closure` means

[tanzilrasheed]

Documentation

(rewritten by @picnixz)

Currently, the documentation does not use the term closure to refer to a function which has access to variables in its enclosing lexical scope. We could add the term to the glossary so as to avoid random definitions appearing on random blogs.

Original text

The wrong definition of closures of python is available all over the internet.

it is mentioned as "To form a closure it must be a nested function" and also it is only emphasizing that it only remembers it's enclosing scope.

instead the definition should be "A function that remembers the scope where it was created....."

First I got confused that it may be different in python as I wasn't able to find any formal official documentation for it, and in all other unofficial websites of python tutorial I found it as a "nested function"

I hope the formal definition will be added soon in python, as it is really terrible if there is no official documentaion and these incorrect definitions are spreading. Also because python is right now largely growing language.

[picnixz]

We define a closure variable (https://docs.python.org/3/glossary.html#term-closure-variable) but the notion of closure is tied to a function, and is defined here: https://docs.python.org/3/reference/datamodel.html#function.__closure__. See also #98176.

I think overall the problem is that people use closure differently depending on the language. When I searched on Google I found:

In Python, a closure is a function object that has access to variables in its enclosing (containing) lexical scope, even when the function is called outside that scope.

which seems correct to me. But it would indeed be good if we could define the word "closure". However, note that the docs do NOT mention "closure" anywhere but rather uses other terminology.

[picnixz]

@JelleZijlstra Do you think it's important to define the term "closure" as to refer to functions that have access to variables in their enclosing lexical scope? we don't use the term closure in the docs (according to the quick search, I didn't check with a grep) and we usually mention closure variables instead (the term is defined).

[tanzilrasheed]

I am sorry for continous problem raising(if it's unnecassary problems), but I am finding some more difficulties.

"A free variable referenced from a nested scope that is defined in an outer scope rather than being resolved at runtime from the globals or builtin namespaces. May be explicitly defined with the nonlocal keyword to allow write access, or implicitly defined if the variable is only being read."

here "A free variable referenced from a nested scope that is defined in an outer scope rather than being resolved at runtime from the globals or builtin namespaces." it seems good enough.

but here in "May be explicitly defined with the nonlocal keyword to allow write access, or implicitly defined if the variable is only being read."

it feels like it only works with enclosing local scope.

I understand the reason why it may be mentioned that way(and maybe is correct technically), but there feels a lack of clarity, which I believe should be avoided.

Maybe a better explantion can be:

"A free variable referenced from the outer namespace of a function(where the function was defined), instead of referenced from surrounding namespace where the function is called" I know my definition must be inaccurate as I am new to python but I hope you understand my point.

And thanks for replying to my queries.

[gvanrossum]

FWIW, I never used the term "closure" to describe what's going on with functions referencing nonlocal (neither local nor global) variables, when the notion was first introduced (PEP 227 by Jeremy Hylton, Python 2.1). This may be connected by my lack of interest in LISP. I'm also not convinced that it's an important concept in Python.

People apparently want to use the term to describe such functions, so I'm okay with adding it to the glossary. I still don't recommend using it in the documentation for nonlocal variable references.

Some definitions floating around apparently include "and survives the scope where it was defined" -- I don't believe that that distinction is necessary.

I don't see any way to make a closure without a nested function, since in Python, globals are explicitly excluded from the concept (hence my use of "nonlocal" above, which has a precise meaning that excludes globals and builtins as well as, obviously, locals).

[tanzilrasheed]

Thanks @gvanrossum and @picnixz for replying

now I am honestly confused if the below code is having "a" variable as a closure variable or not or else how is it working and behaving just like a closure variable.

# extra.py
a = 1
def fun():
    global a
    a += 1
    print(a)

# main.py
from extra import fun

a = 10
fun()
fun()
print(a)

output:

~/code/fastapi$ python3 main.py
2
3
10

if it's not closure it will be helpful if you just tell me what is related to it(topic name), so that I can learn that and can consider it as resolved by my side.

Thanks and regards

[gvanrossum]

Every module has its own globals -- the a in main.py is unrelated to the a in extra.py.

The glossary definition of "closure variable" clarifies that globals aren't considered closure variables (in Python's sense) because they are satisfied at runtime by a dynamic lookup in the globals of the module where the referencing function is defined. Proper (Python) closure variables are detected ("lexically") by the compiler's analysis of namespace nesting (the rules are quite complicated).

Any name reference that isn't found in a lexically enclosing function scope is treated as a global, and dynamically looked up (even if no evidence of its definition is found by the compiler), first in the function's module globals, then in the builtins.

[terryjreedy]

A closure is specifically a function defined/created/nested within a function. This includes functions defined with lambda expressions and within classes within functions, but not functions created with eval(lambda...) (see below). The point is that a closure has access to local variables of enclosing functions that it would not have if not nested. I believe that this is consistent with 'closure' as used in functional languages, where I believe the term originated.

def f(n):
    def f0(): return n
    f1 = lambda: 2*n
    print(f1.__closure__)  # (<cell at ....)
    f2  = eval('lambda n=n: 3*n', globals(), locals())  # n=n, locals() required.
    print(f2.__closure__)  # None, f2 is not a closure
    class C:
        def f3(self): return 4*n
    c = C()
    return f0(), f1(), f2(), c.f3()

print(f.__closure__)  # None, f is not a closure.
print('f(2) =', f(2))   # f(2) = (2, 4, 6, 8).

The definition proposed by OP, "A function that remembers the scope where it was created.....", is too broad if interpreted to include top-level functions. All functions remember the module scope where created (f.globals and use it for global lookups.

[terryjreedy]

Explanation of @tanzilrasheed code

from extra import fun  # This imports extra and adds it to sys.modules.
    # Executing 'a = 1' in extra while importing it sets extra.__dict__{'a'} to 1.
    # Then 'fun' is bound to extra.fun in main.__dict__.  fun is not executed, so 'a' remains 1.
    # fun.__globals__ is extra.__dict__; globals() in fun refers to this.
a = 10  # puts 'a': 10 in main.__dict__, which is invisible within fun.
fun()  # increments extra.__dict__{'a'} to 2 and prints it.
print(a)  # prints main.__dict__{'a'}, which is 10.

[tanzilrasheed]

Thanks @terryjreedy @gvanrossum @picnixz for your time, really appreciated.
By reading these explanations I am getting a better sense of it. Though I still need to learn a lot more to get full sense of your explanations, but I will see the rest myself.

Thanks once again.