Understanding Raku's `&?BLOCK` compile-time variable - metaprogramming

Question

I really appreciate the Raku's &?BLOCK variable – it lets you recurse within an unnamed block, which can be extremely powerful. For example, here's a simple, inline, and anonymous factorial function:
{ when $_ ≤ 1 { 1 };
$_ × &?BLOCK($_ - 1) }(5) # OUTPUT: «120»
However, I have some questions about it when used in more complex situations. Consider this code:
{ say "Part 1:";
my $a = 1;
print ' var one: '; dd $a;
print ' block one: '; dd &?BLOCK ;
{
my $a = 2;
print ' var two: '; dd $a;
print ' outer var: '; dd $OUTER::a;
print ' block two: '; dd &?BLOCK;
print "outer block: "; dd &?OUTER::BLOCK
}
say "\nPart 2:";
print ' block one: '; dd &?BLOCK;
print 'postfix for: '; dd &?BLOCK for (1);
print ' prefix for: '; for (1) { dd &?BLOCK }
};
which yields this output (I've shortened the block IDs):
Part 1:
var one: Int $a = 1
block one: -> ;; $_? is raw = OUTER::<$_> { #`(Block|…6696) ... }
var two: Int $a = 2
outer var: Int $a = 1
block two: -> ;; $_? is raw = OUTER::<$_> { #`(Block|…8496) ... }
outer block: -> ;; $_? is raw = OUTER::<$_> { #`(Block|…8496) ... }
Part 2:
block one: -> ;; $_? is raw = OUTER::<$_> { #`(Block|…6696) ... }
postfix for: -> ;; $_ is raw { #`(Block|…9000) ... }
prefix for: -> ;; $_ is raw { #`(Block|…9360) ... }
Here's what I don't understand about that: why does the &?OUTER::BLOCK refer (based on its ID) to block two rather than block one? Using OUTER with $a correctly causes it to refer to the outer scope, but the same thing doesn't work with &?BLOCK. Is it just not possible to use OUTER with &?BLOCK? If not, is there a way to access the outer block from the inner block? (I know that I can assign &?BLOCK to a named variable in the outer block and then access that variable in the inner block. I view that as a workaround but not a full solution because it sacrifices the ability to refer to unnamed blocks, which is where much of &?BLOCK's power comes from.)
Second, I am very confused by Part 2. I understand why the &?BLOCK that follows the prefix for refers to an inner block. But why does the &?BLOCK that precedes the postfix for also refer to its own block? Is a block implicitly created around the body of the for statement? My understanding is that the postfix forms were useful in large part because they do not require blocks. Is that incorrect?
Finally, why do some of the blocks have OUTER::<$_> in the but others do not? I'm especially confused by Block 2, which is not the outermost block.
Thanks in advance for any help you can offer! (And if any of the code behavior shown above indicates a Rakudo bug, I am happy to write it up as an issue.)

Answer 1

That's some pretty confusing stuff you've encountered. That said, it does all make some kind of sense...
Why does the &?OUTER::BLOCK refer (based on its ID) to block two rather than block one?
Per the doc, &?BLOCK is a "special compile variable", as is the case for all variables that have a ? as their twigil.
As such it's not a symbol that can be looked up at run-time, which is what syntax like $FOO::bar is supposed to be about afaik.
So I think the compiler ought by rights reject use of a "compile variable" with the package lookup syntax. (Though I'm not sure. Does it make sense to do "run-time" lookups in the COMPILING package?)
There may already be a bug filed (in either of the GH repos rakudo/rakudo/issues or raku/old-issues-tracker/issues) about it being erroneous to try to do a run-time lookup of a special compile variable (the ones with a ? twigil). If not, it makes sense to me to file one.
Using OUTER with $a correctly causes it to refer to the outer scope
The symbol associated with the $a variable in the outer block is stored in the stash associated with the outer block. This is what's referenced by OUTER.
Is it just not possible to use OUTER with &?BLOCK?
I reckon not for the reasons given above. Let's see if anyone corrects me.
If not, is there a way to access the outer block from the inner block?
You could pass it as an argument. In other words, close the inner block with }(&?BLOCK); instead of just }. Then you'd have it available as $_ in the inner block.
Why does the &?BLOCK that precedes the postfix for also refer to its own block?
It is surprising until you know why, but...
Is a block implicitly created around the body of the for statement?
Seems so, so the body can take an argument passed by each iteration of the for.
My understanding is that the postfix forms were useful in large part because they do not require blocks.
I've always thought of their benefit as being that they A) avoid a separate lexical scope and B) avoid having to type in the braces.
Is that incorrect?
It seems so. for has to be able to supply a distinct $_ to its statement(s) (you can put a series of statements in parens), so if you don't explicitly write braces, it still has to create a distinct lexical frame, and presumably it was considered better that the &?BLOCK variable tracked that distinct frame with its own $_, and "pretended" that was a "block", and displayed its gist with a {...}, despite there being no explicit {...}.
Why do some of the blocks have OUTER::<$_> in them but others do not?
While for (and given etc) always passes an "it" aka $_ argument to its blocks/statements, other blocks do not have an argument automatically passed to them, but they will accept one if it's manually passed by the writer of code manually passing one.
To support this wonderful idiom in which one can either pass or not pass an argument, blocks other than ones that are automatically fed an $_ are given this default of binding $_ to the outer block's $_.
I'm especially confused by Block 2, which is not the outermost block.
I'm confused by you being especially confused by that. :) If the foregoing hasn't sufficiently cleared this last aspect up for you, please comment on what it is about this last bit that's especially confusing.

Answer 2

During compilation the compiler has to keep track of various things. One of which is the current block that it is compiling.
The block object gets stored in the compiled code wherever it sees the special variable $?BLOCK.
Basically the compile-time variables aren't really variables, but more of a macro.
So whenever it sees $?BLOCK the compiler replaces it with whatever the current block the compiler is currently compiling.
It just happens that $?OUTER::BLOCK is somehow close enough to $?BLOCK that it replaces that too.
I can show you that there really isn't a variable by that name by trying to look it up by name.
{ say ::('&?BLOCK') } # ERROR: No such symbol '&?BLOCK'
Also every pair of {} (that isn't a hash ref or hash index) denotes a new block.
So each of these lines will say something different:
{
say $?BLOCK.WHICH;
say "{ $?BLOCK.WHICH }";
if True { say $?BLOCK.WHICH }
}
That means if you declare a variable inside one of those constructs it is contained to that construct.
"{ my $a = "abc"; say $a }"; # abc
say $a; # COMPILE ERROR: Variable '$a' is not declared
if True { my $b = "def"; say $b } # def
say $b; # COMPILE ERROR: Variable '$b' is not declared
In the case of postfix for, the left side needs to be a lambda/closure so that for can set $_ to the current value.
It was probably just easier to fake it up to be a Block than to create a new Code type just for that use.
Especially since an entire Raku source file is also considered a Block.
A bare Block can have an optional argument.
my &foo;
given 5 {
&foo = { say $_ }
}
foo( ); # 5
foo(42); # 42
If you give it an argument it sets $_ to that value.
If you don't, $_ will point to whatever $_ was outside of that declaration. (Closure)
For many of the uses of that construct, doing that can be very handy.
sub call-it-a (&c){
c()
}
sub call-it-b (&c, $arg){
c( $arg * 10 )
}
for ^5 {
call-it-a( { say $_ } ); # 0␤ 1␤ 2␤ 3␤ 4␤
call-it-b( { say $_ }, $_ ); # 0␤10␤20␤30␤40␤
}
For call-it-a we needed it to be a closure over $_ to work.
For call-it-b we needed it to be an argument instead.
By having :( ;; $_? is raw = OUTER::<$_> ) as the signature it caters to both use-cases.
This makes it easy to create simple lambdas that just do what you want them to do.