I am working on Make a Lisp to learn Rust.
As part of the evaluator step I need to create an associative structure mapping strings (or whatever) to functions. In languages I'm more familiar with (Ruby, Clojure) I would simply define anonymous functions in a hashmap, eg
{ :+ (fn [a b] (+ a b))
:- (fn [a b] (- a b)) } ; etc
In Rust, this isn't possible because of the type error expected closure, found a different closure.
let repl_env = HashMap::new();
repl_env.insert("+", |a, b| a + b);
repl_env.insert("-", |a, b| a - b); // expected closure, found a different closure
I guess what's happening here is
the compiler is inferring the types in the HashMap as <String, WhateverTypeTheFirstClosureIs>
the second closure doesn't have the same time because as the compiler error goes on to say, no two closures, even if identical, have the same type?
I can add types to the arguments and thus make the return type inferrable, but this doesn't help — I assume I need to somehow annotate the types of the closures themselves? I can't find a way to do this in the docs. The error suggests Boxing, but wrapping the closures in Box::new doesn't help. (I've never Boxed before).
I'm also wondering whether Rust closures in a HashMap are the right thing to use here. Should I maybe use an enum of some sort? (I'll want to dynamically add things later when I'm adding user-defined functions to my lisp, so I assume not).
You don't actually need closures here since you're not closing over anything, you're just using anonymous static functions.
That means you can just cast your anonymous functions to function pointers and it'll work fine e.g.
let mut repl_env: HashMap<_, fn(i32, i32) -> i32> = HashMap::new();
repl_env.insert("+", |a, b| a + b);
repl_env.insert("-", |a, b| a - b);
If you did need closures, then you'd have to use some sort of dynamic dispatch (as each closure is an entirely different type), generally something along the lines of `Box e.g.
let mut repl_env: HashMap<_, Box<dyn Fn(i32, i32) -> i32>> = HashMap::new();
repl_env.insert("+", Box::new(|a, b| a + b));
repl_env.insert("-", Box::new(|a, b| a - b));
That latter version is what the error message talks about: boxing the closure [and] (in this case) making it a trait object.
Should I maybe use an enum of some sort?
Using an enum might be a good idea as eventually you'll have functions with different signatures: here you're limited to the signature of (i32, i32) -> i32. Though you can alternatively update the universal signature to e.g. (Values) -> Value and internally perform dispatch / typechecking / ...
I'll want to dynamically add things later when I'm adding user-defined functions to my lisp, so I assume not.
If you want to keep the distinction between "native" and "user" functions, you can always have a variant which stores the result of evaluating a user function in your enum. You'll need a way to distinguish them anyway since at the end of the day the evaluator won't call them the same way.
Though I would think the usual method is to make all visible functions uniformly "userland", and implement those in terms of builtins (which may be distinguished syntactically, or special forms, or even pattern-matched by the compiler, and may or may not be accessible to userland).
I don't know how Lisps usually do it at the root, but e.g. you could have the interpreter evaluate the terms of
(+ a b)
and check that + is the known / standard function and a and b are known compatible types, then do the operation internally without actually calling anything.
The fun bit is that you can provide a properly defined + function as
(define (+ a b)
(+ a b))
which looks nonsensical at first glance since it should recurse on itself, but the inner node can be pattern-matched by the interpreter and so this serves as a hook for higher-order functions.
That is how many Smalltalks handled this, for instance. Non-literal sends would land into the trampoline method, which would then bounce into the built-in for the operation, invisibly.
Related
After some discussion, I'm now a little bit confused about the relation between auto-dereferencing and deref coercion.
It seems that the term "auto-dereferencing" applies only when the target to dereference is a method receiver,
whereas it seems that the term "deref coercion" applies to function arguments and all contexts it needs to.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
Finally, it sounds natural to use the term "autoderef" in all the cases where the compiler implicitly transforms &&&&x to &x:
pub fn foo(_v: &str) -> bool {
false
}
let x="hello world";
foo(&&&&x);
Is this the general consensus of the community?
The parallels between the two cases are rather superficial.
In a method call expression, the compiler first needs to determine which method to call. This decision is based on the type of the receiver. The compiler builds a list of candidate receiver types, which include all types obtained by repeatedly derefencing the receiver, but also &T and &mut T for all types T encountered. This is the reason why you can call a method receiving &mut self directly as x.foo() instead of having to write (&mut x).foo(). For each type in the candidate list, the compiler then looks up inherent methods and methods on visible traits. See the language reference for further details.
A deref coercion is rather different. It only occurs at a coercion site where the compiler exactly knows what type to expect. If the actual type encountered is different from the expected type, the compiler can use any coercion, including a deref coercion, to convert the actual type to the expected type. The list of possible coercions includes unsized coercions, pointer weakening and deref coercions. See the the chapter on coercions in the Nomicon for further details.
So these are really two quite different mechanisms – one for finding the right method, and one for converting types when it is already known what type exactly to expect. The first mechanism also automatically references the receiver, which can never happen in a coercion.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
Not every dereferencing is a deref coercion. If you write *x, you explicitly dereference x. A deref coercion in contrast is performed implicitly by the compiler, and only in places where the compiler knows the expected type.
The semantics of dereferencing depend on whether the type of x is a pointer type, i.e. a reference or a raw pointer, or not. For pointer types, *x denotes the object x points to, while for other types *x is equivalent to *Deref::deref(&x) (or the mutable anlogue of this).
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
I'm not quite sure what your syntax is supposed to mean – it's certainly not valid Rust syntax – but I guess you are asking whether derefencing an instance of &T to T is built into the compiler. As mentioned above, dereferencing of pointer types, including references, is built into the compiler, but there is also a blanket implementation of Deref for &T in the standard library. This blanket implementation is useful for generic code – the trait bound T: Deref<Target = U> otherwise wouldn't allow for T = &U.
It seems asymmetric and basic, so I want to understand the reasoning of this, why is that PartialOrd defines partial_cmp, to return Option<Ordering>;
fn partial_cmp(&self, other: &Rhs) -> Option<Ordering> {}
And PartialEq defines eq to return bool
fn eq(&self, other: &Rhs) -> bool {}
Why doesn't PartialEq::eq return Option<Eq> where None represents things that can't be determined to be equal? Why does PartialEq get to skirt having the knowledge of which values can not compared for equality and thus which prevent the implementation of a full Eq trait, and yet PartialOrd has to know which values can not be compared for a full ordering (so it can return that information to the user as None)?
I can think of a few reason's why these traits are defined the way they are:
PartialEq is the trait for the == operator. If it were to return an Option, language ergonomics around ifs and other control flows would need to at least be reconsidered.
When considering whether values are equal or not, "non-comparable" values fall into the latter category. There need not be a third answer; either they're equal or they're not. This is the same for >, >=, <, and <= provided by PartialOrd, they all return bool regardless if the values are "non-comparable" (they return false in that case).
The meaning of "Partial" in the types differ. "Partial" in PartialOrd means that types may not have a total order. Whereas "Partial" in PartialEq means types may not have complete equivalence relations (may not be reflexive, symmetric, or transitive). The naming is similar because they both cover cases where types don't behave nicely, but they are conveying slightly different concepts. The APIs do not need to be identical.
Because PartialOrd is cheating a bit, by mathematical standards.
Abstractly speaking, a mathematical structure is defined by two things: some operations and some axioms associated to those operations. In a programming language, we can define traits (or typeclasses, or whatever your favorite language calls them) which embody mathematical structures. We define the trait to support certain operations and simply trust that the programmer follows the axioms.
Total orderings (i.e. Ord) and partial orderings (i.e. PartialOrd) are usually modeled using a single operation: <=. And the difference in the two structures is based on what properties we require of <=. Likewise, partial equivalence relations (PartialEq) and equivalence relations (Eq) are defined in terms of ==, and the two differ only in what properties.
If we were to define PartialOrd and Ord in this way in Rust, it would look something like (I'm omitting the RHS type argument for simplicity reasons)
trait PartialOrd : PartialEq {
fn less_or_equal(self, other: Self) -> bool;
}
trait Ord : PartialOrd {}
which looks more like what the PartialEq / Eq dichotomy looks like. Rust chose to implement PartialOrd in terms of another operator: the trichotomy operator partial_cmp and cmp. In any total ordering, we can define an operator cmp(a, b) which returns whether a is smaller than, equal to, or larger than b. And in a total ordering, exactly one of those is true. In a partial ordering, however, we can still define this operator partial_cmp(a, b), but there's a fourth option, namely that the two are simply not comparable.
As for why Rust did it this way, I couldn't say for certain. Haskell has a similar setup, with an Ord typeclass, and in Haskell you can implement either <= or compare, and you get the other one for free. Python used to use __cmp__ (which is basically Rust's cmp) but Python 3 switched completely to defining <= and using functools.total_ordering for comparisons.
We could debate back and forth which mechanism is better. cmp has its benefits in terms of clarity and symmetry, but <= is better for studying abstract mathematics. If you define both structures the mathematical way (== for PartialEq and <= for PartialOrd), then in both cases you get a single binary operator which "returns" bool, and all of the other differences are in the axioms. The reason we don't have that satisfying symmetry in Rust is that Rust's PartialOrd is defined in terms of an operator that only makes total mathematical sense in Ord; we've taken a total ordering operator and removed some of its legs, leaving an operator that only works "some of the time".
After some discussion, I'm now a little bit confused about the relation between auto-dereferencing and deref coercion.
It seems that the term "auto-dereferencing" applies only when the target to dereference is a method receiver,
whereas it seems that the term "deref coercion" applies to function arguments and all contexts it needs to.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
Finally, it sounds natural to use the term "autoderef" in all the cases where the compiler implicitly transforms &&&&x to &x:
pub fn foo(_v: &str) -> bool {
false
}
let x="hello world";
foo(&&&&x);
Is this the general consensus of the community?
The parallels between the two cases are rather superficial.
In a method call expression, the compiler first needs to determine which method to call. This decision is based on the type of the receiver. The compiler builds a list of candidate receiver types, which include all types obtained by repeatedly derefencing the receiver, but also &T and &mut T for all types T encountered. This is the reason why you can call a method receiving &mut self directly as x.foo() instead of having to write (&mut x).foo(). For each type in the candidate list, the compiler then looks up inherent methods and methods on visible traits. See the language reference for further details.
A deref coercion is rather different. It only occurs at a coercion site where the compiler exactly knows what type to expect. If the actual type encountered is different from the expected type, the compiler can use any coercion, including a deref coercion, to convert the actual type to the expected type. The list of possible coercions includes unsized coercions, pointer weakening and deref coercions. See the the chapter on coercions in the Nomicon for further details.
So these are really two quite different mechanisms – one for finding the right method, and one for converting types when it is already known what type exactly to expect. The first mechanism also automatically references the receiver, which can never happen in a coercion.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
Not every dereferencing is a deref coercion. If you write *x, you explicitly dereference x. A deref coercion in contrast is performed implicitly by the compiler, and only in places where the compiler knows the expected type.
The semantics of dereferencing depend on whether the type of x is a pointer type, i.e. a reference or a raw pointer, or not. For pointer types, *x denotes the object x points to, while for other types *x is equivalent to *Deref::deref(&x) (or the mutable anlogue of this).
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
I'm not quite sure what your syntax is supposed to mean – it's certainly not valid Rust syntax – but I guess you are asking whether derefencing an instance of &T to T is built into the compiler. As mentioned above, dereferencing of pointer types, including references, is built into the compiler, but there is also a blanket implementation of Deref for &T in the standard library. This blanket implementation is useful for generic code – the trait bound T: Deref<Target = U> otherwise wouldn't allow for T = &U.
After some discussion, I'm now a little bit confused about the relation between auto-dereferencing and deref coercion.
It seems that the term "auto-dereferencing" applies only when the target to dereference is a method receiver,
whereas it seems that the term "deref coercion" applies to function arguments and all contexts it needs to.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
Finally, it sounds natural to use the term "autoderef" in all the cases where the compiler implicitly transforms &&&&x to &x:
pub fn foo(_v: &str) -> bool {
false
}
let x="hello world";
foo(&&&&x);
Is this the general consensus of the community?
The parallels between the two cases are rather superficial.
In a method call expression, the compiler first needs to determine which method to call. This decision is based on the type of the receiver. The compiler builds a list of candidate receiver types, which include all types obtained by repeatedly derefencing the receiver, but also &T and &mut T for all types T encountered. This is the reason why you can call a method receiving &mut self directly as x.foo() instead of having to write (&mut x).foo(). For each type in the candidate list, the compiler then looks up inherent methods and methods on visible traits. See the language reference for further details.
A deref coercion is rather different. It only occurs at a coercion site where the compiler exactly knows what type to expect. If the actual type encountered is different from the expected type, the compiler can use any coercion, including a deref coercion, to convert the actual type to the expected type. The list of possible coercions includes unsized coercions, pointer weakening and deref coercions. See the the chapter on coercions in the Nomicon for further details.
So these are really two quite different mechanisms – one for finding the right method, and one for converting types when it is already known what type exactly to expect. The first mechanism also automatically references the receiver, which can never happen in a coercion.
I thought that a dereference does not always involve deref coercion, but I'm not sure: does dereferencing always use some Deref::deref trait implementation?
Not every dereferencing is a deref coercion. If you write *x, you explicitly dereference x. A deref coercion in contrast is performed implicitly by the compiler, and only in places where the compiler knows the expected type.
The semantics of dereferencing depend on whether the type of x is a pointer type, i.e. a reference or a raw pointer, or not. For pointer types, *x denotes the object x points to, while for other types *x is equivalent to *Deref::deref(&x) (or the mutable anlogue of this).
If so, is the implementor of T: Deref<Target = U> where T: &U built into the compiler?
I'm not quite sure what your syntax is supposed to mean – it's certainly not valid Rust syntax – but I guess you are asking whether derefencing an instance of &T to T is built into the compiler. As mentioned above, dereferencing of pointer types, including references, is built into the compiler, but there is also a blanket implementation of Deref for &T in the standard library. This blanket implementation is useful for generic code – the trait bound T: Deref<Target = U> otherwise wouldn't allow for T = &U.
I should probably first mention that I'm pretty new to Haskell. Is there a particular reason to keep the let expression in Haskell?
I know that Haskell got rid of the rec keyword that corresponds to the Y-combinator portion of a let statement that indicates it's recursive. Why didn't they get rid of the let statement altogether?
If they did, statements will seem more iterative to some degree. For example, something like:
let y = 1+2
z = 4+6
in y+z
would just be:
y = 1+2
z = 4+6
y+z
Which is more readable and easier for someone new to functional programming to follow. The only reason I can think of to keep it around is something like this:
aaa = let y = 1+2
z = 4+6
in y+z
Which would look this this without the let, which I think ends up being ambiguous grammar:
aaa =
y = 1+2
z = 4+6
y+z
But if Haskell didn't ignore whitespace, and code blocks/scope worked similar to Python, would it be able to remove the let?
Is there a stronger reason to keep around let?
Sorry if this question seems stupid, I'm just trying to understand more about why it's in there.
Syntactically you can easily imagine a language without let. Immediately, we can produce this in Haskell by simply relying on where if we wanted. Beyond that are many possible syntaxes.
Semantically, you might think that let could translate away to something like this
let x = e in g ==> (\x -> g) e
and, indeed, at runtime these two expressions are identical (modulo recursive bindings, but those can be achieved with fix). Traditionally, however, let has special typing semantics (along with where and top-level name definitions... all of which being, effectively, syntax sugar for let).
In particular, in the Hindley-Milner type system which forms the foundation of Haskell there's a notion of let-generalization. Intuitively, it regards situations where we upgrade functions to their most polymorphic form. In particular, if we have a function appearing in an expression somewhere with a type like
a -> b -> c
those variables, a, b, and c, may or may not already have meaning in that expression. In particular, they're assumed to be fixed yet unknown types. Compare that to the type
forall a b c. a -> b -> c
which includes the notion of polymorphism by stating, immediately, that even if there happen to be type variables a, b, and c available in the envionment, these references are fresh.
This is an incredibly important step in the HM inference algorithm as it is how polymorphism is generated allowing HM to reach its more general types. Unfortunately, it's not possible to do this step whenever we please—it must be done at controlled points.
This is what let-generalization does: it says that types should be generalized to polymorphic types when they are let-bound to a particular name. Such generalization does not occur when they are merely passed into functions as arguments.
So, ultimately, you need a form of "let" in order to run the HM inference algorithm. Further, it cannot just be syntax sugar for function application despite them having equivalent runtime characteristics.
Syntactically, this "let" notion might be called let or where or by a convention of top-level name binding (all three are available in Haskell). So long as it exists and is a primary method for generating bound names where people expect polymorphism then it'll have the right behavior.
There are important reasons why Haskell and other functional languages use let. I'll try to describe them step by step:
Quantification of type variables
The Damas-Hindley-Milner type system used in Haskell and other functional languages allows polymorphic types, but the type quantifiers are allowed only in front of a given type expression. For example, if we write
const :: a -> b -> a
const x y = x
then the type of const is polymorphic, it is implicitly universally quantified as
∀a.∀b. a -> b -> a
and const can be specialized to any type that we obtain by substituting two type expressions for a and b.
However, the type system doesn't allow quantifiers inside type expressions, such as
(∀a. a -> a) -> (∀b. b -> b)
Such types are allowed in System F, but then type checking and type inference is undecidable, which means that the compiler wouldn't be able to infer types for us and we would have to explicitly annotate expressions with types.
(For long time the question of decidability of type-checking in System F had been open, and it had been sometimes addressed as "an embarrassing open problem", because the undecidability had been proven for many other systems but this one, until proved by Joe Wells in 1994.)
(GHC allows you to enable such explicit inner quantifiers using the RankNTypes extension, but as mentioned, the types can't be inferred automatically.)
Types of lambda abstractions
Consider the expression λx.M, or in Haskell notation \x -> M,
where M is some term containing x. If the type of x is a and the type of M is b, then the type of the whole expression will be λx.M : a → b. Because of the above restriction, a must not contain ∀, therefore the type of x can't contain type quantifiers, it can't be polymorphic (or in other words it must be monomorphic).
Why lambda abstraction isn't enough
Consider this simple Haskell program:
i :: a -> a
i x = x
foo :: a -> a
foo = i i
Let's disregard for now that foo isn't very useful. The main point is that id in the definition of foo is instantiated with two different types. The first one
i :: (a -> a) -> (a -> a)
and the second one
i :: a -> a
Now if we try to convert this program into the pure lambda calculus syntax without let, we'd end up with
(λi.i i)(λx.x)
where the first part is the definition of foo and the second part is the definition of i. But this term will not type check. The problem is that i must have a monomorphic type (as described above), but we need it polymorphic so that we can instantiate i to the two different types.
Indeed, if you try to typecheck i -> i i in Haskell, it will fail. There is no monomorphic type we can assign to i so that i i would typecheck.
let solves the problem
If we write let i x = x in i i, the situation is different. Unlike in the previous paragraph, there is no lambda here, there is no self-contained expression like λi.i i, where we'd need a polymorphic type for the abstracted variable i. Therefore let can allow i to have a polymorhpic type, in this case ∀a.a → a and so i i typechecks.
Without let, if we compiled a Haskell program and converted it to a single lambda term, every function would have to be assigned a single monomorphic type! This would be pretty useless.
So let is an essential construction that allows polymorhism in languages based on Damas-Hindley-Milner type systems.
The History of Haskell speaks a bit to the fact that Haskell has long since embraced a complex surface syntax.
It took some while to identify the stylistic choice as we have done here, but once we had done so, we engaged in furious debate about which style was “better.” An underlying assumption was that if possible there should be “just one way to do something,” so that, for example, having both let and where would be redundant and confusing.
In the end, we abandoned the underlying assumption, and provided full syntactic support for both styles. This may seem like a classic committee decision, but it is one that the present authors believe was a fine choice, and that we now regard as a strength of the language. Different constructs have different nuances, and real programmers do in practice employ both let and where, both guards and conditionals, both pattern-matching definitions and case expressions—not only in the same program but sometimes in the same function definition. It is certainly true that the additional syntactic sugar makes the language seem more elaborate, but it is a superficial sort of complexity, easily explained by purely syntactic transformations.
This is not a stupid question. It is completely reasonable.
First, let/in bindings are syntactically unambiguous and can be rewritten in a simple mechanical way into lambdas.
Second, and because of this, let ... in ... is an expression: that is, it can be written wherever expressions are allowed. In contrast, your suggested syntax is more similar to where, which is bound to a surrounding syntactic construct, like the pattern matching line of a function definition.
One might also make an argument that your suggested syntax is too imperative in style, but this is certainly subjective.
You might prefer using where to let. Many Haskell developers do. It's a reasonable choice.
There is a good reason why let is there:
let can be used within the do notation.
It can be used within list comprehension.
It can be used within function definition as mentioned here conveniently.
You give the following example as an alternative to let :
y = 1+2
z = 4+6
y+z
The above example will not typecheck and the y and z will also lead to the pollution of global namespace which can be avoided using let.
Part of the reason Haskell's let looks like it does is also the consistent way it manages its indentation sensitivity. Every indentation-sensitive construct works the same way: first there's an introducing keyword (let, where, do, of); then the next token's position determines what is the indentation level for this block; and subsequent lines that start at the same level are considered to be a new element in the block. That's why you can have
let a = 1
b = 2
in a + b
or
let
a = 1
b = 2
in a + b
but not
let a = 1
b = 2
in a + b
I think it might actually be possible to have keywordless indentation-based bindings without making the syntax technically ambiguous. But I think there is value in the current consistency, at least for the principle of least surprise. Once you see how one indentation-sensitive construct works, they all work the same. And as a bonus, they all have the same indentation-insensitive equivalent. This
keyword <element 1>
<element 2>
<element 3>
is always equivalent to
keyword { <element 1>; <element 2>; <element 3> }
In fact, as a mainly F# developer, this is something I envy from Haskell: F#'s indentation rules are more complex and not always consistent.