After dive into Python's source code, I find out that it maintains an array of PyInt_Objects ranging from int(-5) to int(256) (#src/Objects/intobject.c)
A little experiment proves it:
>>> a = 1
>>> b = 1
>>> a is b
True
>>> a = 257
>>> b = 257
>>> a is b
False
But if I run those code together in a py file (or join them with semi-colons), the result is different:
>>> a = 257; b = 257; a is b
True
I'm curious why they are still the same object, so I digg deeper into the syntax tree and compiler, I came up with a calling hierarchy listed below:
PyRun_FileExFlags()
mod = PyParser_ASTFromFile()
node *n = PyParser_ParseFileFlagsEx() //source to cst
parsetoke()
ps = PyParser_New()
for (;;)
PyTokenizer_Get()
PyParser_AddToken(ps, ...)
mod = PyAST_FromNode(n, ...) //cst to ast
run_mod(mod, ...)
co = PyAST_Compile(mod, ...) //ast to CFG
PyFuture_FromAST()
PySymtable_Build()
co = compiler_mod()
PyEval_EvalCode(co, ...)
PyEval_EvalCodeEx()
Then I added some debug code in PyInt_FromLong and before/after PyAST_FromNode, and executed a test.py:
a = 257
b = 257
print "id(a) = %d, id(b) = %d" % (id(a), id(b))
the output looks like:
DEBUG: before PyAST_FromNode
name = a
ival = 257, id = 176046536
name = b
ival = 257, id = 176046752
name = a
name = b
DEBUG: after PyAST_FromNode
run_mod
PyAST_Compile ok
id(a) = 176046536, id(b) = 176046536
Eval ok
It means that during the cst to ast transform, two different PyInt_Objects are created (actually it's performed in the ast_for_atom() function), but they are later merged.
I find it hard to comprehend the source in PyAST_Compile and PyEval_EvalCode, so I'm here to ask for help, I'll be appreciative if some one gives a hint?
Python caches integers in the range [-5, 256], so integers in that range are usually but not always identical.
What you see for 257 is the Python compiler optimizing identical literals when compiled in the same code object.
When typing in the Python shell each line is a completely different statement, parsed and compiled separately, thus:
>>> a = 257
>>> b = 257
>>> a is b
False
But if you put the same code into a file:
$ echo 'a = 257
> b = 257
> print a is b' > testing.py
$ python testing.py
True
This happens whenever the compiler has a chance to analyze the literals together, for example when defining a function in the interactive interpreter:
>>> def test():
... a = 257
... b = 257
... print a is b
...
>>> dis.dis(test)
2 0 LOAD_CONST 1 (257)
3 STORE_FAST 0 (a)
3 6 LOAD_CONST 1 (257)
9 STORE_FAST 1 (b)
4 12 LOAD_FAST 0 (a)
15 LOAD_FAST 1 (b)
18 COMPARE_OP 8 (is)
21 PRINT_ITEM
22 PRINT_NEWLINE
23 LOAD_CONST 0 (None)
26 RETURN_VALUE
>>> test()
True
>>> test.func_code.co_consts
(None, 257)
Note how the compiled code contains a single constant for the 257.
In conclusion, the Python bytecode compiler is not able to perform massive optimizations (like statically typed languages), but it does more than you think. One of these things is to analyze usage of literals and avoid duplicating them.
Note that this does not have to do with the cache, because it works also for floats, which do not have a cache:
>>> a = 5.0
>>> b = 5.0
>>> a is b
False
>>> a = 5.0; b = 5.0
>>> a is b
True
For more complex literals, like tuples, it "doesn't work":
>>> a = (1,2)
>>> b = (1,2)
>>> a is b
False
>>> a = (1,2); b = (1,2)
>>> a is b
False
But the literals inside the tuple are shared:
>>> a = (257, 258)
>>> b = (257, 258)
>>> a[0] is b[0]
False
>>> a[1] is b[1]
False
>>> a = (257, 258); b = (257, 258)
>>> a[0] is b[0]
True
>>> a[1] is b[1]
True
(Note that constant folding and the peephole optimizer can change behaviour even between bugfix versions, so which examples return True or False is basically arbitrary and will change in the future).
Regarding why you see that two PyInt_Object are created, I'd guess that this is done to avoid literal comparison. for example, the number 257 can be expressed by multiple literals:
>>> 257
257
>>> 0x101
257
>>> 0b100000001
257
>>> 0o401
257
The parser has two choices:
Convert the literals to some common base before creating the integer, and see if the literals are equivalent. then create a single integer object.
Create the integer objects and see if they are equal. If yes, keep only a single value and assign it to all the literals, otherwise, you already have the integers to assign.
Probably the Python parser uses the second approach, which avoids rewriting the conversion code and also it's easier to extend (for example it works with floats as well).
Reading the Python/ast.c file, the function that parses all numbers is parsenumber, which calls PyOS_strtoul to obtain the integer value (for intgers) and eventually calls PyLong_FromString:
x = (long) PyOS_strtoul((char *)s, (char **)&end, 0);
if (x < 0 && errno == 0) {
return PyLong_FromString((char *)s,
(char **)0,
0);
}
As you can see here the parser does not check whether it already found an integer with the given value and so this explains why you see that two int objects are created,
and this also means that my guess was correct: the parser first creates the constants and only afterward optimizes the bytecode to use the same object for equal constants.
The code that does this check must be somewhere in Python/compile.c or Python/peephole.c, since these are the files that transform the AST into bytecode.
In particular, the compiler_add_o function seems the one that does it. There is this comment in compiler_lambda:
/* Make None the first constant, so the lambda can't have a
docstring. */
if (compiler_add_o(c, c->u->u_consts, Py_None) < 0)
return 0;
So it seems like compiler_add_o is used to insert constants for functions/lambdas etc.
The compiler_add_o function stores the constants into a dict object, and from this immediately follows that equal constants will fall in the same slot, resulting in a single constant in the final bytecode.
how can I return a new instance from a struct within a function in Julia?
function generateField(type, default, description, max, min)
return Field{type}(default,description,min,max)
end
I have a struct called Field. I just used Java logic hier and it did not work for sure :)
struct Field{T}
type::T
default::T
description::String
min::T
max::T
function Field{T}(description,min,max) where {T}
new{T}(description, min, max)
end
end
Just use deepcopy(obj) - it will make a copy of your struct object (assuming this is what you need):
julia> mutable struct Field2{T<:Real}
a::T
b::Vector{Float64}
end
julia> x = Field2{Int}(5,[4.5])
Field2{Int64}(5, [4.5])
julia> y = deepcopy(x)
Field2{Int64}(5, [4.5])
julia> y == x
false
Note that x and y point to different variables and hence x == y yields false. To compare by value one would need to iterate over all struct fields.
For this struct a function generating it could be defined as :
function f(a::T, b::Vector{Float64}) where T <: Real
return Field2{T}(a,b)
end
And used as:
julia> f(1//2, [3.5])
Field2{Rational{Int64}}(1//2, [3.5])
Finally, note that the function generating an object can have the same name as its type - in that case you are generating new constructor:
julia> Field2(a::T) where T<:Real = Field2{T}(a,Int[]);
julia> Field2(2)
Field2{Int64}(2, Float64[])
for i in 1:2
if i == 2
print(x)
end
if i == 1
x = 0
end
end
UndefVarError : x not defined
Why does the code gives that error instead of printing 0 in julia?
While in python the following code print 0?
for i in range(2):
if i==1:
print(x)
if i==0:
x=0
The reason is because in the loop a variable gets a new binding each time a loop is executed, see https://docs.julialang.org/en/latest/manual/variables-and-scoping/#For-Loops-and-Comprehensions-1.
In fact while loop changed this behavior between Julia 0.6.3 and Julia 0.7 (in Julia 0.6.3 a new binding was not created). Therefore the following code:
function f()
i=0
while i < 2
i+=1
if i == 2
print(x)
end
if i == 1
x = 0
end
end
end
Gives the following output.
Julia 0.6.3
julia> function f()
i=0
while i < 2
i+=1
if i == 2
print(x)
end
if i == 1
x = 0
end
end
end
f (generic function with 1 method)
julia> f()
0
Julia 0.7.0
julia> function f()
i=0
while i < 2
i+=1
if i == 2
print(x)
end
if i == 1
x = 0
end
end
end
f (generic function with 1 method)
julia> f()
ERROR: UndefVarError: x not defined
Stacktrace:
[1] f() at .\REPL[2]:6
[2] top-level scope
For-loop created a new binding already in Julia 0.6.3 at each iteration so it fails both under Julia 0.6.3 and Julia 0.7.0.
EDIT: I have wrapped the examples in a function, but you would get the same result if you executed the while loop in global scope.
Ignore my comment andgo with Bogumil's answer as that's is the real rwason why your x variable disapears in thesecond iteration.
If you want your code to work like in Python, you can add the global keyword to your assignment of x:
for i in 1:2
if i == 2
print(x)
end
if i == 1
global x = 0
end
end
Note that this is not recommended in most cases as it'll make your code performance suffer. Julia likes local variables that the compiler can optimise away easily.
This is somewhat related to David's question here. I'm interested in looping over an array of symbols (pointing to objects) and performing an operation on each step of the loop. For example, suppose I want to write a function that looks at a bunch of variables and converts any scalars to one-element arrays.
widget = "a scalar"
dohickey = ["item 1", "item 2"]
gizmo = "another scalar"
theVariables = [:widget, :dohickey, :gizmo]
for thisVariable in theVariables
if !(typeof(eval(thisVariable)) <: Array)
println("Found a scalar")
#Make it a vector here
end
end
Ideally at the end of this we would have the following
widget = ["a scalar"]
dohickey = ["item 1", "item 2"]
gizmo = ["another scalar"]
This is a fairly lame example, but how can this be done? I thought I should be able to use something like
:($thisVariable) = [:($thisVariable)]
but I can't get it to work.
Edit: DSM's solution below works for the case described above, but in my actual usage I want to do this inside a function. If I define
function getsort(; arga="", argb="", argc="")
filterSpecs = [:arga, :argb, :argc]
for ts in filterSpecs
if !(typeof($ts) <: Array)
#nothing here
end
end
end
and then call getsort() it throws
error compiling getsort: error compiling __getsort#31__: syntax: prefix $ in non-quoted expression
I'm clearly missing something about metaprogramming magic.
This is more a followup to #DSM's answer but writing code in comments is hard. A word of caution: eval evaluates in the global scope which can lead to strange things
julia> a = 3
3
julia> function f(a)
eval(:(a = 1))
println(a)
end
f (generic function with 1 method)
julia> f(a)
3
julia> a
1
From the code generation section of the docs, I think all you need is
#eval $thisVariable = [$thisVariable]
For example:
widget = "a scalar"
dohickey = ["item 1", "item 2"]
gizmo = "another scalar"
theVariables = [:widget, :dohickey, :gizmo]
for thisVariable in theVariables
if !(typeof(eval(thisVariable)) <: Array)
println("Found a scalar")
#eval $thisVariable = [$thisVariable]
end
end
which gives me
Found a scalar
Found a scalar
julia> widget
1-element Array{ASCIIString,1}:
"a scalar"
julia> dohickey
2-element Array{ASCIIString,1}:
"item 1"
"item 2"
julia> gizmo
1-element Array{ASCIIString,1}:
"another scalar"
Are you sure you need a macro here? If you're looking to just handle your function inputs you can do something like:
julia> function newDictOfArrays(d)
wrapped=false
for (k,v) in d
if !(typeof(v) <: AbstractArray)
wrapped = true
d[k] = [v]
end
end
(wrapped, d)
end
newDictOfArrays (generic function with 1 method)
julia> function f(; kwargs...)
(wrapped, d) = newDictOfArrays(Dict(kwargs))
if wrapped
return f(;d...)
end
println("continue knowing all keyword args have been wrapped: $kwargs")
end
f (generic function with 1 method)
julia> f(a=1, b=[2])
continue knowing all keyword args have been wrapped: Any[(:a,[1]),(:b,[2])]
julia> f(a=[1], b=[2])
continue knowing all keyword args have been wrapped: Any[(:a,[1]),(:b,[2])]
newDictOfArrays checks all members of a dictionary for values that are not subtypes of AbstractArray, and overrides that item with a wrapped value. In f, if anything was wrapped, it re-calls the same function again with the new dictionary passed as keyword arguments. No macros, eval, etc, although you could consider writing a macro that injects the code into f automatically if need this frequently.
Given a vector v = [1,..,n], I try to compute all unique sets of n elements with replacements in julia.
Since I want to do this for larger values of n, I'm looking for an efficient solution, possibly using iterators.
For example, let's consider v = [1, 2, 3]: This should results in [1,1,1], [1,1,2], [1,1,3], [1,2,2], [1,2,3], [1,3,3], [2,2,2], [2,2,3], [2,3,3], [3,3,3]. With unique, I mean that if [1,1,2] is a solution, any of its permutations [1,2,1], [2,1,1] is not.
My current solution is based on the partitions function, but does not allow me to restrict the computation on the elements [1,..,n]
for i in n:n^2
for j in partitions(i, n)
## ignore sets which exceed the range [1,n]
if maximum(j) <= n
## accept as solution
end
end
end
In julia v0.5.0, combinatorics.jl has a with_replacement_combinations method.
julia> collect(with_replacement_combinations(1:4,3))
20-element Array{Array{Int64,1},1}:
[1,1,1]
[1,1,2]
[1,1,3]
[1,1,4]
[1,2,2]
[1,2,3]
[1,2,4]
[1,3,3]
[1,3,4]
[1,4,4]
[2,2,2]
[2,2,3]
[2,2,4]
[2,3,3]
[2,3,4]
[2,4,4]
[3,3,3]
[3,3,4]
[3,4,4]
[4,4,4]
I guess, it doesn't get shorter than one-line (using Iterators).
using IterTools
import Combinatorics.combinations
n=3
collect(imap(c -> Int[c[k]-k+1 for k=1:length(c)],combinations(1:(2n-1),n)))
I believe you're looking for the product function from the Iterators package. In your case product(v,v,v) should do what's required.
Here is a function to calculate the required collection:
function calcset(n=3)
res = []
for c in combinations([1:(2n-1)],n-1)
c3 = [c,2n].-[0,c]
push!(res,vcat([fill(i,c3[n-i+1]-1) for i=1:n]...))
end
return res
end
calcset(3)
There is probably some better way to code this, but this should be enough.
Notice the result is generated through repeated push!s, so this is easily turned into an iterator, if necessary.
And in iterator form:
import Base: start, next, done, eltype, length
type ImageTypeIterator
inneritr::Base.Combinations{Array{Int64,1}}
n::Int
end
imagetype(n::Int) = ImageTypeIterator(combinations([1:(2n-1)],n-1),n)
eltype(itr::ImageTypeIterator) = Array{Int64,1}
start(itr::ImageTypeIterator) = start(itr.inneritr)
function next(itr::ImageTypeIterator,s)
(c,s) = next(itr.inneritr,s)
c3 = [c,2*itr.n].-[0,c]
(vcat([fill(i,c3[itr.n-i+1]-1) for i=1:itr.n]...),s)
end
done(itr::ImageTypeIterator,s) = done(itr.inneritr,s)
length(itr::ImageTypeIterator) = length(itr.inneritr)
# test with [1,2,3]
for t in imagetype(3) println(t) ; end
The test at the end should print the collection set in the question.
BTW the name ImageTypeIterator is an attempt to characterize the collection as the distinct types of sizes of preimages when looking at a function f : [1:n] -> [1:n]. But a different interpretation might be appropriate. Other names suggestion welcome in comments.
A faster?/clearer? implementation could use:
imagetype(n::Int) = ImageTypeIterator(combinations([1:(2n-1)],n),n)
function next(itr::ImageTypeIterator,s)
(c,s) = next(itr.inneritr,s)
v = Array(Int,itr.n)
j = 1 ; p = 1
for k=1:itr.n
while !(j in c) j += 1 ; p += 1 ; end
v[k] = p
j += 1
end
(v,s)
end
Its the same logic as above, but without too much slicing. The logic takes a subset of 2n-1 and views non-gaps as repeated values and gaps as a trigger to advance to next value.
OK, a simpler version using Iterators.jl:
using Iterators
function ff(c)
v = Array(Int,length(c))
j = 1 ; p = 1
for k=1:length(c)
while !(j in c) j += 1 ; p += 1 ; end
v[k] = p
j += 1
end
v
end
# test
n = 3
for t in imap(ff,combinations([1:(2n-1)],n)) println(t) ; end
This is perhaps the simplest version, although equivalent in methods to the other answers.
And in the spirit of brevity:
using Iterators
ff(c) = begin
j=1;p=1; [(while !(j in c) j+=1;p+=1 ; end ; j+=1 ; p) for k=1:length(c)]
end
n = 3 # test
for t in imap(ff,combinations([1:(2n-1)],n)) println(t) ; end