I am a bit confused about the behavior of the range() function in a specific use case.
When I was testing some code I wrote using nested FOR loops, in some cases, the statements in certain loops never seemed to execute at all. I eventually realized that I was in some cases feeding a range() call with an input like:
range(i,2) # where i is 2, giving range(2,2)
...which threw no error, but apparently never executed the for loop contents. After some reading on Python3's FOR implementation, I then added "else:" statements to my loop:
for i in range(a,b): # where a=b, i.e. range(2,2)
[skipped code]
else:
[other code]
...and the else-case code executed fine, as I guess all possible iterators for the given range values were (already) exhausted, and the for-else case was triggered as it's designed to be when that happens.
From what I can see in the documentation for range(), I found: "A range object will be empty if r[0] does not meet the value constraint." ( https://docs.python.org/3/library/stdtypes.html#range ). I'm not quite sure what the "value constraint" is in this case, but if I'm understanding right, "range(a,b)" will return an empty list if a >= b.
My question is, is my understanding correct about when range() returns []? Also, are there any other kinds of input cases where range(a,b) returns [], or other obscure edge case behaviors I should be aware of? Thank you.
as you can see in this documentation, when you use range(a,b) you're setting its start and stop parameters.
what you need to know is that stop parameter is always excluded just like in lists slicing.
another remark is that you can set the step, so if you set a negative step you can actually use a >= b like in this case:
range(10,4,-1)
Also please notice that all parameters need to be integers.
I recommend you visit the documentation provided above it's quite helpful.
range(n) generates an iterator to progress the integer numbers starting with 0 and ending with (n-1).
With reference to your FOR loop, it was not executed because the ending number (i.e. n - 1 = 2 - 1 = 1) is less than the starting number, 2. Since the step argument is omitted in your FOR loop, it defaults to 1. The step can be both negative and positive, but not zero.
Syntax:
range(begin, end[, step])
Examples:
Both examples below will produce empty list.
list(range(0))
list(range(2,2))
Related
Been preparing of Coding Interviews. I have been trying to beef up my dynamic programming skills and came across Alvin's awesome channel and this great video about some of the approaches to take.
In one of the programming sections - The problem statement is "Can you construct the target string with the contents in an array" - he goes on to correctly use recursion as an approach. But I got stuck in his base case.
He indicates for example that this should return true:
canConstruct('StakeBoard', ['sta', 'te', 'bo', 'ard']
He goes on to settle the recursion base case as the following returning true by literally saying "Because to generate and empty string, you can generally take no-zero elements from the array!". This is the part I did not understand. What does non-zero elements of the array mean here to make this a base case?
canConstruct('', ['cat', dog'])
"Because to generate an empty string, you can generally take no-zero elements from the array!"
It should be something like
"Because to generate an empty string, you can generally take no or zero elements from the array!"
The next question states that
What does non-zero elements of the array mean here to make this a base case?
Actually, he meant
In case the target is empty, no matter what array of words are given, taking no or zero words will return the answer true.
We call a case base case when we have a definite answer for that case. And the case described above definitely looks like one thereby addressed as the base case.
I'm not asking for an answer to the question, but rather how I, on my own, could have gotten the answer.
Original Question:
Does the following code cause Python to make a new list of size (len(nums) - 1) in memory that then gets iterated over?
for item in nums[1:]:
# do stuff with item
Original Answer
A similarish question is asked here and there is a subcomment by Srinivas Reddy Thatiparthy saying that a new sublist is created.
But, there is no detail given about how he arrived at this answer, which I think makes it very different from what I'm looking for.
Question:
How could I have figured out on my own what the answer to my question is?
I've had similar questions before. For instance, I learned that if I do my_function(nums[1:]), I don't pass in a "slice" but rather a completely new, different sublist! I found this out by just testing whether the original list passed into my_function was modified post-function (it wasn't).
But I don't see an immediate way to figure out if Python is making a new sublist for the for loop example. Please help me to know how to do this.
side note
By the way, this is the current solution I'm using from the original stackoverflow post solutions:
for indx, item in enumerate(nums):
if indx == 0:
continue
# do stuff w items
In general, the easy way to learn if you have a new chunk of data or just a new reference to an existing chunk of data is to modify the data through one reference, and then see if it is also modified through the other. (It sounds like that's "the hard way" you did, but I would recommend it as a general technique.) Some psuedocode would look like:
function areSameRef(thing1, thing2){
thing1.modify()
return thing1.equals(thing2) //make sure this is not just a referential equality check
}
It is very rare that this will fail, and essentially requires behind-the-scenes optimizations where data isn't cloned immediately but only when modified. In this case the fact that the underlying data is the same is being hidden from you, and in most cases, you should just trust that whoever did the hiding knows what they're doing. Exceptions are if they did it wrong, or if you have some complex performance issues. For that you may need to turn to more language-specific debugging or profiling tools. (See below for more)
Do also be careful about cases where part of the data may be shared - for instance, look up cons lists and tail sharing. In those cases if you do something like:
function foo(list1, list2){
list1.append(someElement)
return list1.length == list2.length
}
will return false - the element is only added to the first list, but something like
function bar(list1, list2){
list1.set(someIndex, someElement)
return list1.get(someIndex)==list2.get(someIndex)
}
will return true (though in practice, lists created that way usually don't have an interface that allows mutability.)
I don't see a question in part 2, but yes, your conclusion looks valid to me.
EDIT: More on actual memory usage
As you pointed out, there are situations where that sort of test won't work because you don't actually have two references, as in the for i in [nums 1:] case. In that case I would say turn to a profiler, but you couldn't really trust the results.
The reason for that comes down to how compilers/interpreters work, and the contract they fulfill in the language specification. The general rule is that the interpreter is allowed to re-arrange and modify the execution of your code in any way that does not change the results, but may change the memory or time performance. So, if the state of your code and all the I/O are the same, it should not be possible for foo(5) to return 6 in one interpreter implementation/execution and 7 in another, but it is valid for them to take very different amounts of time and memory.
This matters because a lot of what interpreters and compilers do is behind-the-scenes optimizations; they will try to make your code run as fast as possible and with as small a memory footprint as possible, so long as the results are the same. However, it can only do so when it can prove that the changes will not modify the results.
This means that if you write a simple test case, the interpreter may optimize it behind the scenes to minimize the memory usage and give you one result - "no new list is created." But, if you try to trust that result in real code, the real code may be too complex for the compiler to tell if the optimization is safe, and it may fail. It can also depend upon the specific interpreter version, environmental variables or available hardware resources.
Here's an example:
def foo(x : int):
l = range(9999)
return 5
def bar(x:int):
l = range(9999)
if (x + 1 != (x*2+2)/2):
return l[x]
else:
return 5
I can't promise this for any particular language, but I would usually expect foo and bar to have much different memory usages. In foo, any moderately-well-created interpreter should be able to tell that l is never referenced before it goes out of scope, and thus can freely skip actually allocating any memory at all as a safe operation. In bar (unless I failed at arithmetic), l will never be used either - but knowing that requires some reasoning about the condition of the if statement. It takes a much smarter interpreter to recognize that, so even though these two code snippets might look the same logically, they can have very different behind-the-scenes performances.
EDIT: As has been pointed out to my, Python specifically may not be able to optimize either of these, given the dynamic nature of the language; the range function and the list type may both have been re-assigned or altered from elsewhere in the code. Without specific expertise in the python optimization world I can't say what they do or don't do. Anyway I'm leaving this here for edification on the general concept of optimizations, but take my error as a case lesson in "reasoning about optimization is hard".
All of that being said: FWIW, I strongly suspect that the python interpreter is smart enough to recognize that for i in nums[1:] should not actually allocate new memory, but just iterate over a slice. That looks to my eyes to be a relatively simple, safe and valuable transformation on a very common use case, so I would expect the (highly optimized) python interpreter to handle it.
EDIT2: As a final (opinionated) note, I'm less confident about that in Python than I am in almost any other language, because Python syntax is so flexible and allows so many strange things. This makes it much more difficult for the python interpreter (or a human, for that matter) to say anything with confidence, because the space of "legal python code" is so large. This is a big part of why I prefer much stricter languages like Rust, which force the programmer to color inside the lines but result in much more predictable behaviors.
EDIT3: As a post-final note, usually for things like this it's best to trust that the execution environment is handling these sorts of low-level optimizations. Nine times out of ten, don't try to solve this kind of performance problem until something actually breaks.
As for knowing how list slice works, from the language reference Sequence Types — list, tuple, range, we know that
s[i:j] - The slice of s from i to j is defined as the sequence of
items with index k such that i <= k < j.
So, the slice creates a new sequence but we don't know whether that sequence is a list or whether there is some clever way that the same list object somehow represents both of these sequences. That's not too surprising with the python language spec where lists are described as part of the general discussion of sequences and the spec never really tries to cover all of the details for object implementation.
That's because in the end, something like nums[1:] is really just syntactic sugar for nums.__getitem__(slice(1, None)), meaning that lists get to decide for themselves what slicing means. And you need to go to the source for the implementation. See the list_subscript function in listobject.c.
But we can experiment. Looking at the doucmentation for The for statement,
for_stmt ::= "for" target_list "in" starred_list ":" suite
["else" ":" suite]
The starred_list expression is evaluated once; it should yield an iterable object.
So, nums[1:] is an expression that must yield an iterable object and we can assign that object to an intermediate variable.
nums = [1 ,2, 3]
tmp = nums[1:]
for item in tmp:
pass
tmp[0] = "new stuff"
assert id(nums) != id(tmp), "List slice creates a new object"
assert type(tmp) == type(nums), "List slice creates a new list"
assert 999 not in nums, "List slice doesn't affect original"
Run that, and if neither assertion error is raised, you know that a new list was created.
Other sequence-like objects may work radically different. In a numpy array, for instance, two array objects may indeed reference the same memory. In this example, that final assert will be raised because the slice is another view into the same array. Yes, this can keep you up all night.
import numpy as np
nums = np.array([1,2,3])
tmp = nums[1:]
for item in tmp:
pass
tmp[0] = 999
assert id(nums) != id(tmp), "array slice creates a new object"
assert type(tmp) == type(nums), "array slice creates a new list"
assert 999 not in nums, "array slice doesn't affect original"
You can use the new Walrus operator := to capture the temporary object created by Python for the slice. A little investigation demonstrates that they aren't the same object.
import sys
print(sys.version)
a = list(range(1000))
for i in (b := a[1:]):
b[0] = 906
print(b is a)
print(a[:10])
print(b[:10])
print(sys.getsizeof(a))
print(sys.getsizeof(b))
Generates the following output:
3.11.0 (main, Nov 4 2022, 00:14:47) [GCC 7.5.0]
False
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[906, 2, 3, 4, 5, 6, 7, 8, 9, 10]
8056
8048
See for yourself on the Godbolt Compiler Explorer where you can also see the compiler generated code.
I will start this off by saying that I have not done any schooling. All of my programming knowledge has come from 12 years of doing various projects in which I had to write a program of some sort in some language.
That said. I am helping my friend who is just getting into programming and who is taking a introductory python class. Her class is currently learning about recursive functions. Due to my lack of schooling this is the first time I have heard about them. So when she asked me to explain why the function she had worked I couldn't do it. I had to learn them myself.
I have been looking around at various posts about solving this same problem. I found one here at geeksforgeeks that is a function that does exactly what we need. With my elementary understanding of recursion this is the function that I would have thought would have been the right choice.
def bintodec(n):
if len(n) == 1:
bin_digit= int(n)
return bin_digit * 2**(len(n) - 1)
else:
bin_digit = int(n[0])
return bintodec(n[1:]) + bin_digit * 2**(len(n) - 1)
This is the function she came up with
def convertToDecimal(binNum):
if len(binNum) == 0:
return 0
else:
return convertToDecimal(binNum[:-1]) * 2 + int(binNum[-1])
When I print the function call it works.
print(convertToDecimal("11111111"))
# results in 255
print(convertToDecimal("00000111"))
# results in 7
I understand that sometimes there is a shorthand way to things. I can't see any shorthand methods mentions in the documentation that I have read.
The thing that really confuses me is how it takes that string and does math with it. I see the typecast for int, but the other side doesn't have it.
This is where everything falls apart and my brain starts melting. I am thinking there is a core mechanic of recursion that I am missing. Normally that is the case.
So along to figuring out why that works, I would love to know how this method would compare to say the method we found over at geeksforgeeks
What your friend has implemented is the typical implementation of Horner's method for polynomial evaluation. Here is the formula.
Now think of the binary number as a polynomial with a's equal to one or zero, and x equals to 2.
The thing that really confuses me is how it takes that string and does math with it. I see the typecast for int, but the other side doesn't have it.
The "other side" will take the value as int number which is result of latest recursive function call. in this case it will be 0.
Ok, so in words, what this program is doing is, on each invocation, taking the string and splitting it into 2 parts, lets call them a and b. a contains the entire string, apart from the final character, while b only contains the final digit.
Next, it takes a and calls the same function again, but this time with the shorter string, and then takes the result of this and doubles it. The doubling is done, as if you were to add an additional 0 to the end of a binary number, you would be doubling it.
Finally, it converts the value of b into an integer, either 1, or 0, and adds this to the previous result, which will be the decimal version of your binary string.
In other words, this function is only computing the result one character at a time, then it calls back to itself as a way of 'looping' to the next character.
It's important that there is an exit condition in a recursive function, to prevent infinite looping, in this case, when the string is empty, the program just returns 0, ending the loop.
Now on to the syntax. The only potentially confusing thing here I can see is python's array/slice syntax. Firstly, by trying to access the -1 index in an array, you are actually accessing the final element.
Also in that snippet is slice notation, which is the colon : in the array index. This is essentially used to select a subset of an array, in this case, all elements but the final one.
I honestly couldn’t make her function run as written. I got the below error
if len(binNum) == 0:
TypeError: object of type 'int' has no len()
I'm guessing however that under testing even working this would fail at some point, I’d like to see if you have it returning say, 221 (11011101) where the 1s and 0s are not consecutive and see if that works or fails.
Lastly, back to my error, I’m assuming the intention is to go out of the loop if it’s a zero. Even if zero wasn’t a null character, len(binNum) == 1 would still exit the loop as written. A try/catch block would be better
This is a first run-in with not only bitwise ops in python, but also strange (to me) syntax.
for i in range(2**len(set_)//2):
parts = [set(), set()]
for item in set_:
parts[i&1].add(item)
i >>= 1
For context, set_ is just a list of 4 letters.
There's a bit to unpack here. First, I've never seen [set(), set()]. I must be using the wrong keywords, as I couldn't find it in the docs. It looks like it creates a matrix in pythontutor, but I cannot say for certain. Second, while parts[i&1] is a slicing operation, I'm not entirely sure why a bitwise operation is required. For example, 0&1 should be 1 and 1&1 should be 0 (carry the one), so binary 10 (or 2 in decimal)? Finally, the last bitwise operation is completely bewildering. I believe a right shift is the same as dividing by two (I hope), but why i>>=1? I don't know how to interpret that. Any guidance would be sincerely appreciated.
[set(), set()] creates a list consisting of two empty sets.
0&1 is 0, 1&1 is 1. There is no carry in bitwise operations. parts[i&1] therefore refers to the first set when i is even, the second when i is odd.
i >>= 1 shifts right by one bit (which is indeed the same as dividing by two), then assigns the result back to i. It's the same basic concept as using i += 1 to increment a variable.
The effect of the inner loop is to partition the elements of _set into two subsets, based on the bits of i. If the limit in the outer loop had been simply 2 ** len(_set), the code would generate every possible such partitioning. But since that limit was divided by two, only half of the possible partitions get generated - I couldn't guess what the point of that might be, without more context.
I've never seen [set(), set()]
This isn't anything interesting, just a list with two new sets in it. So you have seen it, because it's not new syntax. Just a list and constructors.
parts[i&1]
This tests the least significant bit of i and selects either parts[0] (if the lsb was 0) or parts[1] (if the lsb was 1). Nothing fancy like slicing, just plain old indexing into a list. The thing you get out is a set, .add(item) does the obvious thing: adds something to whichever set was selected.
but why i>>=1? I don't know how to interpret that
Take the bits in i and move them one position to the right, dropping the old lsb, and keeping the sign. Sort of like this
Except of course that in Python you have arbitrary-precision integers, so it's however long it needs to be instead of 8 bits.
For positive numbers, the part about copying the sign is irrelevant.
You can think of right shift by 1 as a flooring division by 2 (this is different from truncation, negative numbers are rounded towards negative infinity, eg -1 >> 1 = -1), but that interpretation is usually more complicated to reason about.
Anyway, the way it is used here is just a way to loop through the bits of i, testing them one by one from low to high, but instead of changing which bit it tests it moves the bit it wants to test into the same position every time.
I know that something like the following
=IF(ISERROR(LONG_FORMULA), 0, LONG_FORMULA)
can be replaced with
=IFERROR(LONG_FORMULA, 0)
However I am looking for an expression to avoid having to type REALLY_LONG_FORMULA twice in
=IF(REALLY_LONG_FORMULA < threshold, 0, REALLY_LONG_FORMULA)
How can I do this?
I was able to come up with the following:
=IFERROR(EXP(LN(REALLY_LONG_FORMULA – threshold)) + threshold, 0)
It works by utilizing the fact that the log of a negative number produces an error and that EXP and LN are inverses of each other.
The biggest benefit of this is that it avoids accidentally introducing errors into your spreadsheet when you change something in one copy of REALLY_LONG_FORMULA without remembering to apply the same change to the other copy of REALLY_LONG_FORMULA in your IF statement.
Greater than comparisons as in
=IF(REALLY_LONG_FORMULA>=threshold,0,REALLY_LONG_FORMULA)
can be replaced with
=IFERROR(threshold-EXP(LN(threshold-REALLY_LONG_FORMULA)),0)
Example below (provided by #Jeeped):
For strict inequality comparisons use SQRT(_)^2 as pointed out by #Tom Sharpe.
If you're comparing against a threshold amount, I would consider checking out ExcelJet's recent blog post about
Replacing Ugly IFs with MAX() or MIN().
Also, the MAX() and MIN() functions are much more intuitive than using lessor known functions like EXP() and LN().
Comparing Ln Exp with SQRT ^2:-
because SQRT(0) gives 0 but ln(0) gives #NUM!
So you can choose which one to use depending whether you want the equality or not.
These also work for negative numbers - in theory.