Python: for loop skipping array elements [duplicate] - python-3.x
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria.
for tup in somelist:
if determine(tup):
code_to_remove_tup
What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.
You can use a list comprehension to create a new list containing only the elements you don't want to remove:
somelist = [x for x in somelist if not determine(x)]
Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want:
somelist[:] = [x for x in somelist if not determine(x)]
This approach could be useful if there are other references to somelist that need to reflect the changes.
Instead of a comprehension, you could also use itertools. In Python 2:
from itertools import ifilterfalse
somelist[:] = ifilterfalse(determine, somelist)
Or in Python 3:
from itertools import filterfalse
somelist[:] = filterfalse(determine, somelist)
The answers suggesting list comprehensions are almost correct—except that they build a completely new list and then give it the same name the old list as, they do not modify the old list in place. That's different from what you'd be doing by selective removal, as in Lennart's suggestion—it's faster, but if your list is accessed via multiple references the fact that you're just reseating one of the references and not altering the list object itself can lead to subtle, disastrous bugs.
Fortunately, it's extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration—just code:
somelist[:] = [tup for tup in somelist if determine(tup)]
Note the subtle difference with other answers: this one is not assigning to a barename. It's assigning to a list slice that just happens to be the entire list, thereby replacing the list contents within the same Python list object, rather than just reseating one reference (from the previous list object to the new list object) like the other answers.
You need to take a copy of the list and iterate over it first, or the iteration will fail with what may be unexpected results.
For example (depends on what type of list):
for tup in somelist[:]:
etc....
An example:
>>> somelist = range(10)
>>> for x in somelist:
... somelist.remove(x)
>>> somelist
[1, 3, 5, 7, 9]
>>> somelist = range(10)
>>> for x in somelist[:]:
... somelist.remove(x)
>>> somelist
[]
for i in range(len(somelist) - 1, -1, -1):
if some_condition(somelist, i):
del somelist[i]
You need to go backwards otherwise it's a bit like sawing off the tree-branch that you are sitting on :-)
Python 2 users: replace range by xrange to avoid creating a hardcoded list
Overview of workarounds
Either:
use a linked list implementation/roll your own.
A linked list is the proper data structure to support efficient item removal, and does not force you to make space/time tradeoffs.
A CPython list is implemented with dynamic arrays as mentioned here, which is not a good data type to support removals.
There doesn't seem to be a linked list in the standard library however:
Is there a linked list predefined library in Python?
https://github.com/ajakubek/python-llist
start a new list() from scratch, and .append() back at the end as mentioned at: https://stackoverflow.com/a/1207460/895245
This time efficient, but less space efficient because it keeps an extra copy of the array around during iteration.
use del with an index as mentioned at: https://stackoverflow.com/a/1207485/895245
This is more space efficient since it dispenses the array copy, but it is less time efficient, because removal from dynamic arrays requires shifting all following items back by one, which is O(N).
Generally, if you are doing it quick and dirty and don't want to add a custom LinkedList class, you just want to go for the faster .append() option by default unless memory is a big concern.
Official Python 2 tutorial 4.2. "for Statements"
https://docs.python.org/2/tutorial/controlflow.html#for-statements
This part of the docs makes it clear that:
you need to make a copy of the iterated list to modify it
one way to do it is with the slice notation [:]
If you need to modify the sequence you are iterating over while inside the loop (for example to duplicate selected items), it is recommended that you first make a copy. Iterating over a sequence does not implicitly make a copy. The slice notation makes this especially convenient:
>>> words = ['cat', 'window', 'defenestrate']
>>> for w in words[:]: # Loop over a slice copy of the entire list.
... if len(w) > 6:
... words.insert(0, w)
...
>>> words
['defenestrate', 'cat', 'window', 'defenestrate']
Python 2 documentation 7.3. "The for statement"
https://docs.python.org/2/reference/compound_stmts.html#for
This part of the docs says once again that you have to make a copy, and gives an actual removal example:
Note: There is a subtlety when the sequence is being modified by the loop (this can only occur for mutable sequences, i.e. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) item from the sequence, the next item will be skipped (since it gets the index of the current item which has already been treated). Likewise, if the suite inserts an item in the sequence before the current item, the current item will be treated again the next time through the loop. This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence, e.g.,
for x in a[:]:
if x < 0: a.remove(x)
However, I disagree with this implementation, since .remove() has to iterate the entire list to find the value.
Could Python do this better?
It seems like this particular Python API could be improved. Compare it, for instance, with:
Java ListIterator::remove which documents "This call can only be made once per call to next or previous"
C++ std::vector::erase which returns a valid interator to the element after the one removed
both of which make it crystal clear that you cannot modify a list being iterated except with the iterator itself, and gives you efficient ways to do so without copying the list.
Perhaps the underlying rationale is that Python lists are assumed to be dynamic array backed, and therefore any type of removal will be time inefficient anyways, while Java has a nicer interface hierarchy with both ArrayList and LinkedList implementations of ListIterator.
There doesn't seem to be an explicit linked list type in the Python stdlib either: Python Linked List
Your best approach for such an example would be a list comprehension
somelist = [tup for tup in somelist if determine(tup)]
In cases where you're doing something more complex than calling a determine function, I prefer constructing a new list and simply appending to it as I go. For example
newlist = []
for tup in somelist:
# lots of code here, possibly setting things up for calling determine
if determine(tup):
newlist.append(tup)
somelist = newlist
Copying the list using remove might make your code look a little cleaner, as described in one of the answers below. You should definitely not do this for extremely large lists, since this involves first copying the entire list, and also performing an O(n) remove operation for each element being removed, making this an O(n^2) algorithm.
for tup in somelist[:]:
# lots of code here, possibly setting things up for calling determine
if determine(tup):
newlist.append(tup)
For those who like functional programming:
somelist[:] = filter(lambda tup: not determine(tup), somelist)
or
from itertools import ifilterfalse
somelist[:] = list(ifilterfalse(determine, somelist))
I needed to do this with a huge list, and duplicating the list seemed expensive, especially since in my case the number of deletions would be few compared to the items that remain. I took this low-level approach.
array = [lots of stuff]
arraySize = len(array)
i = 0
while i < arraySize:
if someTest(array[i]):
del array[i]
arraySize -= 1
else:
i += 1
What I don't know is how efficient a couple of deletes are compared to copying a large list. Please comment if you have any insight.
Most of the answers here want you to create a copy of the list. I had a use case where the list was quite long (110K items) and it was smarter to keep reducing the list instead.
First of all you'll need to replace foreach loop with while loop,
i = 0
while i < len(somelist):
if determine(somelist[i]):
del somelist[i]
else:
i += 1
The value of i is not changed in the if block because you'll want to get value of the new item FROM THE SAME INDEX, once the old item is deleted.
It might be smart to also just create a new list if the current list item meets the desired criteria.
so:
for item in originalList:
if (item != badValue):
newList.append(item)
and to avoid having to re-code the entire project with the new lists name:
originalList[:] = newList
note, from Python documentation:
copy.copy(x)
Return a shallow copy of x.
copy.deepcopy(x)
Return a deep copy of x.
This answer was originally written in response to a question which has since been marked as duplicate:
Removing coordinates from list on python
There are two problems in your code:
1) When using remove(), you attempt to remove integers whereas you need to remove a tuple.
2) The for loop will skip items in your list.
Let's run through what happens when we execute your code:
>>> L1 = [(1,2), (5,6), (-1,-2), (1,-2)]
>>> for (a,b) in L1:
... if a < 0 or b < 0:
... L1.remove(a,b)
...
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
TypeError: remove() takes exactly one argument (2 given)
The first problem is that you are passing both 'a' and 'b' to remove(), but remove() only accepts a single argument. So how can we get remove() to work properly with your list? We need to figure out what each element of your list is. In this case, each one is a tuple. To see this, let's access one element of the list (indexing starts at 0):
>>> L1[1]
(5, 6)
>>> type(L1[1])
<type 'tuple'>
Aha! Each element of L1 is actually a tuple. So that's what we need to be passing to remove(). Tuples in python are very easy, they're simply made by enclosing values in parentheses. "a, b" is not a tuple, but "(a, b)" is a tuple. So we modify your code and run it again:
# The remove line now includes an extra "()" to make a tuple out of "a,b"
L1.remove((a,b))
This code runs without any error, but let's look at the list it outputs:
L1 is now: [(1, 2), (5, 6), (1, -2)]
Why is (1,-2) still in your list? It turns out modifying the list while using a loop to iterate over it is a very bad idea without special care. The reason that (1, -2) remains in the list is that the locations of each item within the list changed between iterations of the for loop. Let's look at what happens if we feed the above code a longer list:
L1 = [(1,2),(5,6),(-1,-2),(1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
### Outputs:
L1 is now: [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]
As you can infer from that result, every time that the conditional statement evaluates to true and a list item is removed, the next iteration of the loop will skip evaluation of the next item in the list because its values are now located at different indices.
The most intuitive solution is to copy the list, then iterate over the original list and only modify the copy. You can try doing so like this:
L2 = L1
for (a,b) in L1:
if a < 0 or b < 0 :
L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
print L2 is L1
del L1
L1 = L2; del L2
print ("L1 is now: ", L1)
However, the output will be identical to before:
'L1 is now: ', [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]
This is because when we created L2, python did not actually create a new object. Instead, it merely referenced L2 to the same object as L1. We can verify this with 'is' which is different from merely "equals" (==).
>>> L2=L1
>>> L1 is L2
True
We can make a true copy using copy.copy(). Then everything works as expected:
import copy
L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
L2 = copy.copy(L1)
for (a,b) in L1:
if a < 0 or b < 0 :
L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
del L1
L1 = L2; del L2
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]
Finally, there is one cleaner solution than having to make an entirely new copy of L1. The reversed() function:
L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
for (a,b) in reversed(L1):
if a < 0 or b < 0 :
L1.remove((a,b))
print ("L1 is now: ", L1)
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]
Unfortunately, I cannot adequately describe how reversed() works. It returns a 'listreverseiterator' object when a list is passed to it. For practical purposes, you can think of it as creating a reversed copy of its argument. This is the solution I recommend.
If you want to delete elements from a list while iterating, use a while-loop so you can alter the current index and end index after each deletion.
Example:
i = 0
length = len(list1)
while i < length:
if condition:
list1.remove(list1[i])
i -= 1
length -= 1
i += 1
The other answers are correct that it is usually a bad idea to delete from a list that you're iterating. Reverse iterating avoids some of the pitfalls, but it is much more difficult to follow code that does that, so usually you're better off using a list comprehension or filter.
There is, however, one case where it is safe to remove elements from a sequence that you are iterating: if you're only removing one item while you're iterating. This can be ensured using a return or a break. For example:
for i, item in enumerate(lst):
if item % 4 == 0:
foo(item)
del lst[i]
break
This is often easier to understand than a list comprehension when you're doing some operations with side effects on the first item in a list that meets some condition and then removing that item from the list immediately after.
If you want to do anything else during the iteration, it may be nice to get both the index (which guarantees you being able to reference it, for example if you have a list of dicts) and the actual list item contents.
inlist = [{'field1':10, 'field2':20}, {'field1':30, 'field2':15}]
for idx, i in enumerate(inlist):
do some stuff with i['field1']
if somecondition:
xlist.append(idx)
for i in reversed(xlist): del inlist[i]
enumerate gives you access to the item and the index at once. reversed is so that the indices that you're going to later delete don't change on you.
One possible solution, useful if you want not only remove some things, but also do something with all elements in a single loop:
alist = ['good', 'bad', 'good', 'bad', 'good']
i = 0
for x in alist[:]:
if x == 'bad':
alist.pop(i)
i -= 1
# do something cool with x or just print x
print(x)
i += 1
A for loop will be iterate through an index...
Consider you have a list,
[5, 7, 13, 29, 65, 91]
You have used a list variable called lis. And you use the same to remove...
Your variable
lis = [5, 7, 13, 29, 35, 65, 91]
0 1 2 3 4 5 6
during the 5th iteration,
Your number 35 was not a prime, so you removed it from a list.
lis.remove(y)
And then the next value (65) move on to the previous index.
lis = [5, 7, 13, 29, 65, 91]
0 1 2 3 4 5
so the 4th iteration done pointer moved onto the 5th...
That’s why your loop doesn’t cover 65 since it’s moved into the previous index.
So you shouldn't reference a list into another variable which still references the original instead of a copy.
ite = lis # Don’t do it will reference instead copy
So do a copy of the list using list[::].
Now you will give,
[5, 7, 13, 29]
The problem is you removed a value from a list during iteration and then your list index will collapse.
So you can try list comprehension instead.
Which supports all the iterable like, list, tuple, dict, string, etc.
You might want to use filter() available as the built-in.
For more details check here
You can try for-looping in reverse so for some_list you'll do something like:
list_len = len(some_list)
for i in range(list_len):
reverse_i = list_len - 1 - i
cur = some_list[reverse_i]
# some logic with cur element
if some_condition:
some_list.pop(reverse_i)
This way the index is aligned and doesn't suffer from the list updates (regardless whether you pop cur element or not).
I needed to do something similar and in my case the problem was memory - I needed to merge multiple dataset objects within a list, after doing some stuff with them, as a new object, and needed to get rid of each entry I was merging to avoid duplicating all of them and blowing up memory. In my case having the objects in a dictionary instead of a list worked fine:
```
k = range(5)
v = ['a','b','c','d','e']
d = {key:val for key,val in zip(k, v)}
print d
for i in range(5):
print d[i]
d.pop(i)
print d
```
The most effective method is list comprehension, many people show their case, of course, it is also a good way to get an iterator through filter.
Filter receives a function and a sequence. Filter applies the passed function to each element in turn, and then decides whether to retain or discard the element depending on whether the function return value is True or False.
There is an example (get the odds in the tuple):
list(filter(lambda x:x%2==1, (1, 2, 4, 5, 6, 9, 10, 15)))
# result: [1, 5, 9, 15]
Caution: You can also not handle iterators. Iterators are sometimes better than sequences.
TLDR:
I wrote a library that allows you to do this:
from fluidIter import FluidIterable
fSomeList = FluidIterable(someList)
for tup in fSomeList:
if determine(tup):
# remove 'tup' without "breaking" the iteration
fSomeList.remove(tup)
# tup has also been removed from 'someList'
# as well as 'fSomeList'
It's best to use another method if possible that doesn't require modifying your iterable while iterating over it, but for some algorithms it might not be that straight forward. And so if you are sure that you really do want the code pattern described in the original question, it is possible.
Should work on all mutable sequences not just lists.
Full answer:
Edit: The last code example in this answer gives a use case for why you might sometimes want to modify a list in place rather than use a list comprehension. The first part of the answers serves as tutorial of how an array can be modified in place.
The solution follows on from this answer (for a related question) from senderle. Which explains how the the array index is updated while iterating through a list that has been modified. The solution below is designed to correctly track the array index even if the list is modified.
Download fluidIter.py from here https://github.com/alanbacon/FluidIterator, it is just a single file so no need to install git. There is no installer so you will need to make sure that the file is in the python path your self. The code has been written for python 3 and is untested on python 2.
from fluidIter import FluidIterable
l = [0,1,2,3,4,5,6,7,8]
fluidL = FluidIterable(l)
for i in fluidL:
print('initial state of list on this iteration: ' + str(fluidL))
print('current iteration value: ' + str(i))
print('popped value: ' + str(fluidL.pop(2)))
print(' ')
print('Final List Value: ' + str(l))
This will produce the following output:
initial state of list on this iteration: [0, 1, 2, 3, 4, 5, 6, 7, 8]
current iteration value: 0
popped value: 2
initial state of list on this iteration: [0, 1, 3, 4, 5, 6, 7, 8]
current iteration value: 1
popped value: 3
initial state of list on this iteration: [0, 1, 4, 5, 6, 7, 8]
current iteration value: 4
popped value: 4
initial state of list on this iteration: [0, 1, 5, 6, 7, 8]
current iteration value: 5
popped value: 5
initial state of list on this iteration: [0, 1, 6, 7, 8]
current iteration value: 6
popped value: 6
initial state of list on this iteration: [0, 1, 7, 8]
current iteration value: 7
popped value: 7
initial state of list on this iteration: [0, 1, 8]
current iteration value: 8
popped value: 8
Final List Value: [0, 1]
Above we have used the pop method on the fluid list object. Other common iterable methods are also implemented such as del fluidL[i], .remove, .insert, .append, .extend. The list can also be modified using slices (sort and reverse methods are not implemented).
The only condition is that you must only modify the list in place, if at any point fluidL or l were reassigned to a different list object the code would not work. The original fluidL object would still be used by the for loop but would become out of scope for us to modify.
i.e.
fluidL[2] = 'a' # is OK
fluidL = [0, 1, 'a', 3, 4, 5, 6, 7, 8] # is not OK
If we want to access the current index value of the list we cannot use enumerate, as this only counts how many times the for loop has run. Instead we will use the iterator object directly.
fluidArr = FluidIterable([0,1,2,3])
# get iterator first so can query the current index
fluidArrIter = fluidArr.__iter__()
for i, v in enumerate(fluidArrIter):
print('enum: ', i)
print('current val: ', v)
print('current ind: ', fluidArrIter.currentIndex)
print(fluidArr)
fluidArr.insert(0,'a')
print(' ')
print('Final List Value: ' + str(fluidArr))
This will output the following:
enum: 0
current val: 0
current ind: 0
[0, 1, 2, 3]
enum: 1
current val: 1
current ind: 2
['a', 0, 1, 2, 3]
enum: 2
current val: 2
current ind: 4
['a', 'a', 0, 1, 2, 3]
enum: 3
current val: 3
current ind: 6
['a', 'a', 'a', 0, 1, 2, 3]
Final List Value: ['a', 'a', 'a', 'a', 0, 1, 2, 3]
The FluidIterable class just provides a wrapper for the original list object. The original object can be accessed as a property of the fluid object like so:
originalList = fluidArr.fixedIterable
More examples / tests can be found in the if __name__ is "__main__": section at the bottom of fluidIter.py. These are worth looking at because they explain what happens in various situations. Such as: Replacing a large sections of the list using a slice. Or using (and modifying) the same iterable in nested for loops.
As I stated to start with: this is a complicated solution that will hurt the readability of your code and make it more difficult to debug. Therefore other solutions such as the list comprehensions mentioned in David Raznick's answer should be considered first. That being said, I have found times where this class has been useful to me and has been easier to use than keeping track of the indices of elements that need deleting.
Edit: As mentioned in the comments, this answer does not really present a problem for which this approach provides a solution. I will try to address that here:
List comprehensions provide a way to generate a new list but these approaches tend to look at each element in isolation rather than the current state of the list as a whole.
i.e.
newList = [i for i in oldList if testFunc(i)]
But what if the result of the testFunc depends on the elements that have been added to newList already? Or the elements still in oldList that might be added next? There might still be a way to use a list comprehension but it will begin to lose it's elegance, and for me it feels easier to modify a list in place.
The code below is one example of an algorithm that suffers from the above problem. The algorithm will reduce a list so that no element is a multiple of any other element.
randInts = [70, 20, 61, 80, 54, 18, 7, 18, 55, 9]
fRandInts = FluidIterable(randInts)
fRandIntsIter = fRandInts.__iter__()
# for each value in the list (outer loop)
# test against every other value in the list (inner loop)
for i in fRandIntsIter:
print(' ')
print('outer val: ', i)
innerIntsIter = fRandInts.__iter__()
for j in innerIntsIter:
innerIndex = innerIntsIter.currentIndex
# skip the element that the outloop is currently on
# because we don't want to test a value against itself
if not innerIndex == fRandIntsIter.currentIndex:
# if the test element, j, is a multiple
# of the reference element, i, then remove 'j'
if j%i == 0:
print('remove val: ', j)
# remove element in place, without breaking the
# iteration of either loop
del fRandInts[innerIndex]
# end if multiple, then remove
# end if not the same value as outer loop
# end inner loop
# end outerloop
print('')
print('final list: ', randInts)
The output and the final reduced list are shown below
outer val: 70
outer val: 20
remove val: 80
outer val: 61
outer val: 54
outer val: 18
remove val: 54
remove val: 18
outer val: 7
remove val: 70
outer val: 55
outer val: 9
remove val: 18
final list: [20, 61, 7, 55, 9]
For anything that has the potential to be really big, I use the following.
import numpy as np
orig_list = np.array([1, 2, 3, 4, 5, 100, 8, 13])
remove_me = [100, 1]
cleaned = np.delete(orig_list, remove_me)
print(cleaned)
That should be significantly faster than anything else.
In some situations, where you're doing more than simply filtering a list one item at time, you want your iteration to change while iterating.
Here is an example where copying the list beforehand is incorrect, reverse iteration is impossible and a list comprehension is also not an option.
""" Sieve of Eratosthenes """
def generate_primes(n):
""" Generates all primes less than n. """
primes = list(range(2,n))
idx = 0
while idx < len(primes):
p = primes[idx]
for multiple in range(p+p, n, p):
try:
primes.remove(multiple)
except ValueError:
pass #EAFP
idx += 1
yield p
I can think of three approaches to solve your problem. As an example, I will create a random list of tuples somelist = [(1,2,3), (4,5,6), (3,6,6), (7,8,9), (15,0,0), (10,11,12)]. The condition that I choose is sum of elements of a tuple = 15. In the final list we will only have those tuples whose sum is not equal to 15.
What I have chosen is a randomly chosen example. Feel free to change the list of tuples and the condition that I have chosen.
Method 1.> Use the framework that you had suggested (where one fills in a code inside a for loop). I use a small code with del to delete a tuple that meets the said condition. However, this method will miss a tuple (which satisfies the said condition) if two consecutively placed tuples meet the given condition.
for tup in somelist:
if ( sum(tup)==15 ):
del somelist[somelist.index(tup)]
print somelist
>>> [(1, 2, 3), (3, 6, 6), (7, 8, 9), (10, 11, 12)]
Method 2.> Construct a new list which contains elements (tuples) where the given condition is not met (this is the same thing as removing elements of list where the given condition is met). Following is the code for that:
newlist1 = [somelist[tup] for tup in range(len(somelist)) if(sum(somelist[tup])!=15)]
print newlist1
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]
Method 3.> Find indices where the given condition is met, and then use remove elements (tuples) corresponding to those indices. Following is the code for that.
indices = [i for i in range(len(somelist)) if(sum(somelist[i])==15)]
newlist2 = [tup for j, tup in enumerate(somelist) if j not in indices]
print newlist2
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]
Method 1 and method 2 are faster than method 3. Method2 and method3 are more efficient than method1. I prefer method2. For the aforementioned example, time(method1) : time(method2) : time(method3) = 1 : 1 : 1.7
If you will use the new list later, you can simply set the elem to None, and then judge it in the later loop, like this
for i in li:
i = None
for elem in li:
if elem is None:
continue
In this way, you dont't need copy the list and it's easier to understand.
Related
Why does my For Loop skip over elements in my list? [duplicate]
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions. I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria. for tup in somelist: if determine(tup): code_to_remove_tup What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.
You can use a list comprehension to create a new list containing only the elements you don't want to remove: somelist = [x for x in somelist if not determine(x)] Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want: somelist[:] = [x for x in somelist if not determine(x)] This approach could be useful if there are other references to somelist that need to reflect the changes. Instead of a comprehension, you could also use itertools. In Python 2: from itertools import ifilterfalse somelist[:] = ifilterfalse(determine, somelist) Or in Python 3: from itertools import filterfalse somelist[:] = filterfalse(determine, somelist)
The answers suggesting list comprehensions are almost correct—except that they build a completely new list and then give it the same name the old list as, they do not modify the old list in place. That's different from what you'd be doing by selective removal, as in Lennart's suggestion—it's faster, but if your list is accessed via multiple references the fact that you're just reseating one of the references and not altering the list object itself can lead to subtle, disastrous bugs. Fortunately, it's extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration—just code: somelist[:] = [tup for tup in somelist if determine(tup)] Note the subtle difference with other answers: this one is not assigning to a barename. It's assigning to a list slice that just happens to be the entire list, thereby replacing the list contents within the same Python list object, rather than just reseating one reference (from the previous list object to the new list object) like the other answers.
You need to take a copy of the list and iterate over it first, or the iteration will fail with what may be unexpected results. For example (depends on what type of list): for tup in somelist[:]: etc.... An example: >>> somelist = range(10) >>> for x in somelist: ... somelist.remove(x) >>> somelist [1, 3, 5, 7, 9] >>> somelist = range(10) >>> for x in somelist[:]: ... somelist.remove(x) >>> somelist []
for i in range(len(somelist) - 1, -1, -1): if some_condition(somelist, i): del somelist[i] You need to go backwards otherwise it's a bit like sawing off the tree-branch that you are sitting on :-) Python 2 users: replace range by xrange to avoid creating a hardcoded list
Overview of workarounds Either: use a linked list implementation/roll your own. A linked list is the proper data structure to support efficient item removal, and does not force you to make space/time tradeoffs. A CPython list is implemented with dynamic arrays as mentioned here, which is not a good data type to support removals. There doesn't seem to be a linked list in the standard library however: Is there a linked list predefined library in Python? https://github.com/ajakubek/python-llist start a new list() from scratch, and .append() back at the end as mentioned at: https://stackoverflow.com/a/1207460/895245 This time efficient, but less space efficient because it keeps an extra copy of the array around during iteration. use del with an index as mentioned at: https://stackoverflow.com/a/1207485/895245 This is more space efficient since it dispenses the array copy, but it is less time efficient, because removal from dynamic arrays requires shifting all following items back by one, which is O(N). Generally, if you are doing it quick and dirty and don't want to add a custom LinkedList class, you just want to go for the faster .append() option by default unless memory is a big concern. Official Python 2 tutorial 4.2. "for Statements" https://docs.python.org/2/tutorial/controlflow.html#for-statements This part of the docs makes it clear that: you need to make a copy of the iterated list to modify it one way to do it is with the slice notation [:] If you need to modify the sequence you are iterating over while inside the loop (for example to duplicate selected items), it is recommended that you first make a copy. Iterating over a sequence does not implicitly make a copy. The slice notation makes this especially convenient: >>> words = ['cat', 'window', 'defenestrate'] >>> for w in words[:]: # Loop over a slice copy of the entire list. ... if len(w) > 6: ... words.insert(0, w) ... >>> words ['defenestrate', 'cat', 'window', 'defenestrate'] Python 2 documentation 7.3. "The for statement" https://docs.python.org/2/reference/compound_stmts.html#for This part of the docs says once again that you have to make a copy, and gives an actual removal example: Note: There is a subtlety when the sequence is being modified by the loop (this can only occur for mutable sequences, i.e. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) item from the sequence, the next item will be skipped (since it gets the index of the current item which has already been treated). Likewise, if the suite inserts an item in the sequence before the current item, the current item will be treated again the next time through the loop. This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence, e.g., for x in a[:]: if x < 0: a.remove(x) However, I disagree with this implementation, since .remove() has to iterate the entire list to find the value. Could Python do this better? It seems like this particular Python API could be improved. Compare it, for instance, with: Java ListIterator::remove which documents "This call can only be made once per call to next or previous" C++ std::vector::erase which returns a valid interator to the element after the one removed both of which make it crystal clear that you cannot modify a list being iterated except with the iterator itself, and gives you efficient ways to do so without copying the list. Perhaps the underlying rationale is that Python lists are assumed to be dynamic array backed, and therefore any type of removal will be time inefficient anyways, while Java has a nicer interface hierarchy with both ArrayList and LinkedList implementations of ListIterator. There doesn't seem to be an explicit linked list type in the Python stdlib either: Python Linked List
Your best approach for such an example would be a list comprehension somelist = [tup for tup in somelist if determine(tup)] In cases where you're doing something more complex than calling a determine function, I prefer constructing a new list and simply appending to it as I go. For example newlist = [] for tup in somelist: # lots of code here, possibly setting things up for calling determine if determine(tup): newlist.append(tup) somelist = newlist Copying the list using remove might make your code look a little cleaner, as described in one of the answers below. You should definitely not do this for extremely large lists, since this involves first copying the entire list, and also performing an O(n) remove operation for each element being removed, making this an O(n^2) algorithm. for tup in somelist[:]: # lots of code here, possibly setting things up for calling determine if determine(tup): newlist.append(tup)
For those who like functional programming: somelist[:] = filter(lambda tup: not determine(tup), somelist) or from itertools import ifilterfalse somelist[:] = list(ifilterfalse(determine, somelist))
I needed to do this with a huge list, and duplicating the list seemed expensive, especially since in my case the number of deletions would be few compared to the items that remain. I took this low-level approach. array = [lots of stuff] arraySize = len(array) i = 0 while i < arraySize: if someTest(array[i]): del array[i] arraySize -= 1 else: i += 1 What I don't know is how efficient a couple of deletes are compared to copying a large list. Please comment if you have any insight.
Most of the answers here want you to create a copy of the list. I had a use case where the list was quite long (110K items) and it was smarter to keep reducing the list instead. First of all you'll need to replace foreach loop with while loop, i = 0 while i < len(somelist): if determine(somelist[i]): del somelist[i] else: i += 1 The value of i is not changed in the if block because you'll want to get value of the new item FROM THE SAME INDEX, once the old item is deleted.
It might be smart to also just create a new list if the current list item meets the desired criteria. so: for item in originalList: if (item != badValue): newList.append(item) and to avoid having to re-code the entire project with the new lists name: originalList[:] = newList note, from Python documentation: copy.copy(x) Return a shallow copy of x. copy.deepcopy(x) Return a deep copy of x.
This answer was originally written in response to a question which has since been marked as duplicate: Removing coordinates from list on python There are two problems in your code: 1) When using remove(), you attempt to remove integers whereas you need to remove a tuple. 2) The for loop will skip items in your list. Let's run through what happens when we execute your code: >>> L1 = [(1,2), (5,6), (-1,-2), (1,-2)] >>> for (a,b) in L1: ... if a < 0 or b < 0: ... L1.remove(a,b) ... Traceback (most recent call last): File "<stdin>", line 3, in <module> TypeError: remove() takes exactly one argument (2 given) The first problem is that you are passing both 'a' and 'b' to remove(), but remove() only accepts a single argument. So how can we get remove() to work properly with your list? We need to figure out what each element of your list is. In this case, each one is a tuple. To see this, let's access one element of the list (indexing starts at 0): >>> L1[1] (5, 6) >>> type(L1[1]) <type 'tuple'> Aha! Each element of L1 is actually a tuple. So that's what we need to be passing to remove(). Tuples in python are very easy, they're simply made by enclosing values in parentheses. "a, b" is not a tuple, but "(a, b)" is a tuple. So we modify your code and run it again: # The remove line now includes an extra "()" to make a tuple out of "a,b" L1.remove((a,b)) This code runs without any error, but let's look at the list it outputs: L1 is now: [(1, 2), (5, 6), (1, -2)] Why is (1,-2) still in your list? It turns out modifying the list while using a loop to iterate over it is a very bad idea without special care. The reason that (1, -2) remains in the list is that the locations of each item within the list changed between iterations of the for loop. Let's look at what happens if we feed the above code a longer list: L1 = [(1,2),(5,6),(-1,-2),(1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)] ### Outputs: L1 is now: [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)] As you can infer from that result, every time that the conditional statement evaluates to true and a list item is removed, the next iteration of the loop will skip evaluation of the next item in the list because its values are now located at different indices. The most intuitive solution is to copy the list, then iterate over the original list and only modify the copy. You can try doing so like this: L2 = L1 for (a,b) in L1: if a < 0 or b < 0 : L2.remove((a,b)) # Now, remove the original copy of L1 and replace with L2 print L2 is L1 del L1 L1 = L2; del L2 print ("L1 is now: ", L1) However, the output will be identical to before: 'L1 is now: ', [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)] This is because when we created L2, python did not actually create a new object. Instead, it merely referenced L2 to the same object as L1. We can verify this with 'is' which is different from merely "equals" (==). >>> L2=L1 >>> L1 is L2 True We can make a true copy using copy.copy(). Then everything works as expected: import copy L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)] L2 = copy.copy(L1) for (a,b) in L1: if a < 0 or b < 0 : L2.remove((a,b)) # Now, remove the original copy of L1 and replace with L2 del L1 L1 = L2; del L2 >>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)] Finally, there is one cleaner solution than having to make an entirely new copy of L1. The reversed() function: L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)] for (a,b) in reversed(L1): if a < 0 or b < 0 : L1.remove((a,b)) print ("L1 is now: ", L1) >>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)] Unfortunately, I cannot adequately describe how reversed() works. It returns a 'listreverseiterator' object when a list is passed to it. For practical purposes, you can think of it as creating a reversed copy of its argument. This is the solution I recommend.
If you want to delete elements from a list while iterating, use a while-loop so you can alter the current index and end index after each deletion. Example: i = 0 length = len(list1) while i < length: if condition: list1.remove(list1[i]) i -= 1 length -= 1 i += 1
The other answers are correct that it is usually a bad idea to delete from a list that you're iterating. Reverse iterating avoids some of the pitfalls, but it is much more difficult to follow code that does that, so usually you're better off using a list comprehension or filter. There is, however, one case where it is safe to remove elements from a sequence that you are iterating: if you're only removing one item while you're iterating. This can be ensured using a return or a break. For example: for i, item in enumerate(lst): if item % 4 == 0: foo(item) del lst[i] break This is often easier to understand than a list comprehension when you're doing some operations with side effects on the first item in a list that meets some condition and then removing that item from the list immediately after.
If you want to do anything else during the iteration, it may be nice to get both the index (which guarantees you being able to reference it, for example if you have a list of dicts) and the actual list item contents. inlist = [{'field1':10, 'field2':20}, {'field1':30, 'field2':15}] for idx, i in enumerate(inlist): do some stuff with i['field1'] if somecondition: xlist.append(idx) for i in reversed(xlist): del inlist[i] enumerate gives you access to the item and the index at once. reversed is so that the indices that you're going to later delete don't change on you.
One possible solution, useful if you want not only remove some things, but also do something with all elements in a single loop: alist = ['good', 'bad', 'good', 'bad', 'good'] i = 0 for x in alist[:]: if x == 'bad': alist.pop(i) i -= 1 # do something cool with x or just print x print(x) i += 1
A for loop will be iterate through an index... Consider you have a list, [5, 7, 13, 29, 65, 91] You have used a list variable called lis. And you use the same to remove... Your variable lis = [5, 7, 13, 29, 35, 65, 91] 0 1 2 3 4 5 6 during the 5th iteration, Your number 35 was not a prime, so you removed it from a list. lis.remove(y) And then the next value (65) move on to the previous index. lis = [5, 7, 13, 29, 65, 91] 0 1 2 3 4 5 so the 4th iteration done pointer moved onto the 5th... That’s why your loop doesn’t cover 65 since it’s moved into the previous index. So you shouldn't reference a list into another variable which still references the original instead of a copy. ite = lis # Don’t do it will reference instead copy So do a copy of the list using list[::]. Now you will give, [5, 7, 13, 29] The problem is you removed a value from a list during iteration and then your list index will collapse. So you can try list comprehension instead. Which supports all the iterable like, list, tuple, dict, string, etc.
You might want to use filter() available as the built-in. For more details check here
You can try for-looping in reverse so for some_list you'll do something like: list_len = len(some_list) for i in range(list_len): reverse_i = list_len - 1 - i cur = some_list[reverse_i] # some logic with cur element if some_condition: some_list.pop(reverse_i) This way the index is aligned and doesn't suffer from the list updates (regardless whether you pop cur element or not).
I needed to do something similar and in my case the problem was memory - I needed to merge multiple dataset objects within a list, after doing some stuff with them, as a new object, and needed to get rid of each entry I was merging to avoid duplicating all of them and blowing up memory. In my case having the objects in a dictionary instead of a list worked fine: ``` k = range(5) v = ['a','b','c','d','e'] d = {key:val for key,val in zip(k, v)} print d for i in range(5): print d[i] d.pop(i) print d ```
The most effective method is list comprehension, many people show their case, of course, it is also a good way to get an iterator through filter. Filter receives a function and a sequence. Filter applies the passed function to each element in turn, and then decides whether to retain or discard the element depending on whether the function return value is True or False. There is an example (get the odds in the tuple): list(filter(lambda x:x%2==1, (1, 2, 4, 5, 6, 9, 10, 15))) # result: [1, 5, 9, 15] Caution: You can also not handle iterators. Iterators are sometimes better than sequences.
TLDR: I wrote a library that allows you to do this: from fluidIter import FluidIterable fSomeList = FluidIterable(someList) for tup in fSomeList: if determine(tup): # remove 'tup' without "breaking" the iteration fSomeList.remove(tup) # tup has also been removed from 'someList' # as well as 'fSomeList' It's best to use another method if possible that doesn't require modifying your iterable while iterating over it, but for some algorithms it might not be that straight forward. And so if you are sure that you really do want the code pattern described in the original question, it is possible. Should work on all mutable sequences not just lists. Full answer: Edit: The last code example in this answer gives a use case for why you might sometimes want to modify a list in place rather than use a list comprehension. The first part of the answers serves as tutorial of how an array can be modified in place. The solution follows on from this answer (for a related question) from senderle. Which explains how the the array index is updated while iterating through a list that has been modified. The solution below is designed to correctly track the array index even if the list is modified. Download fluidIter.py from here https://github.com/alanbacon/FluidIterator, it is just a single file so no need to install git. There is no installer so you will need to make sure that the file is in the python path your self. The code has been written for python 3 and is untested on python 2. from fluidIter import FluidIterable l = [0,1,2,3,4,5,6,7,8] fluidL = FluidIterable(l) for i in fluidL: print('initial state of list on this iteration: ' + str(fluidL)) print('current iteration value: ' + str(i)) print('popped value: ' + str(fluidL.pop(2))) print(' ') print('Final List Value: ' + str(l)) This will produce the following output: initial state of list on this iteration: [0, 1, 2, 3, 4, 5, 6, 7, 8] current iteration value: 0 popped value: 2 initial state of list on this iteration: [0, 1, 3, 4, 5, 6, 7, 8] current iteration value: 1 popped value: 3 initial state of list on this iteration: [0, 1, 4, 5, 6, 7, 8] current iteration value: 4 popped value: 4 initial state of list on this iteration: [0, 1, 5, 6, 7, 8] current iteration value: 5 popped value: 5 initial state of list on this iteration: [0, 1, 6, 7, 8] current iteration value: 6 popped value: 6 initial state of list on this iteration: [0, 1, 7, 8] current iteration value: 7 popped value: 7 initial state of list on this iteration: [0, 1, 8] current iteration value: 8 popped value: 8 Final List Value: [0, 1] Above we have used the pop method on the fluid list object. Other common iterable methods are also implemented such as del fluidL[i], .remove, .insert, .append, .extend. The list can also be modified using slices (sort and reverse methods are not implemented). The only condition is that you must only modify the list in place, if at any point fluidL or l were reassigned to a different list object the code would not work. The original fluidL object would still be used by the for loop but would become out of scope for us to modify. i.e. fluidL[2] = 'a' # is OK fluidL = [0, 1, 'a', 3, 4, 5, 6, 7, 8] # is not OK If we want to access the current index value of the list we cannot use enumerate, as this only counts how many times the for loop has run. Instead we will use the iterator object directly. fluidArr = FluidIterable([0,1,2,3]) # get iterator first so can query the current index fluidArrIter = fluidArr.__iter__() for i, v in enumerate(fluidArrIter): print('enum: ', i) print('current val: ', v) print('current ind: ', fluidArrIter.currentIndex) print(fluidArr) fluidArr.insert(0,'a') print(' ') print('Final List Value: ' + str(fluidArr)) This will output the following: enum: 0 current val: 0 current ind: 0 [0, 1, 2, 3] enum: 1 current val: 1 current ind: 2 ['a', 0, 1, 2, 3] enum: 2 current val: 2 current ind: 4 ['a', 'a', 0, 1, 2, 3] enum: 3 current val: 3 current ind: 6 ['a', 'a', 'a', 0, 1, 2, 3] Final List Value: ['a', 'a', 'a', 'a', 0, 1, 2, 3] The FluidIterable class just provides a wrapper for the original list object. The original object can be accessed as a property of the fluid object like so: originalList = fluidArr.fixedIterable More examples / tests can be found in the if __name__ is "__main__": section at the bottom of fluidIter.py. These are worth looking at because they explain what happens in various situations. Such as: Replacing a large sections of the list using a slice. Or using (and modifying) the same iterable in nested for loops. As I stated to start with: this is a complicated solution that will hurt the readability of your code and make it more difficult to debug. Therefore other solutions such as the list comprehensions mentioned in David Raznick's answer should be considered first. That being said, I have found times where this class has been useful to me and has been easier to use than keeping track of the indices of elements that need deleting. Edit: As mentioned in the comments, this answer does not really present a problem for which this approach provides a solution. I will try to address that here: List comprehensions provide a way to generate a new list but these approaches tend to look at each element in isolation rather than the current state of the list as a whole. i.e. newList = [i for i in oldList if testFunc(i)] But what if the result of the testFunc depends on the elements that have been added to newList already? Or the elements still in oldList that might be added next? There might still be a way to use a list comprehension but it will begin to lose it's elegance, and for me it feels easier to modify a list in place. The code below is one example of an algorithm that suffers from the above problem. The algorithm will reduce a list so that no element is a multiple of any other element. randInts = [70, 20, 61, 80, 54, 18, 7, 18, 55, 9] fRandInts = FluidIterable(randInts) fRandIntsIter = fRandInts.__iter__() # for each value in the list (outer loop) # test against every other value in the list (inner loop) for i in fRandIntsIter: print(' ') print('outer val: ', i) innerIntsIter = fRandInts.__iter__() for j in innerIntsIter: innerIndex = innerIntsIter.currentIndex # skip the element that the outloop is currently on # because we don't want to test a value against itself if not innerIndex == fRandIntsIter.currentIndex: # if the test element, j, is a multiple # of the reference element, i, then remove 'j' if j%i == 0: print('remove val: ', j) # remove element in place, without breaking the # iteration of either loop del fRandInts[innerIndex] # end if multiple, then remove # end if not the same value as outer loop # end inner loop # end outerloop print('') print('final list: ', randInts) The output and the final reduced list are shown below outer val: 70 outer val: 20 remove val: 80 outer val: 61 outer val: 54 outer val: 18 remove val: 54 remove val: 18 outer val: 7 remove val: 70 outer val: 55 outer val: 9 remove val: 18 final list: [20, 61, 7, 55, 9]
For anything that has the potential to be really big, I use the following. import numpy as np orig_list = np.array([1, 2, 3, 4, 5, 100, 8, 13]) remove_me = [100, 1] cleaned = np.delete(orig_list, remove_me) print(cleaned) That should be significantly faster than anything else.
In some situations, where you're doing more than simply filtering a list one item at time, you want your iteration to change while iterating. Here is an example where copying the list beforehand is incorrect, reverse iteration is impossible and a list comprehension is also not an option. """ Sieve of Eratosthenes """ def generate_primes(n): """ Generates all primes less than n. """ primes = list(range(2,n)) idx = 0 while idx < len(primes): p = primes[idx] for multiple in range(p+p, n, p): try: primes.remove(multiple) except ValueError: pass #EAFP idx += 1 yield p
I can think of three approaches to solve your problem. As an example, I will create a random list of tuples somelist = [(1,2,3), (4,5,6), (3,6,6), (7,8,9), (15,0,0), (10,11,12)]. The condition that I choose is sum of elements of a tuple = 15. In the final list we will only have those tuples whose sum is not equal to 15. What I have chosen is a randomly chosen example. Feel free to change the list of tuples and the condition that I have chosen. Method 1.> Use the framework that you had suggested (where one fills in a code inside a for loop). I use a small code with del to delete a tuple that meets the said condition. However, this method will miss a tuple (which satisfies the said condition) if two consecutively placed tuples meet the given condition. for tup in somelist: if ( sum(tup)==15 ): del somelist[somelist.index(tup)] print somelist >>> [(1, 2, 3), (3, 6, 6), (7, 8, 9), (10, 11, 12)] Method 2.> Construct a new list which contains elements (tuples) where the given condition is not met (this is the same thing as removing elements of list where the given condition is met). Following is the code for that: newlist1 = [somelist[tup] for tup in range(len(somelist)) if(sum(somelist[tup])!=15)] print newlist1 >>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)] Method 3.> Find indices where the given condition is met, and then use remove elements (tuples) corresponding to those indices. Following is the code for that. indices = [i for i in range(len(somelist)) if(sum(somelist[i])==15)] newlist2 = [tup for j, tup in enumerate(somelist) if j not in indices] print newlist2 >>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)] Method 1 and method 2 are faster than method 3. Method2 and method3 are more efficient than method1. I prefer method2. For the aforementioned example, time(method1) : time(method2) : time(method3) = 1 : 1 : 1.7
If you will use the new list later, you can simply set the elem to None, and then judge it in the later loop, like this for i in li: i = None for elem in li: if elem is None: continue In this way, you dont't need copy the list and it's easier to understand.
Getting rid of duplicates from a pair of corresponding lists
This is a program that I recently made. The goal of this code is to a pair of corresponding lists. So randomStringpt1[0] corresponds to randomStringpt2[0]. I want to compare randomStringpt1[0] and randomString2[0] to the rest of the pairs that the user gave in the randomStrings. But after using this code, it looks like I have duplicated each pair many times, which is the opposite of what I was looking for. I was thinking of using a dictionary, but then realized that a dictionary key could only have one value, which wouldn't help my case if the user used a number twice. Does anyone know how I can reduce the duplicates? (The tests I have been running have been with the numbers randomStringpt1 = [1,3,1,1,3] and randomStringpy2 = [2,4,2,3,4] ) randomStringpt1 = [1, 2, 3, 4, 5] #Pair of strings that correspond to each other("1,2,3,4,5" doesn't actually matter) randomStringpt2 = [1, 2, 3, 4, 5] for i in range(len(randomStringpt1)): randomStringpt1[i] = input("Values for the first string: ") randomStringpt2[i] = input("Corresponding value for the second string: ") print(randomStringpt1) #numbers that the user chose for the first number of the pair print(randomStringpt2) #numbers that the user chose for the second number of the pair newStart = [] newEnd = [] for num1 in range(len(randomStringpt1)): for num2 in range(len(randomStringpt1)): if (int(randomStringpt1[num1]) != int(randomStringpt1[num2]) and int(randomStringpt2[num1]) != int(randomStringpt2[num2])): newStart.append(randomStringpt1[num1]) # Adding the pairs that aren't equal to each other to a new list newEnd.append(randomStringpt2[num1]) newStart.append(randomStringpt1[num2]) newEnd.append(randomStringpt2[num2]) # else: # print("The set of numbers from the randomStrings of num1 are not equal to the ones in num2") print(newStart) print(newEnd)
First let's analyze the 2 bugs in your code, the if condition inside the loop is true every time a pair compares to a different one. this means for your example it should output [1, 1, 3, 3, 3, 1, 1, 1, 1, 3, 3, 3] [2, 2, 4, 4, 4, 2, 2, 3, 3, 4, 4, 4] since you compare every pair to any other pair that exists. But your output is different because you append both pairs every time and getting a very big result, so you shouldn't append the num2 pairs. Now, from what you described that you want, you should loop every pair and check if it already exists in the output list. So the for loop part can change like this filtered = [] for pair in zip(randomStringpt1,randomStringpt2): if pair not in filtered: filtered.append(pair) # Adding the pairs that aren't equal to each other to a new list the zip function takes the 2 lists and for every loop it returns 2 values one from each list the first value pair, then the second values and goes on. the filtered list will be in the following format [(1, 2), (3, 4), (1, 3)] Alternative it can be as a one liner like this: filtered = list(dict.fromkeys(zip(randomStringpt1, randomStringpt2))) using the dictionary to identify unique elements and then turn it back into a list after all that you can get the original format of the lists you had in your code by splitting them like this newStart = [pair[0] for pair in filtered] newEnd = [pair[1] for pair in filtered] Finally i should tell you to read a little more on python and it's for loops, since the range(len(yourlist)) is not the python intended way to loop over lists, as python for loops are equivalent to for each loops on other languages and iterate over the list for you instead on relying in a value to get list elements like yourlist[value].
The best way of iterating through an array whose length changes in Python
I am implementing an algorithm which might affect the size of some array, and I need to iterate through the entire array. Basically a 'for x in arrayname' would not work because it does not update if the contents of arrayname are changed in the loop. I came up with an ugly solution which is shown in the following example: test = np.array([1,2,3]) N = len(test) ii=0 while ii < N: N = len(test) print(test[ii]) if test[ii] ==2: test = np.append(test,4) ii+=1 I am wondering whether a cleaner solution exists. Thanks in advance!
Assuming all the elements are going to be added at the end and no elements are being deleted you could store the new elements in a separate list: master_list = [1,2,3] curr_elems = master_list while len(curr_elems) > 0: # keep looping over new elements added new_elems = [] for item in curr_elems: # loop over the current list of elements, initially the list but then all the added elements on second run etc if should_add_element(item): new_elems.append(generate_new_element(item)) master_list.extend(new_elems) # add all the new elements to our master list curr_elems = new_elems # and prep to iterate over the new elements for next iteration of the while loop
The while loop seems the best solution. As the condition is re-evaluated at each iteration, you don’t need to reset the length of the list in the loop, you can do it inside the condition: import random l = [1, 2, 3, 4, 5] i = 0 while i < len(l): if random.choice([True, False]): del l[i] else: i += 1 print(f'{l=}') This example gives a blueprint for a more complex algorithm. Of course, in this simple case, it could be coded more simply with a filter, or like this: l = [1, 2, 3, 4, 5] [x for x in l if random.choice([True, False])] You might want to check this related post for more creative solutions: How to remove items from a list while iterating?
Is there a way to sort an unsorted list with some repeated elements?
I am trying to sort an unsorted list [4, 5, 9, 9, 0, 1, 8] The list has two repeated elements. I have tried to approach the question by having a loop that goes through each element comparing each element with the next in the list and then placing the smaller element at the start of the list. def sort(ls: ls[x] x = [4, 5, 9, 9, 0, 1, 8] while len(x) > 0: for i in the range(0, len(x)): lowest = x[i] ls.append(lowest) Please, could someone explain where I am going wrong and how the code should work? It may be that I have incorrectly thought about the problem and my reasoning for how the code should work is not right
I do not know, if this is exactly what you are looking for but try: sorted(ListObject). sorted() returns the elements of the list from the smallest to the biggest. If one element is repeated, the repeated element is right after the original element. Hope that helped.
Yes, you can try x.sort() or sorted(x). Check this out https://www.programiz.com/python-programming/methods/built-in/sorted. Also, in your program I don't see you making any comparisons, for example, if x[i] <= x[i+1] then ... This block of code is just gonna append all the elements in the same order, till n*n times. Also check this https://en.wikipedia.org/wiki/Insertion_sort
For a built-in Python function to sort, let y be your original list, you can use either sorted(y) or y.sort().Keep in mind that sorted(y) will return a new list so you would need to assign it to a variable such as x=sorted(y); whereas if you use x.sort() it will mutate the original list in-place, so you would just call it as is. If you're looking to actually implement a sorting function, you can try Merge Sort or Quick Sort which run in O (n log n) in which will handle elements with the same value. You can check this out if you want -> https://www.geeksforgeeks.org/python-program-for-merge-sort/ . For an easier to understand sorting algorithm, Insertion or Bubble sort also handle duplicate as well but have a longer runtime O (n^2) -> https://www.geeksforgeeks.org/python-program-for-bubble-sort/ . But yea, I agree with Nameet, what you've currently posted looks like it would just append in the same order. Try one of the above suggestions and hopefully this helps point you in the right direction to if you're looking for a built-in function or to implement a sort, which can be done in multiple ways with different adv and disadv to each one. Hope this helps and good luck!
There are several popular ways for sorting. take bubble sort as an example, def bubbleSort(array): x = len(array) while(x > 1): # the code below make sense only there are at least 2 elements in the list for i in range(x-1): # maximum of i is x-2, the last element in arr is arr[x-1] if array[i] > array[i+1]: array[i], array[i+1] = array[i+1], array[i] x -= 1 return array x = [4, 5, 9, 9, 0, 1, 8] bubbleSort(x) your code has the same logic as below def sorts(x): ls = [] while len(x) > 0: lowest = min(x) ls.append(lowest) x.remove(lowest) return ls x = [4, 5, 9, 9, 0, 1, 8] sorts(x) #output is [0, 1, 4, 5, 8, 9, 9]
Get name of elements of a OrderedDict in pandas [duplicate]
With Python 2.7, I can get dictionary keys, values, or items as a list: >>> newdict = {1:0, 2:0, 3:0} >>> newdict.keys() [1, 2, 3] With Python >= 3.3, I get: >>> newdict.keys() dict_keys([1, 2, 3]) How do I get a plain list of keys with Python 3?
This will convert the dict_keys object to a list: list(newdict.keys()) On the other hand, you should ask yourself whether or not it matters. It is Pythonic to assume duck typing -- if it looks like a duck and it quacks like a duck, it is a duck. The dict_keys object can be iterated over just like a list. For instance: for key in newdict.keys(): print(key) Note that dict_keys doesn't support insertion newdict[k] = v, though you may not need it.
Python >= 3.5 alternative: unpack into a list literal [*newdict] New unpacking generalizations (PEP 448) were introduced with Python 3.5 allowing you to now easily do: >>> newdict = {1:0, 2:0, 3:0} >>> [*newdict] [1, 2, 3] Unpacking with * works with any object that is iterable and, since dictionaries return their keys when iterated through, you can easily create a list by using it within a list literal. Adding .keys() i.e [*newdict.keys()] might help in making your intent a bit more explicit though it will cost you a function look-up and invocation. (which, in all honesty, isn't something you should really be worried about). The *iterable syntax is similar to doing list(iterable) and its behaviour was initially documented in the Calls section of the Python Reference manual. With PEP 448 the restriction on where *iterable could appear was loosened allowing it to also be placed in list, set and tuple literals, the reference manual on Expression lists was also updated to state this. Though equivalent to list(newdict) with the difference that it's faster (at least for small dictionaries) because no function call is actually performed: %timeit [*newdict] 1000000 loops, best of 3: 249 ns per loop %timeit list(newdict) 1000000 loops, best of 3: 508 ns per loop %timeit [k for k in newdict] 1000000 loops, best of 3: 574 ns per loop with larger dictionaries the speed is pretty much the same (the overhead of iterating through a large collection trumps the small cost of a function call). In a similar fashion, you can create tuples and sets of dictionary keys: >>> *newdict, (1, 2, 3) >>> {*newdict} {1, 2, 3} beware of the trailing comma in the tuple case!
list(newdict) works in both Python 2 and Python 3, providing a simple list of the keys in newdict. keys() isn't necessary.
You can also use a list comprehension: >>> newdict = {1:0, 2:0, 3:0} >>> [k for k in newdict.keys()] [1, 2, 3] Or, shorter, >>> [k for k in newdict] [1, 2, 3] Note: Order is not guaranteed on versions under 3.7 (ordering is still only an implementation detail with CPython 3.6).
A bit off on the "duck typing" definition -- dict.keys() returns an iterable object, not a list-like object. It will work anywhere an iterable will work -- not any place a list will. a list is also an iterable, but an iterable is NOT a list (or sequence...) In real use-cases, the most common thing to do with the keys in a dict is to iterate through them, so this makes sense. And if you do need them as a list you can call list(). Very similarly for zip() -- in the vast majority of cases, it is iterated through -- why create an entire new list of tuples just to iterate through it and then throw it away again? This is part of a large trend in python to use more iterators (and generators), rather than copies of lists all over the place. dict.keys() should work with comprehensions, though -- check carefully for typos or something... it works fine for me: >>> d = dict(zip(['Sounder V Depth, F', 'Vessel Latitude, Degrees-Minutes'], [None, None])) >>> [key.split(", ") for key in d.keys()] [['Sounder V Depth', 'F'], ['Vessel Latitude', 'Degrees-Minutes']]
If you need to store the keys separately, here's a solution that requires less typing than every other solution presented thus far, using Extended Iterable Unpacking (Python3.x+): newdict = {1: 0, 2: 0, 3: 0} *k, = newdict k # [1, 2, 3] Operation no. Of characters k = list(d) 9 characters (excluding whitespace) k = [*d] 6 characters *k, = d 5 characters
Converting to a list without using the keys method makes it more readable: list(newdict) and, when looping through dictionaries, there's no need for keys(): for key in newdict: print key unless you are modifying it within the loop which would require a list of keys created beforehand: for key in list(newdict): del newdict[key] On Python 2 there is a marginal performance gain using keys().
Yes, There is a better and simplest way to do this in python3.X use inbuild list() function #Devil newdict = {1:0, 2:0, 3:0} key_list = list(newdict) print(key_list) #[1, 2, 3]
I can think of 2 ways in which we can extract the keys from the dictionary. Method 1: - To get the keys using .keys() method and then convert it to list. some_dict = {1: 'one', 2: 'two', 3: 'three'} list_of_keys = list(some_dict.keys()) print(list_of_keys) -->[1,2,3] Method 2: - To create an empty list and then append keys to the list via a loop. You can get the values with this loop as well (use .keys() for just keys and .items() for both keys and values extraction) list_of_keys = [] list_of_values = [] for key,val in some_dict.items(): list_of_keys.append(key) list_of_values.append(val) print(list_of_keys) -->[1,2,3] print(list_of_values) -->['one','two','three']
Beyond the classic (and probably more correct) way to do this (some_dict.keys()) there is also a more "cool" and surely more interesting way to do this: some_dict = { "foo": "bar", "cool": "python!" } print( [*some_dict] == ["foo", "cool"] ) # True Note: this solution shouldn't be used in a develop environment; I showed it here just because I thought it was quite interesting from the *-operator-over-dictionary side of view. Also, I'm not sure whether this is a documented feature or not, and its behaviour may change in later versions :)
You can you use simple method like below keys = newdict.keys() print(keys)
This is the best way to get key List in one line of code dict_variable = {1:"a",2:"b",3:"c"} [key_val for key_val in dict_variable.keys()]