What's the difference between tuples/lists and what are their advantages/disadvantages?
Apart from tuples being immutable there is also a semantic distinction that should guide their usage. Tuples are heterogeneous data structures (i.e., their entries have different meanings), while lists are homogeneous sequences. Tuples have structure, lists have order.
Using this distinction makes code more explicit and understandable.
One example would be pairs of page and line number to reference locations in a book, e.g.:
my_location = (42, 11) # page number, line number
You can then use this as a key in a dictionary to store notes on locations. A list on the other hand could be used to store multiple locations. Naturally one might want to add or remove locations from the list, so it makes sense that lists are mutable. On the other hand it doesn't make sense to add or remove items from an existing location - hence tuples are immutable.
There might be situations where you want to change items within an existing location tuple, for example when iterating through the lines of a page. But tuple immutability forces you to create a new location tuple for each new value. This seems inconvenient on the face of it, but using immutable data like this is a cornerstone of value types and functional programming techniques, which can have substantial advantages.
There are some interesting articles on this issue, e.g. "Python Tuples are Not Just Constant Lists" or "Understanding tuples vs. lists in Python". The official Python documentation also mentions this
"Tuples are immutable, and usually contain an heterogeneous sequence ...".
In a statically typed language like Haskell the values in a tuple generally have different types and the length of the tuple must be fixed. In a list the values all have the same type and the length is not fixed. So the difference is very obvious.
Finally there is the namedtuple in Python, which makes sense because a tuple is already supposed to have structure. This underlines the idea that tuples are a light-weight alternative to classes and instances.
Difference between list and tuple
Literal
someTuple = (1,2)
someList = [1,2]
Size
a = tuple(range(1000))
b = list(range(1000))
a.__sizeof__() # 8024
b.__sizeof__() # 9088
Due to the smaller size of a tuple operation, it becomes a bit faster, but not that much to mention about until you have a huge number of elements.
Permitted operations
b = [1,2]
b[0] = 3 # [3, 2]
a = (1,2)
a[0] = 3 # Error
That also means that you can't delete an element or sort a tuple.
However, you could add a new element to both list and tuple with the only difference that since the tuple is immutable, you are not really adding an element but you are creating a new tuple, so the id of will change
a = (1,2)
b = [1,2]
id(a) # 140230916716520
id(b) # 748527696
a += (3,) # (1, 2, 3)
b += [3] # [1, 2, 3]
id(a) # 140230916878160
id(b) # 748527696
Usage
As a list is mutable, it can't be used as a key in a dictionary, whereas a tuple can be used.
a = (1,2)
b = [1,2]
c = {a: 1} # OK
c = {b: 1} # Error
If you went for a walk, you could note your coordinates at any instant in an (x,y) tuple.
If you wanted to record your journey, you could append your location every few seconds to a list.
But you couldn't do it the other way around.
The key difference is that tuples are immutable. This means that you cannot change the values in a tuple once you have created it.
So if you're going to need to change the values use a List.
Benefits to tuples:
Slight performance improvement.
As a tuple is immutable it can be used as a key in a dictionary.
If you can't change it neither can anyone else, which is to say you don't need to worry about any API functions etc. changing your tuple without being asked.
Lists are mutable; tuples are not.
From docs.python.org/2/tutorial/datastructures.html
Tuples are immutable, and usually contain an heterogeneous sequence of
elements that are accessed via unpacking (see later in this section)
or indexing (or even by attribute in the case of namedtuples). Lists
are mutable, and their elements are usually homogeneous and are
accessed by iterating over the list.
This is an example of Python lists:
my_list = [0,1,2,3,4]
top_rock_list = ["Bohemian Rhapsody","Kashmir","Sweet Emotion", "Fortunate Son"]
This is an example of Python tuple:
my_tuple = (a,b,c,d,e)
celebrity_tuple = ("John", "Wayne", 90210, "Actor", "Male", "Dead")
Python lists and tuples are similar in that they both are ordered collections of values. Besides the shallow difference that lists are created using brackets "[ ... , ... ]" and tuples using parentheses "( ... , ... )", the core technical "hard coded in Python syntax" difference between them is that the elements of a particular tuple are immutable whereas lists are mutable (...so only tuples are hashable and can be used as dictionary/hash keys!). This gives rise to differences in how they can or can't be used (enforced a priori by syntax) and differences in how people choose to use them (encouraged as 'best practices,' a posteriori, this is what smart programers do). The main difference a posteriori in differentiating when tuples are used versus when lists are used lies in what meaning people give to the order of elements.
For tuples, 'order' signifies nothing more than just a specific 'structure' for holding information. What values are found in the first field can easily be switched into the second field as each provides values across two different dimensions or scales. They provide answers to different types of questions and are typically of the form: for a given object/subject, what are its attributes? The object/subject stays constant, the attributes differ.
For lists, 'order' signifies a sequence or a directionality. The second element MUST come after the first element because it's positioned in the 2nd place based on a particular and common scale or dimension. The elements are taken as a whole and mostly provide answers to a single question typically of the form, for a given attribute, how do these objects/subjects compare? The attribute stays constant, the object/subject differs.
There are countless examples of people in popular culture and programmers who don't conform to these differences and there are countless people who might use a salad fork for their main course. At the end of the day, it's fine and both can usually get the job done.
To summarize some of the finer details
Similarities:
Duplicates - Both tuples and lists allow for duplicates
Indexing, Selecting, & Slicing - Both tuples and lists index using integer values found within brackets. So, if you want the first 3 values of a given list or tuple, the syntax would be the same:
>>> my_list[0:3]
[0,1,2]
>>> my_tuple[0:3]
[a,b,c]
Comparing & Sorting - Two tuples or two lists are both compared by their first element, and if there is a tie, then by the second element, and so on. No further attention is paid to subsequent elements after earlier elements show a difference.
>>> [0,2,0,0,0,0]>[0,0,0,0,0,500]
True
>>> (0,2,0,0,0,0)>(0,0,0,0,0,500)
True
Differences: - A priori, by definition
Syntax - Lists use [], tuples use ()
Mutability - Elements in a given list are mutable, elements in a given tuple are NOT mutable.
# Lists are mutable:
>>> top_rock_list
['Bohemian Rhapsody', 'Kashmir', 'Sweet Emotion', 'Fortunate Son']
>>> top_rock_list[1]
'Kashmir'
>>> top_rock_list[1] = "Stairway to Heaven"
>>> top_rock_list
['Bohemian Rhapsody', 'Stairway to Heaven', 'Sweet Emotion', 'Fortunate Son']
# Tuples are NOT mutable:
>>> celebrity_tuple
('John', 'Wayne', 90210, 'Actor', 'Male', 'Dead')
>>> celebrity_tuple[5]
'Dead'
>>> celebrity_tuple[5]="Alive"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
Hashtables (Dictionaries) - As hashtables (dictionaries) require that its keys are hashable and therefore immutable, only tuples can act as dictionary keys, not lists.
#Lists CAN'T act as keys for hashtables(dictionaries)
>>> my_dict = {[a,b,c]:"some value"}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
#Tuples CAN act as keys for hashtables(dictionaries)
>>> my_dict = {("John","Wayne"): 90210}
>>> my_dict
{('John', 'Wayne'): 90210}
Differences - A posteriori, in usage
Homo vs. Heterogeneity of Elements - Generally list objects are homogenous and tuple objects are heterogeneous. That is, lists are used for objects/subjects of the same type (like all presidential candidates, or all songs, or all runners) whereas although it's not forced by), whereas tuples are more for heterogenous objects.
Looping vs. Structures - Although both allow for looping (for x in my_list...), it only really makes sense to do it for a list. Tuples are more appropriate for structuring and presenting information (%s %s residing in %s is an %s and presently %s % ("John","Wayne",90210, "Actor","Dead"))
It's been mentioned that the difference is largely semantic: people expect a tuple and list to represent different information. But this goes further than a guideline; some libraries actually behave differently based on what they are passed. Take NumPy for example (copied from another post where I ask for more examples):
>>> import numpy as np
>>> a = np.arange(9).reshape(3,3)
>>> a
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
>>> idx = (1,1)
>>> a[idx]
4
>>> idx = [1,1]
>>> a[idx]
array([[3, 4, 5],
[3, 4, 5]])
The point is, while NumPy may not be part of the standard library, it's a major Python library, and within NumPy lists and tuples are completely different things.
Lists are for looping, tuples are for structures i.e. "%s %s" %tuple.
Lists are usually homogeneous, tuples are usually heterogeneous.
Lists are for variable length, tuples are for fixed length.
The values of list can be changed any time but the values of tuples can't be change.
The advantages and disadvantages depends upon the use. If you have such a data which you never want to change then you should have to use tuple, otherwise list is the best option.
Difference between list and tuple
Tuples and lists are both seemingly similar sequence types in Python.
Literal syntax
We use parenthesis () to construct tuples and square brackets [ ] to get a new list. Also, we can use call of the appropriate type to get required structure — tuple or list.
someTuple = (4,6)
someList = [2,6]
Mutability
Tuples are immutable, while lists are mutable. This point is the base the for the following ones.
Memory usage
Due to mutability, you need more memory for lists and less memory for tuples.
Extending
You can add a new element to both tuples and lists with the only difference that the id of the tuple will be changed (i.e., we’ll have a new object).
Hashing
Tuples are hashable and lists are not. It means that you can use a tuple as a key in a dictionary. The list can't be used as a key in a dictionary, whereas a tuple can be used
tup = (1,2)
list_ = [1,2]
c = {tup : 1} # ok
c = {list_ : 1} # error
Semantics
This point is more about best practice. You should use tuples as heterogeneous data structures, while lists are homogenous sequences.
Lists are intended to be homogeneous sequences, while tuples are heterogeneous data structures.
As people have already answered here that tuples are immutable while lists are mutable, but there is one important aspect of using tuples which we must remember
If the tuple contains a list or a dictionary inside it, those can be changed even if the tuple itself is immutable.
For example, let's assume we have a tuple which contains a list and a dictionary as
my_tuple = (10,20,30,[40,50],{ 'a' : 10})
we can change the contents of the list as
my_tuple[3][0] = 400
my_tuple[3][1] = 500
which makes new tuple looks like
(10, 20, 30, [400, 500], {'a': 10})
we can also change the dictionary inside tuple as
my_tuple[4]['a'] = 500
which will make the overall tuple looks like
(10, 20, 30, [400, 500], {'a': 500})
This happens because list and dictionary are the objects and these objects are not changing, but the contents its pointing to.
So the tuple remains immutable without any exception
The PEP 484 -- Type Hints says that the types of elements of a tuple can be individually typed; so that you can say Tuple[str, int, float]; but a list, with List typing class can take only one type parameter: List[str], which hints that the difference of the 2 really is that the former is heterogeneous, whereas the latter intrinsically homogeneous.
Also, the standard library mostly uses the tuple as a return value from such standard functions where the C would return a struct.
As people have already mentioned the differences I will write about why tuples.
Why tuples are preferred?
Allocation optimization for small tuples
To reduce memory fragmentation and speed up allocations, Python reuses old tuples. If a
tuple no longer needed and has less than 20 items instead of deleting
it permanently Python moves it to a free list.
A free list is divided into 20 groups, where each group represents a
list of tuples of length n between 0 and 20. Each group can store up
to 2 000 tuples. The first (zero) group contains only 1 element and
represents an empty tuple.
>>> a = (1,2,3)
>>> id(a)
4427578104
>>> del a
>>> b = (1,2,4)
>>> id(b)
4427578104
In the example above we can see that a and b have the same id. That is
because we immediately occupied a destroyed tuple which was on the
free list.
Allocation optimization for lists
Since lists can be modified, Python does not use the same optimization as in tuples. However,
Python lists also have a free list, but it is used only for empty
objects. If an empty list is deleted or collected by GC, it can be
reused later.
>>> a = []
>>> id(a)
4465566792
>>> del a
>>> b = []
>>> id(b)
4465566792
Source: https://rushter.com/blog/python-lists-and-tuples/
Why tuples are efficient than lists? -> https://stackoverflow.com/a/22140115
The most important difference is time ! When you do not want to change the data inside the list better to use tuple ! Here is the example why use tuple !
import timeit
print(timeit.timeit(stmt='[1,2,3,4,5,6,7,8,9,10]', number=1000000)) #created list
print(timeit.timeit(stmt='(1,2,3,4,5,6,7,8,9,10)', number=1000000)) # created tuple
In this example we executed both statements 1 million times
Output :
0.136621
0.013722200000000018
Any one can clearly notice the time difference.
A direction quotation from the documentation on 5.3. Tuples and Sequences:
Though tuples may seem similar to lists, they are often used in different situations and for different purposes. Tuples are immutable, and usually contain a heterogeneous sequence of elements that are accessed via unpacking (see later in this section) or indexing (or even by attribute in the case of namedtuples). Lists are mutable, and their elements are usually homogeneous and are accessed by iterating over the list.
In other words, TUPLES are used to store group of elements where the contents/members of the group would not change while LISTS are used to store group of elements where the members of the group can change.
For instance, if i want to store IP of my network in a variable, it's best i used a tuple since the the IP is fixed. Like this my_ip = ('192.168.0.15', 33, 60). However, if I want to store group of IPs of places I would visit in the next 6 month, then I should use a LIST, since I will keep updating and adding new IP to the group. Like this
places_to_visit = [
('192.168.0.15', 33, 60),
('192.168.0.22', 34, 60),
('192.168.0.1', 34, 60),
('192.168.0.2', 34, 60),
('192.168.0.8', 34, 60),
('192.168.0.11', 34, 60)
]
First of all, they both are the non-scalar objects (also known as a compound objects) in Python.
Tuples, ordered sequence of elements (which can contain any object with no aliasing issue)
Immutable (tuple, int, float, str)
Concatenation using + (brand new tuple will be created of course)
Indexing
Slicing
Singleton (3,) # -> (3) instead of (3) # -> 3
List (Array in other languages), ordered sequence of values
Mutable
Singleton [3]
Cloning new_array = origin_array[:]
List comprehension [x**2 for x in range(1,7)] gives you
[1,4,9,16,25,36] (Not readable)
Using list may also cause an aliasing bug (two distinct paths
pointing to the same object).
Just a quick extension to list vs tuple responses:
Due to dynamic nature, list allocates more bit buckets than the actual memory required. This is done to prevent costly reallocation operation in case extra items are appended in the future.
On the other hand, being static, lightweight tuple object does not reserve extra memory required to store them.
Lists are mutable and tuples are immutable.
Just consider this example.
a = ["1", "2", "ra", "sa"] #list
b = ("1", "2", "ra", "sa") #tuple
Now change index values of list and tuple.
a[2] = 1000
print a #output : ['1', '2', 1000, 'sa']
b[2] = 1000
print b #output : TypeError: 'tuple' object does not support item assignment.
Hence proved the following code is invalid with tuple, because we attempted to update a tuple, which is not allowed.
Lists are mutable. whereas tuples are immutable. Accessing an offset element with index makes more sense in tuples than lists, Because the elements and their index cannot be changed.
List is mutable and tuples is immutable. The main difference between mutable and immutable is memory usage when you are trying to append an item.
When you create a variable, some fixed memory is assigned to the variable. If it is a list, more memory is assigned than actually used. E.g. if current memory assignment is 100 bytes, when you want to append the 101th byte, maybe another 100 bytes will be assigned (in total 200 bytes in this case).
However, if you know that you are not frequently add new elements, then you should use tuples. Tuples assigns exactly size of the memory needed, and hence saves memory, especially when you use large blocks of memory.
Related
I've noticed that many operations on lists that modify the list's contents will return None, rather than returning the list itself. Examples:
>>> mylist = ['a', 'b', 'c']
>>> empty = mylist.clear()
>>> restored = mylist.extend(range(3))
>>> backwards = mylist.reverse()
>>> with_four = mylist.append(4)
>>> in_order = mylist.sort()
>>> without_one = mylist.remove(1)
>>> mylist
[0, 2, 4]
>>> [empty, restored, backwards, with_four, in_order, without_one]
[None, None, None, None, None, None]
What is the thought process behind this decision?
To me, it seems hampering, since it prevents "chaining" of list processing (e.g. mylist.reverse().append('a string')[:someLimit]). I imagine it might be that "The Powers That Be" decided that list comprehension is a better paradigm (a valid opinion), and so didn't want to encourage other methods - but it seems perverse to prevent an intuitive method, even if better alternatives exist.
This question is specifically about Python's design decision to return None from mutating list methods like .append. Novices often write incorrect code that expects .append (in particular) to return the same list that was just modified.
For the simple question of "how do I append to a list?" (or debugging questions that boil down to that problem), see Why does "x = x.append([i])" not work in a for loop?.
To get modified versions of the list, see:
For .sort: How can I get a sorted copy of a list?
For .reverse: How can I get a reversed copy of a list (avoid a separate statement when chaining a method after .reverse)?
The same issue applies to some methods of other built-in data types, e.g. set.discard (see How to remove specific element from sets inside a list using list comprehension) and dict.update (see Why doesn't a python dict.update() return the object?).
The same reasoning applies to designing your own APIs. See Is making in-place operations return the object a bad idea?.
The general design principle in Python is for functions that mutate an object in-place to return None. I'm not sure it would have been the design choice I'd have chosen, but it's basically to emphasise that a new object is not returned.
Guido van Rossum (our Python BDFL) states the design choice on the Python-Dev mailing list:
I'd like to explain once more why I'm so adamant that sort() shouldn't
return 'self'.
This comes from a coding style (popular in various other languages, I
believe especially Lisp revels in it) where a series of side effects
on a single object can be chained like this:
x.compress().chop(y).sort(z)
which would be the same as
x.compress()
x.chop(y)
x.sort(z)
I find the chaining form a threat to readability; it requires that the
reader must be intimately familiar with each of the methods. The
second form makes it clear that each of these calls acts on the same
object, and so even if you don't know the class and its methods very
well, you can understand that the second and third call are applied to
x (and that all calls are made for their side-effects), and not to
something else.
I'd like to reserve chaining for operations that return new values,
like string processing operations:
y = x.rstrip("\n").split(":").lower()
There are a few standard library modules that encourage chaining of
side-effect calls (pstat comes to mind). There shouldn't be any new
ones; pstat slipped through my filter when it was weak.
I can't speak for the developers, but I find this behavior very intuitive.
If a method works on the original object and modifies it in-place, it doesn't return anything, because there is no new information - you obviously already have a reference to the (now mutated) object, so why return it again?
If, however, a method or function creates a new object, then of course it has to return it.
So l.reverse() returns nothing (because now the list has been reversed, but the identfier l still points to that list), but reversed(l) has to return the newly generated list because l still points to the old, unmodified list.
EDIT: I just learned from another answer that this principle is called Command-Query separation.
One could argue that the signature itself makes it clear that the function mutates the list rather than returning a new one: if the function returned a list, its behavior would have been much less obvious.
If you were sent here after asking for help fixing your code:
In the future, please try to look for problems in the code yourself, by carefully studying what happens when the code runs. Rather than giving up because there is an error message, check the result of each calculation, and see where the code starts working differently from what you expect.
If you had code calling a method like .append or .sort on a list, you will notice that the return value is None, while the list is modified in place. Study the example carefully:
>>> x = ['e', 'x', 'a', 'm', 'p', 'l', 'e']
>>> y = x.sort()
>>> print(y)
None
>>> print(x)
['a', 'e', 'e', 'l', 'm', 'p', 'x']
y got the special None value, because that is what was returned. x changed, because the sort happened in place.
It works this way on purpose, so that code like x.sort().reverse() breaks. See the other answers to understand why the Python developers wanted it that way.
To fix the problem
First, think carefully about the intent of the code. Should x change? Do we actually need a separate y?
Let's consider .sort first. If x should change, then call x.sort() by itself, without assigning the result anywhere.
If a sorted copy is needed instead, use y = x.sorted(). See How can I get a sorted copy of a list? for details.
For other methods, we can get modified copies like so:
.clear -> there is no point to this; a "cleared copy" of the list is just an empty list. Just use y = [].
.append and .extend -> probably the simplest way is to use the + operator. To add multiple elements from a list l, use y = x + l rather than .extend. To add a single element e wrap it in a list first: y = x + [e]. Another way in 3.5 and up is to use unpacking: y = [*x, *l] for .extend, y = [*x, e] for .append. See also How to allow list append() method to return the new list for .append and How do I concatenate two lists in Python? for .extend.
.reverse -> First, consider whether an actual copy is needed. The built-in reversed gives you an iterator that can be used to loop over the elements in reverse order. To make an actual copy, simply pass that iterator to list: y = list(reversed(x)). See How can I get a reversed copy of a list (avoid a separate statement when chaining a method after .reverse)? for details.
.remove -> Figure out the index of the element that will be removed (using .index), then use slicing to find the elements before and after that point and put them together. As a function:
def without(a_list, value):
index = a_list.index(value)
return a_list[:index] + a_list[index+1:]
(We can translate .pop similarly to make a modified copy, though of course .pop actually returns an element from the list.)
See also A quick way to return list without a specific element in Python.
(If you plan to remove multiple elements, strongly consider using a list comprehension (or filter) instead. It will be much simpler than any of the workarounds needed for removing items from the list while iterating over it. This way also naturally gives a modified copy.)
For any of the above, of course, we can also make a modified copy by explicitly making a copy and then using the in-place method on the copy. The most elegant approach will depend on the context and on personal taste.
As we know list in python is a mutable object and one of characteristics of mutable object is the ability to modify the state of this object without the need to assign its new state to a variable. we should demonstrate more about this topic to understand the root of this issue.
An object whose internal state can be changed is mutable. On the other hand, immutable doesn’t allow any change in the object once it has been created. Object mutability is one of the characteristics that makes Python a dynamically typed language.
Every object in python has three attributes:
Identity – This refers to the address that the object refers to in the computer’s memory.
Type – This refers to the kind of object that is created. For example integer, list, string etc.
Value – This refers to the value stored by the object. For example str = "a".
While ID and Type cannot be changed once it’s created, values can be changed for Mutable objects.
let us discuss the below code step-by-step to depict what it means in Python:
Creating a list which contains name of cities
cities = ['London', 'New York', 'Chicago']
Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(cities)))
Output [1]: 0x1691d7de8c8
Adding a new city to the list cities
cities.append('Delhi')
Printing the elements from the list cities, separated by a comma
for city in cities:
print(city, end=', ')
Output [2]: London, New York, Chicago, Delhi
Printing the location of the object created in the memory address in hexadecimal format
print(hex(id(cities)))
Output [3]: 0x1691d7de8c8
The above example shows us that we were able to change the internal state of the object cities by adding one more city 'Delhi' to it, yet, the memory address of the object did not change. This confirms that we did not create a new object, rather, the same object was changed or mutated. Hence, we can say that the object which is a type of list with reference variable name cities is a MUTABLE OBJECT.
While the immutable object internal state can not be changed. For instance, consider the below code and associated error message with it, while trying to change the value of a Tuple at index 0
Creating a Tuple with variable name foo
foo = (1, 2)
Changing the index 0 value from 1 to 3
foo[0] = 3
TypeError: 'tuple' object does not support item assignment
We can conclude from the examples why mutable object shouldn't return anything when executing operations on it because it's modifying the internal state of the object directly and there is no point in returning new modified object. unlike immutable object which should return new object of the modified state after executing operations on it.
First of All, I should tell that what I am suggesting is without a doubt, a bad programming practice but if you want to use append in lambda function and you don't care about the code readability, there is way to just do that.
Imagine you have a list of lists and you want to append a element to each inner lists using map and lambda. here is how you can do that:
my_list = [[1, 2, 3, 4],
[3, 2, 1],
[1, 1, 1]]
my_new_element = 10
new_list = list(map(lambda x: [x.append(my_new_element), x][1], my_list))
print(new_list)
How it works:
when lambda wants to calculate to output, first it should calculate the [x.append(my_new_element), x] expression. To calculate this expression the append function will run and the result of expression will be [None, x] and by specifying that you want the second element of the list the result of [None,x][1] will be x
Using custom function is more readable and the better option:
def append_my_list(input_list, new_element):
input_list.append(new_element)
return input_list
my_list = [[1, 2, 3, 4],
[3, 2, 1],
[1, 1, 1]]
my_new_element = 10
new_list = list(map(lambda x: append_my_list(x, my_new_element), my_list))
print(new_list)
Why two zip objects (made using same two iterables) aren't equal?
list1 = [1, 2, 3, 4]
list2 = [5, 6, 7, 8]
a = zip(list1, list2)
b = zip(list1, list2)
print(a == b)
The above code is printing False.
Shouldn't it give `True' since the zip objects are similar?
The returned zip object is an iterator. To be able to test the value equality of it, potentially the entire iterator would need to be walked, which would exhaust it, so you'd never be able to use it after. And if only part of the iterator was walked during the test, it would be left in a "difficult to determine state", so it wouldn't be usable there either.
I'm not sure if zip is considered to be a "user-defined class", but note this line in the docs:
User-defined classes have __eq__() and __hash__() methods by default; with them, all objects compare unequal (except with themselves)
In theory they could have made a workaround that would allow equality testing of a zip object (in cases where the object that was zipped wasn't itself an iterator), but there was likely no need. I can't think of a scenario where you would need to test such a thing, and where comparing the object forced as a list isn't an option.
I am trying to understand the Python hash function under the hood. I created a custom class where all instances return the same hash value.
class C:
def __hash__(self):
return 42
I just assumed that only one instance of the above class can be in a dict at any time, but in fact a dict can have multiple elements with the same hash.
c, d = C(), C()
x = {c: 'c', d: 'd'}
print(x)
# {<__main__.C object at 0x7f0824087b80>: 'c', <__main__.C object at 0x7f0823ae2d60>: 'd'}
# note that the dict has 2 elements
I experimented a little more and found that if I override the __eq__ method such that all the instances of the class compare equal, then the dict only allows one instance.
class D:
def __hash__(self):
return 42
def __eq__(self, other):
return True
p, q = D(), D()
y = {p: 'p', q: 'q'}
print(y)
# {<__main__.D object at 0x7f0823a9af40>: 'q'}
# note that the dict only has 1 element
So I am curious to know how a dict can have multiple elements with the same hash.
Here is everything about Python dicts that I was able to put together (probably more than anyone would like to know; but the answer is comprehensive). A shout out to Duncan for pointing out that Python dicts use slots and leading me down this rabbit hole.
Python dictionaries are implemented as hash tables.
Hash tables must allow for hash collisions i.e. even if two keys have same hash value, the implementation of the table must have a strategy to insert and retrieve the key and value pairs unambiguously.
Python dict uses open addressing to resolve hash collisions (explained below) (see dictobject.c:296-297).
Python hash table is just a continguous block of memory (sort of like an array, so you can do O(1) lookup by index).
Each slot in the table can store one and only one entry. This is important
Each entry in the table actually a combination of the three values - . This is implemented as a C struct (see dictobject.h:51-56)
The figure below is a logical representation of a python hash table. In the figure below, 0, 1, ..., i, ... on the left are indices of the slots in the hash table (they are just for illustrative purposes and are not stored along with the table obviously!).
# Logical model of Python Hash table
-+-----------------+
0| <hash|key|value>|
-+-----------------+
1| ... |
-+-----------------+
.| ... |
-+-----------------+
i| ... |
-+-----------------+
.| ... |
-+-----------------+
n| ... |
-+-----------------+
When a new dict is initialized it starts with 8 slots. (see dictobject.h:49)
When adding entries to the table, we start with some slot, i that is based on the hash of the key. CPython uses initial i = hash(key) & mask. Where mask = PyDictMINSIZE - 1, but that's not really important). Just note that the initial slot, i, that is checked depends on the hash of the key.
If that slot is empty, the entry is added to the slot (by entry, I mean, <hash|key|value>). But what if that slot is occupied!? Most likely because another entry has the same hash (hash collision!)
If the slot is occupied, CPython (and even PyPy) compares the the hash AND the key (by compare I mean == comparison not the is comparison) of the entry in the slot against the key of the current entry to be inserted (dictobject.c:337,344-345). If both match, then it thinks the entry already exists, gives up and moves on to the next entry to be inserted. If either hash or the key don't match, it starts probing.
Probing just means it searches the slots by slot to find an empty slot. Technically we could just go one by one, i+1, i+2, ... and use the first available one (that's linear probing). But for reasons explained beautifully in the comments (see dictobject.c:33-126), CPython uses random probing. In random probing, the next slot is picked in a pseudo random order. The entry is added to the first empty slot. For this discussion, the actual algorithm used to pick the next slot is not really important (see dictobject.c:33-126 for the algorithm for probing). What is important is that the slots are probed until first empty slot is found.
The same thing happens for lookups, just starts with the initial slot i (where i depends on the hash of the key). If the hash and the key both don't match the entry in the slot, it starts probing, until it finds a slot with a match. If all slots are exhausted, it reports a fail.
BTW, the dict will be resized if it is two-thirds full. This avoids slowing down lookups. (see dictobject.h:64-65)
There you go! The Python implementation of dict checks for both hash equality of two keys and the normal equality (==) of the keys when inserting items. So in summary, if there are two keys, a and b and hash(a)==hash(b), but a!=b, then both can exist harmoniously in a Python dict. But if hash(a)==hash(b) and a==b, then they cannot both be in the same dict.
Because we have to probe after every hash collision, one side effect of too many hash collisions is that the lookups and insertions will become very slow (as Duncan points out in the comments).
I guess the short answer to my question is, "Because that's how it's implemented in the source code ;)"
While this is good to know (for geek points?), I am not sure how it can be used in real life. Because unless you are trying to explicitly break something, why would two objects that are not equal, have same hash?
For a detailed description of how Python's hashing works see my answer to Why is early return slower than else?
Basically it uses the hash to pick a slot in the table. If there is a value in the slot and the hash matches, it compares the items to see if they are equal.
If the hash matches but the items aren't equal, then it tries another slot. There's a formula to pick this (which I describe in the referenced answer), and it gradually pulls in unused parts of the hash value; but once it has used them all up, it will eventually work its way through all slots in the hash table. That guarantees eventually we either find a matching item or an empty slot. When the search finds an empty slot, it inserts the value or gives up (depending whether we are adding or getting a value).
The important thing to note is that there are no lists or buckets: there is just a hash table with a particular number of slots, and each hash is used to generate a sequence of candidate slots.
Edit: the answer below is one of possible ways to deal with hash collisions, it is however not how Python does it. Python's wiki referenced below is also incorrect. The best source given by #Duncan below is the implementation itself: https://github.com/python/cpython/blob/master/Objects/dictobject.c I apologize for the mix-up.
It stores a list (or bucket) of elements at the hash then iterates through that list until it finds the actual key in that list. A picture says more than a thousand words:
Here you see John Smith and Sandra Dee both hash to 152. Bucket 152 contains both of them. When looking up Sandra Dee it first finds the list in bucket 152, then loops through that list until Sandra Dee is found and returns 521-6955.
The following is wrong it's only here for context: On Python's wiki you can find (pseudo?) code how Python performs the lookup.
There's actually several possible solutions to this problem, check out the wikipedia article for a nice overview: http://en.wikipedia.org/wiki/Hash_table#Collision_resolution
Hash tables, in general have to allow for hash collisions! You will get unlucky and two things will eventually hash to the same thing. Underneath, there is a set of objects in a list of items that has that same hash key. Usually, there is only one thing in that list, but in this case, it'll keep stacking them into the same one. The only way it knows they are different is through the equals operator.
When this happens, your performance will degrade over time, which is why you want your hash function to be as "random as possible".
In the thread I did not see what exactly python does with instances of a user-defined classes when we put it into a dictionary as a keys. Let's read some documentation: it declares that only hashable objects can be used as a keys. Hashable are all immutable built-in classes and all user-defined classes.
User-defined classes have __cmp__() and
__hash__() methods by default; with them, all objects
compare unequal (except with themselves) and
x.__hash__() returns a result derived from id(x).
So if you have a constantly __hash__ in your class, but not providing any __cmp__ or __eq__ method, then all your instances are unequal for the dictionary.
In the other hand, if you providing any __cmp__ or __eq__ method, but not providing __hash__, your instances are still unequal in terms of dictionary.
class A(object):
def __hash__(self):
return 42
class B(object):
def __eq__(self, other):
return True
class C(A, B):
pass
dict_a = {A(): 1, A(): 2, A(): 3}
dict_b = {B(): 1, B(): 2, B(): 3}
dict_c = {C(): 1, C(): 2, C(): 3}
print(dict_a)
print(dict_b)
print(dict_c)
Output
{<__main__.A object at 0x7f9672f04850>: 1, <__main__.A object at 0x7f9672f04910>: 3, <__main__.A object at 0x7f9672f048d0>: 2}
{<__main__.B object at 0x7f9672f04990>: 2, <__main__.B object at 0x7f9672f04950>: 1, <__main__.B object at 0x7f9672f049d0>: 3}
{<__main__.C object at 0x7f9672f04a10>: 3}
The best way to build efficiently is to understand the toolkit one is building with. However, while trying to understand the core functions of python, it occurred to me that the map function gave similar, if not the same, results as a generic generator expression.
Take the next bit of code as a simplified example.
These two objects, mapped and generated, behave astoundingly similar in whatever situation you throw them.
def concatenate(string1 = "", string2 = ""):
return string1.join(" ", string2)
foo = ["One", "Two"]
bar = ["Blue", "Green"]
mapped = map(concatenate, foo, bar)
generated = (concatenate(string1 = a, string2 = b) for a, b in zip(foo, bar))
Okay, I know that it is a longer line of code, but I find it hard to believe that's all of map's reason of existence, so in my quest to understand python.
What does map still do in python? Is it really just a relic of olden times, and if not, where can I best put this tool to use?
The reason both exist is because list comprehension returns a new list where as map (in Python3) returns a generator thus map is more memory efficient if you don't need the resulting list right away.
This can be considered a technicality though since often when you use list comprehension to do something a map can do, you overwrite the original variable:
a = [1, 2, 3]
# The following line creates a new list but
# since we assign that list to `a` we give the old list to
# the garbage collector
a = [x**2 for x in a]
# or
a = list(map(lambda x: x**2, a))
# both of which are basically the same.
The power of map can come into play if you aren't working with the same variable:
a = [1, 2, 3] # If we want to save this then we don't want to overwrite it
b = [x**2 for x in a] # A full new list is now in b
c = map(lambda x: x**2, a) # c is just a generator object.
print(a) # [1, 2, 3]
print(b) # [1, 4, 9]
print(c) # <map object ...>
for x in c:
print(x)
# We never create a full list from c, we just use each object as we go thus we save memory.
In the previous example b is a whole new list of memory where as c is just a generator object. When working with such a small a, b and c are probably pretty close in memory but if a was large c would be significantly less memory than b.
Most common use cases of map involve the first case, thus map has no real benefit but in the second case it is much more beneficial to use map.
For the record "one way to do something" from the zen is sorta vague. I could implement a merge sort the right way (i.e. perfect speed and space) but my code my look completely different then someone else who implemented merge sort. "one way" doesn't really mean only one way to code it, it more implies there is only one methodology to be used or one process.
Why does this statement produce an error even though a string is actually a list of character constants?
string_name = ""
string_name.append("hello word")
The reason I expect this to work is because when we use for-loop, we are allowed to use this statement:
for i in string_name:
...
I think string_name is considered as a list here(?)
That's what they teach you in an algorithms and data structures class, that deal with algorithmic languages (unreal) rather than real programming languages, in Python, a string is a string, and a list is a list, they're different objects, you can "append" to a string using what is called string concatenation (which is basically an addition operation on strings):
string_name = "hello"
string_name = string_name + " world"
print(string_name) # => "hello world"
Or a shorthand concatenation:
string_name = "hello"
string_name += " world"
print(string_name) # => "hello world"
Lists and strings belong to this type called iterable. iterables are as they're name suggests, iterables, meaning you can iterate through them with the key word in, but that doesn't mean they're the same type of objects:
for i in '123': # valid, using a string
for i in [1, 2, 3]: # valid, using a list
for i in (1, 2, 3): # valid, using a tuple
for i in 1, 2, 3: # valid, using an implicit-tuple
# all valid, all different types
I strongly recommend that you read the Python Documentation and/or take the Python's Tutorial.
From Docs Glossary:
iterable
An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as list, str, and tuple) and some non-sequence types like dict, file objects, and objects of any classes you define with an __iter__() or __getitem__() method. Iterables can be used in a for loop and in many other places where a sequence is needed (zip(), map(), …). When an iterable object is passed as an argument to the built-in function iter(), it returns an iterator for the object. This iterator is good for one pass over the set of values. When using iterables, it is usually not necessary to call iter() or deal with iterator objects yourself. The for statement does that automatically for you, creating a temporary unnamed variable to hold the iterator for the duration of the loop. See also iterator, sequence, and generator.
More about iterables.
Error is given when u try to append strings.So better first take list and then convert list to string.Code :
n='qwerty'
for i in range(1):
temp=[]
temp.append(n[-3:])
temp.append('t')
newtemp=' '
newtemp=temp[i]+temp[i+1]
print(newtemp)
Output:rtyt