I have some data that I want to process before feeding/training a model. For this example I want to do a max pool 2d. I wrote a short function to do that with tensorflow.
import tensorflow
import tensorflow.nn as nn
def _tfMaxPool(arr, pool=(4,4), sess=None):
op = nn.max_pool(arr, (1, 1, pool[0], 1), (1, 1, pool[0], 1 ), padding="VALID")
op = nn.max_pool(op, (1, 1, 1, pool[1]), (1, 1, 1, pool[1]), padding="VALID")
if sess is None:
sess = tensorflow.Session();
return sess.run(op)
The problem is this can add nodes to my graph each time, which seems to clutter my session. One alternative way is to create a model.
import keras
seq = keras.Sequential([
keras.layers.InputLayer((1, 512, 512)),
keras.layers.MaxPool2D((4, 4), (4, 4), data_format="channels_first")
])
def _tfMaxPool2(arr, pool=(4,4), sess=None):
swapped = arr.swapaxes(0,1)
return seq.predict(swapped).swapaxes(0,1)
The model is nearly exactly like what I want, but I think I am missing something fundamental.
Why don't you reuse your graph with various input? In the following code,tfMaxpool is defined only once.
def _tfMaxPool(arr, pool=(4,4)):
op = nn.max_pool(arr, (1, 1, pool[0], 1), (1, 1, pool[0], 1 ), padding="VALID")
op = nn.max_pool(op, (1, 1, 1, pool[1]), (1, 1, 1, pool[1]), padding="VALID")
return op
input = tf.placeholder()
output = _tfMaxPool(input)
with tf.Session() as sess:
sess.run(output, feed_dict={input:arr1})
sess.run(output, feed_dict={input:arr2})
Related
Hello dear users of neuorlab, I want to change the activation function nodes of the hidden layer to ReLU and keep Linear function in output nodes
import numpy as np
import neurolab as nl
# Create train samples
input = np.random.uniform(-1, 1, (5, 2))
target = (input[:, 0] + input[:, 1]).reshape(5, 1)
net = nl.net.newff([[-1, 1]]*2, [4, 1])
# What I try to do
import numpy as np
import neurolab as nl
# Create train samples
input = np.random.uniform(-1, 1, (5, 2))
target = (input[:, 0] + input[:, 1]).reshape(5, 1)
net = nl.net.newff([[-1, 1]]*2, [4, 1],[nl.trans.PoseLin(), nl.trans.PureLin()])
Having this frustrating issue where i want to pass through the tuples in the following list
through a method on another list of instances of a class that i have created
list_1=[(0, 20), (10, 1), (0, 1), (0, 10), (5, 5), (10, 50)]
instances=[instance[0], instance[1],...instance[n]]
results=[]
pos_list=[]
for i in range(len(list_1)):
a,b=List_1[i]
result=sum(instance.method(a,b) for instance in instances)
results.append(result)
if result>=0:
pos_list.append((a,b))
print(results)
print(pos_list)
the issue is that all instances are taking the same tuple, where as i want the method on the first instance to take the first tuple and so on.
I ultimately want to see it append to the new list (pos_list) if the sum is >0.
Anyone know how i can iterate this properly?
EDIT
It will make it clearer if I print the result of the sum also.
Basically I want the sum to perform as follows:
result = instance[0].method(0,20), instance[1].method(10,1), instance[2].method(0,1), instance[3].method(0,10), instance[4].method(5,5), instance[5].method(10,50)
For info the method is just the +/- product of the two values depending on the attributes of the instance.
So results for above would be:
result = [0*20 - 10*1 - 0*1 + 0*10 - 5*5 + 10*50] = [465]
pos_list=[(0, 20), (10, 1), (0, 1), (0, 10), (5, 5), (10, 50)]
except what is actually doing is using the same tuple for all instances like this:
result = instance[0].method(0,20), instance[1].method(0,20), instance[2].method(0,20), instance[3].method(0,20), instance[4].method(0,20), instance[5].method(0,20)
result = [0*20 - 0*20 - 0*20 + 0*20 - 0*20 + 0*20] = [0]
pos_list=[]
and so on for (10,1) etc.
How do I make it work like the first example?
You can compute your sum using zip to generate all the pairs of correspondent instances and tuples.
result=sum(instance.payout(*t) for instance, t in zip(instances, List_1))
The zip will stop as soon as it reaches the end of the shortest of the two iterators. So if you have 10 instances and 100 tuples, zip will produce only 10 pairs, using the first 10 elements of both lists.
The problem I see in your code is that you are computing this sum for each element of List_1, so if payout produces always the same result with the same inputs (e.g., it has no memory or randomness), the value of result will be the same at each iteration. So, in the end, results will be composed by the same value repeated a number of times equal to the length of List_1, while pos_list will contain all (the sum is greater than 0) or none (the sum is less or equal to zero) of the input tuples.
Instead, it would make sense if items of List_1 were lists or tuples themselves:
List_1 = [
[(0, 1), (2, 3), (4, 5)],
[(6, 7), (8, 9), (10, 11)],
[(12, 13), (14, 15), (16, 17)],
]
So, in this case, supposing that your class for instances is something like this:
class Goofy:
def __init__(self, positive_sum=True):
self.positive_sum = positive_sum
def payout(self, *args):
if self.positive_sum:
return sum(args)
else:
return -1 * sum(args)
instances = [Goofy(i) for i in [True, True, False]]
you can rewrite your code in this way:
results=[]
pos_list=[]
for el in List_1:
result = sum(g.payout(*t) for g, t in zip(instances, el))
results.append(result)
if result >= 0:
pos_list.append(el)
Running the previous code, results will be:
[-3, 9, 21]
while pop_list:
[[(6, 7), (8, 9), (10, 11)], [(12, 13), (14, 15), (16, 17)]]
If you are interested only in pop_list, you can compact your code in only one line:
pop_list = list(filter(lambda el: sum(g.payout(*t) for g, t in zip(instances, el)) > 0, List_1))
many thanks for the above! I have it working now.
Wasn't able to use args given my method had a bit more to it but the use of zip is what made it click
import random
rand=random.choices(list_1, k=len(instances))
results=[]
pos_list=[]
for r in rand:
x,y=r
result=sum(instance.method(x,y) for instance,(x,y) in zip(instances, rand))
results.append(result)
if result>=0:
pos_list.append(rand)
print(results)
print(pos_list)
for list of e.g.
rand=[(20, 5), (0, 2), (0, 100), (2, 50), (5, 10), (50, 100)]
this returns the following
results=[147]
pos_list=[(20, 5), (0, 2), (0, 100), (2, 50), (5, 10), (50, 100)]
so exactly what I wanted. Thanks again!
I wanted to find a better way to loop through orthodiagonal indices in order, I am currently using numpy but I think I'm making an unnecessary number of function calls.
import numpy as np
len_x, len_y = 50, 50 #they don't have to equal
index_arr = np.add.outer(np.arange(len_x), np.arange(len_y))
Currently, I am looping through like this:
for i in range(np.max(index_arr)):
orthodiag_indices = zip(*np.where(index_arr == i))
for index in orthodiag_indices:
# DO FUNCTION OF index #
I have an arbitrary function of the index tuple, index and other parameters outside of this loop. It feels like I don't need the second for loop, and I should be able to do the whole thing in one loop. On top of this, I'm making a lot of function calls from zip(*np.where(index_arr == i)) for every i. What's the most efficient way to do this?
Edit: should mention that it's important that the function applies to index_arr == i in order, i.e., it does 0 first, then 1, then 2 etc. (the order of the second loop doesn't matter).
Edit 2: I guess what I want is a way to get the indices [(0,0), (0,1), (1,0), (2,0), (1,1), (2,0), ...] efficiently. I don't think I can apply a vectorized function because I am populating an np.zeros((len_x, len_y)) array, and going back to the first edit, the order matters.
You could use tril/triu_indices. Since the order of the (former) inner loop doesn't matter dimensions can be swapped as needed, I'll assume L>=S:
L,S = 4,3
a0,a1 = np.tril_indices(L,0,S)
b0,b1 = np.triu_indices(S,1)
C0 = np.concatenate([a0-a1,b0+L-b1])
C1 = np.concatenate([a1,b1])
*zip(C0,C1),
# ((0, 0), (1, 0), (0, 1), (2, 0), (1, 1), (0, 2), (3, 0), (2, 1), (1, 2), (3, 1), (2, 2), (3, 2))
I think itertools.product() will be of use here
import itertools as it
x,y = 2,3
a=list(it.product(range(x),range(y))
which gives a as
[(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2)]
If you need them in order then,
b=np.argsort(np.sum(a,1))
np.array(a)[b]
which gives,
array([[0, 0],
[0, 1],
[1, 0],
[0, 2],
[1, 1],
[1, 2]])
Hope that helps!
In Python is there a way to get all 3 pairs or n pairs of a list from a list?
For example list = [1,2,3,4]
result : [[1,2,3],[2,3,4],[1,2,4]]
I want to find all possible n pairs of lists from a Python list. I do not want to import any other functions like itertools.
You can use the module itertools. It comes inside Python by default (you don't need to install it throught third party modules):
>>> import itertools
>>> print(itertools.permutations([1,2,3,4], 3))
[(1, 2, 3), (1, 2, 4), (1, 3, 2), (1, 3, 4), (1, 4, 2), (1, 4, 3), (2, 1, 3), (2, 1, 4), (2, 3, 1), (2, 3, 4), (2, 4, 1), (2, 4, 3), (3, 1, 2), (3, 1, 4), (3, 2, 1), (3, 2, 4), (3, 4, 1), (3, 4, 2), (4, 1, 2), (4, 1, 3), (4, 2, 1), (4, 2, 3), (4, 3, 1), (4, 3, 2)]
This itertools.permutations(iterable, r=None) produces all the possible permutations of a given iterable element (like a list). If r is not specified or is None, then r defaults to the length of the iterable and all possible full-length permutations are generated.
If you are looking only for n pair of permutations instead of all the possible permutations just delete the rest of them:
>>> print(list(itertools.permutations([1,2,3,4], 3))[:3])
[(1, 2, 3), (1, 2, 4), (1, 3, 2)]
As you asked in comments you can do that without importing any module. itertools.permutations is just a function, which you can make by yourself:
def permutations(iterable, r=None):
pool = tuple(iterable)
n = len(pool)
r = n if r is None else r
if r > n:
return
indices = list(range(n))
cycles = list(range(n, n-r, -1))
yield tuple(pool[i] for i in indices[:r])
while n:
for i in reversed(range(r)):
cycles[i] -= 1
if cycles[i] == 0:
indices[i:] = indices[i+1:] + indices[i:i+1]
cycles[i] = n - i
else:
j = cycles[i]
indices[i], indices[-j] = indices[-j], indices[i]
yield tuple(pool[i] for i in indices[:r])
break
else:
return
But I strongly advise your importing it. If you don't want to import the whole module, just this function simply do from itertools import permutations.
I used to python 3.5 and Based on gensim samples I created a project and added these codes in my project:
class MyCorpus(object):
def __iter__(self):
for line in open('files/2/mycorpus.txt'):
# assume there's one document per line, tokens separated by whitespace
yield dictionary.doc2bow(line.lower().split())
corpus_memory_friendly = MyCorpus() # doesn't load the corpus into memory!
print(corpus_memory_friendly)
But after running I have these error in my pycharm console :
Traceback (most recent call last):
File "D:/Python-Workspace(s)/GensimSamples/2.Gensim_CorpusStreaming.py", line 31, in <module>
for vector in corpus_memory_friendly: # load one vector into memory at a time
File "D:/Python-Workspace(s)/GensimSamples/2.Gensim_CorpusStreaming.py", line 17, in __iter__
yield dictionary.doc2bow(line.lower().split())
AttributeError: module 'gensim.corpora.dictionary' has no attribute 'doc2bow'
How can I solve this issue?
We just need to have the dictionary prepared beforehand and make it available for the class, MyCorpus. A sample class that creates a memory friendly corpus could be:
import logging
from pprint import pprint
from six import iteritems
from gensim import corpora
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
class MyCorpus(object):
def __init__(self, text_file='text_corpus.txt', dictionary=None):
"""
Checks if a dictionary has been given as a parameter.
If no dictionary has been given, it creates one and saves it in the disk.
"""
self.file_name = text_file
if dictionary is None:
self.prepare_dictionary()
else:
self.dictionary = dictionary
def __iter__(self):
for line in open(self.file_name):
# assume there's one document per line, tokens separated by whitespace
yield self.dictionary.doc2bow(line.lower().split())
def prepare_dictionary(self):
stop_list = set('for a of the and to in'.split()) # List of stop words which can also be loaded from a file.
# Creating a dictionary using stored the text file and the Dictionary class defined by Gensim.
self.dictionary = corpora.Dictionary(line.lower().split() for line in open(self.file_name))
# Collecting the id's of the tokens which exist in the stop-list
stop_ids = [self.dictionary.token2id[stop_word] for stop_word in stop_list if
stop_word in self.dictionary.token2id]
# Collecting the id's of the token which appear only once
once_ids = [token_id for token_id, doc_freq in iteritems(self.dictionary.dfs) if doc_freq == 1]
# Removing the unwanted tokens using collected id's
self.dictionary.filter_tokens(stop_ids + once_ids)
# Saving dictionary in the disk for later use:
self.dictionary.save('dictionary.dict')
my_memory_fiendly_corpus = MyCorpus()
# Saving the corpus
# corpora.MmCorpus.serialize('corpus.mm', my_memory_fiendly_corpus)
# To load the saved corpus:
# corpus = corpora.MmCorpus('corpus.mm')
print('\t:::The dictionary::::')
pprint(my_memory_fiendly_corpus.dictionary.token2id)
print(my_memory_fiendly_corpus)
print('\n\t:::The corpus::::')
for vector in my_memory_fiendly_corpus:
print(vector)
Output (Without the log information):
:::The dictionary::::
{'computer': 2,
'eps': 8,
'graph': 10,
'human': 0,
'interface': 1,
'minors': 11,
'response': 6,
'survey': 3,
'system': 5,
'time': 7,
'trees': 9,
'user': 4}
<__main__.MyCorpus object at 0x7fe0e9ac5c18>
:::The corpus::::
[(0, 1), (1, 1), (2, 1)]
[(2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)]
[(1, 1), (4, 1), (5, 1), (8, 1)]
[(0, 1), (5, 2), (8, 1)]
[(4, 1), (6, 1), (7, 1)]
[(9, 1)]
[(9, 1), (10, 1)]
[(9, 1), (10, 1), (11, 1)]
[(3, 1), (10, 1), (11, 1)]
As I am quite new to both Gensim and Python, I have faced the similar kinds of problems too. And this mailing-list has been really helpful for learning Gensim.