I had an old script that from a pandas dataframe calculates new columns from others, but also from the previous result of that column being calculated.
This script used for loops, and it was quite slow.
For this reason, I replaced the for loops with recursive functions.
The new script is around 100 times faster than the old one, which is good news. But I am now encountering a limit that I did not have before. As soon as I have more than 29952 rows in my dataset, I get the following error:
"Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)"
I made this little script with lists reflecting my problem :
If I increase the size of the lists (list_lenght) to more than 29952, the script crashes (on my computer)
import random
import sys
def list_generator(min_value, max_value, list_lenght):
return [random.randrange(min_value,max_value) for i in range(list_lenght)]
def recursive_function(list_1, list_2, n, result):
if n == len(list_1):
return result
elif list_1[n] <= list_2[n]:
result.append(1 + result[n - 1])
else:
result.append(0)
return recursive_function(list_1, list_2, (n + 1), result)
list_lenght = 29952 # How to increase this limit without generating an error?
min_value = 10
max_value = 20
list_one = list_generator(min_value, max_value, list_lenght)
list_two = list_generator(min_value, max_value, list_lenght)
# Set recursion limit
sys.setrecursionlimit(list_lenght * 2)
# Compute a new list from list_one and list_two
list_result = recursive_function(list_one, list_two, 1, [0])
I suspect a memory problem, but how do you take advantage of all the power of python's recursive functions while avoiding this limit as well as possible?
Thanks in advance
Following comment from #trincot, here is the version of the code without recursion function... which is ultimately faster than the version above with a recursive function ! And with which there are no more limits
def no_recursive_function(list_1, list_2, n, result):
if list_1[n] <= list_2[n]:
return 1 + result[n - 1]
else:
return 0
list_lenght = 29952
min_value = 10
max_value = 20
list_one = list_generator(min_value, max_value, list_lenght)
list_two = list_generator(min_value, max_value, list_lenght)
# Set recursion limit
sys.setrecursionlimit(list_lenght * 2)
list_result_2 = [0]
for n in range(list_lenght - 1):
result = no_recursive_function(list_one, list_two, n + 1, list_result_2)
list_result_2.append(result)
I'm trying to automate data collection from an SR245 Boxcar using Python 3.6 and the PyVisa library (version 1.11.1). 9/10 times, it works great. However, three times over the course of two days it has caused the entire computer to crash and reboot (running on Windows 10). This has resulted in a lot of data loss, and I'm trying to figure out what I'm doing wrong that is leading to the whole system crashing. Code is below (it is part of a larger program, but I also run this piece of code by itself, and it has caused crashes). The data_processing file is not shown, but the functions there are simple calculations (e.g. divide the values in a list by the values in another list, return the average value from a list of integers, etc.)
import pyvisa
from pyvisa.constants import SerialTermination
import time
import numpy as np
from data_processing import *
def connect_boxcar(pNum):
rm = pyvisa.ResourceManager()
port = "COM"+pNum
sr = rm.open_resource(port)
return sr
def config_boxcar(boxcar):
#Configure the boxcar settings
boxcar.write_termination = '\r'
boxcar.read_termination='\r'
boxcar.baud_rate=19200
boxcar.end_output = SerialTermination.termination_char
def preset_scan(boxcar):
#Reset boxcar settings
boxcar.write('MR')
boxcar.write('MS;ET;T1;I2;W0')
def scan(boxcar, num):
#Send the SCAN command to the boxcar, set to the specified number of data points
command = 'SC1,2:' + str(num)
boxcar.write(command)
def read_data(boxcar, num):
#Read the stored scan data and return it as a value list
data_list = []
for x in range(num * 2):
data_list.append(float(boxcar.query('N')))
return data_list
def collect_baseline(boxcar, n):
#Get a baseline signal for later processing
config_boxcar(boxcar)
preset_scan(boxcar)
scan(boxcar, n)
raw_data = read_data(boxcar, n)
chan1 = raw_data[::2]
chan2 = raw_data[1::2]
normal_data = normalize(chan1, chan2, n)
return average_list(normal_data)
def main():
rm = pyvisa.ResourceManager()
n = 10
sleep_timer = 0.1 * n + 0.5
sr245 = rm.open_resource('COM5')
#Configure/preset
config_boxcar(sr245)
preset_scan(sr245)
#Set a timer to measure scanning time
t0 = time.time()
scan(sr245, n)
time.sleep(sleep_timer)
raw_data = read_data(sr245, n)
t1 = time.time()
#Breakdown data by channel and normalize
chan1 = raw_data[::2]
chan2 = raw_data[1::2]
normal_data = normalize(chan1, chan2, n)
elapsed_time = t1 - t0
print('Elapsed time: ', elapsed_time)
print('Channel 1: ', chan1)
print('Channel 2: ', chan2)
print('Normalized Data: ', normal_data)
print('Average Normalized Data: ', average_list(normal_data))
print('Standard Deviation: ', np.std(normal_data))
if __name__ == '__main__':
main()
I've been working in Reaction-Diffusion cellular automata with the cellpylib library for a course in my university (I wrote it all in one script so you don't have to install/download anything). I'd like to save the evolution of the automata data to a csv file to run some statistics. That is, I'd like to save the data in columns where the first column is 'number of "1"' and the second column: 'time steps'.
Thus, I need help in:
(1) Creating a variable that saves the amount of '1' per time step (I think so).
(2) I need to export all that data to a csv file (number of "1" and the corresponding iteration, from 1 to time_steps in the code below).
The code is the following.
#Libraries
import matplotlib
matplotlib.matplotlib_fname()
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.animation as animation
import numpy as np
import csv
# Conditions
#############################
theta = 1 # this is the condition for Moore neighbourhood
Int = 100 # this is the iteration speed (just for visualization)
time_steps = 100 # Iterations
size = 8 # this is the size of the matrix (8x8)
#############################
# Definitions
def plot2d_animate(ca, title=''):
c = mpl.colors.ListedColormap(['green', 'red', 'black', 'gray'])
n = mpl.colors.Normalize(vmin=0,vmax=3)
fig = plt.figure()
plt.title(title)
im = plt.imshow(ca[0], animated=True, cmap=c, norm=n)
i = {'index': 0}
def updatefig(*args):
i['index'] += 1
if i['index'] == len(ca):
i['index'] = 0
im.set_array(ca[i['index']])
return im,
ani = animation.FuncAnimation(fig, updatefig, interval=Int, blit=True)
plt.show()
def init_simple2d(rows, cols, val=1, dtype=np.int):
x = np.zeros((rows, cols), dtype=dtype)
x[x.shape[0]//2][x.shape[1]//2] = val
return np.array([x])
def evolve2d(cellular_automaton, timesteps, apply_rule, r=1, neighbourhood='Moore'):
_, rows, cols = cellular_automaton.shape
array = np.zeros((timesteps, rows, cols), dtype=cellular_automaton.dtype)
array[0] = cellular_automaton
von_neumann_mask = np.zeros((2*r + 1, 2*r + 1), dtype=bool)
for i in range(len(von_neumann_mask)):
mask_size = np.absolute(r - i)
von_neumann_mask[i][:mask_size] = 1
if mask_size != 0:
von_neumann_mask[i][-mask_size:] = 1
def get_neighbourhood(cell_layer, row, col):
row_indices = [0]*(2*r+1)
for i in range(-r,r+1):
row_indices[i+r]=(i+row) % cell_layer.shape[0]
col_indices = [0]*(2*r+1)
for i in range(-r,r+1):
col_indices[i+r]=(i+col) % cell_layer.shape[1]
n = cell_layer[np.ix_(row_indices, col_indices)]
if neighbourhood == 'Moore':
return n
elif neighbourhood == 'von Neumann':
return np.ma.masked_array(n, von_neumann_mask)
else:
raise Exception("unknown neighbourhood type: %s" % neighbourhood)
for t in range(1, timesteps):
cell_layer = array[t - 1]
for row, cell_row in enumerate(cell_layer):
for col, cell in enumerate(cell_row):
n = get_neighbourhood(cell_layer, row, col)
array[t][row][col] = apply_rule(n, (row, col), t)
return array
def ca_reaction_diffusion(neighbourhood, c, t):
center_cell = neighbourhood[1][1]
total = np.sum(neighbourhood==1)
if total >= theta and center_cell==0:
return 1
elif center_cell == 1:
return 2
elif center_cell == 2:
return 3
elif center_cell == 3:
return 0
else:
return 0
# Initial condition
cellular_automaton = init_simple2d(size, size, val=0, dtype=int)
# Excitable initial cells
cellular_automaton[:, [1,2], [1,1]] = 1
# The evolution
cellular_automaton = evolve2d(cellular_automaton,
timesteps=time_steps,
neighbourhood='Moore',
apply_rule=ca_reaction_diffusion)
animation=plot2d_animate(cellular_automaton)
Explanation of the code:
As you can see, there are 4 states: 0 (green), 1 (red), 2 (black) and 3 (gray). The way the automata evolves is with the cellular_automaton conditions. That is, for example, if a center cell has a value of 0 (excitable cell) and at least one cell (theta value) on its Moore neighbourhood is in state 1, in the following time step the same cell will be at state 1 (excited).
To notice:
The configuration of this matrix is toroidal, and the definitions are taken from the cellpylib library.
I've been stuck with this for over a week, so I'd really appreciate some help. Thanks in advance!
I am not well-experienced in this subject matter (and I was not fully clear on what you intended for me to do). I went through and implemented the counting of the specific "0", "1", "2" and "3" value cells in "evolve2d" function. This code should be viewed as "starter code"; whatever specifically you are trying to do should piggyback off of what I have given you. Additionally, this task could have been accomplished through some better code design and definitely, better planning of your function locations (as part of better coding practice and overall cleaner code that is easy to debug). Please peruse and UNDERSTAND the changes that I made.
#Libraries
import matplotlib
matplotlib.matplotlib_fname()
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.animation as animation
import numpy as np
import csv
# Conditions
#############################
theta = 1 # this is the condition for Moore neighbourhood
iter_speed = 100 # this is the iteration speed (just for visualization)
time_steps = 100 # Iterations
size = 8 # this is the size of the matrix (8x8)
#############################
# Definitions
def plot2d_animate(ca, title=''):
c = mpl.colors.ListedColormap(['green', 'red', 'black', 'gray'])
n = mpl.colors.Normalize(vmin=0,vmax=3)
fig = plt.figure()
plt.title(title)
im = plt.imshow(ca[0], animated=True, cmap=c, norm=n)
i = {'index': 0}
def updatefig(*args):
i['index'] += 1
if i['index'] == len(ca):
i['index'] = 0
im.set_array(ca[i['index']])
return im,
ani = animation.FuncAnimation(fig, updatefig, interval=iter_speed, blit=True)
plt.show()
#############I ADDED EXTRA ARGUMENTs FOR THE FUNCTION BELOW
def get_neighbourhood(cell_layer, row, col, r = 1, neighbourhood = "Moore"):
row_indices = [0]*(2*r+1)
for i in range(-r,r+1):
row_indices[i+r]=(i+row) % cell_layer.shape[0]
col_indices = [0]*(2*r+1)
for i in range(-r,r+1):
col_indices[i+r]=(i+col) % cell_layer.shape[1]
n = cell_layer[np.ix_(row_indices, col_indices)]
if neighbourhood == 'Moore':
return n
elif neighbourhood == 'von Neumann':
return np.ma.masked_array(n, von_neumann_mask)
else:
raise Exception("unknown neighbourhood type: %s" % neighbourhood)
def init_simple2d(rows, cols, val=1, dtype=np.int):
x = np.zeros((rows, cols), dtype=dtype)
x[x.shape[0]//2][x.shape[1]//2] = val
return np.array([x])
#Inner functions was moved due to bad coding practice. Arguments were also changed. Make sure you understand what I did.
def evolve2d(cellular_automaton, timesteps, apply_rule, r=1, neighbourhood='Moore'):
_, rows, cols = cellular_automaton.shape
array = np.zeros((timesteps, rows, cols), dtype=cellular_automaton.dtype)
array[0] = cellular_automaton
von_neumann_mask = np.zeros((2*r + 1, 2*r + 1), dtype=bool)
for i in range(len(von_neumann_mask)):
mask_size = np.absolute(r - i)
von_neumann_mask[i][:mask_size] = 1
if mask_size != 0:
von_neumann_mask[i][-mask_size:] = 1
#################################################
#These lists keep track of values over the course of the function:
Result_0 = ["Number of 0"]
Result_1 = ["Number of 1"]
Result_2 = ["Number of 2"]
Result_3 = ["Number of 3"]
#################################################
for t in range(1, timesteps):
#################################################
#This dictionary keeps track of values per timestep
value_iter_tracker = {0: 0, 1: 0, 2: 0, 3: 0 }
#################################################
cell_layer = array[t - 1]
for row, cell_row in enumerate(cell_layer):
for col, cell in enumerate(cell_row):
n = get_neighbourhood(cell_layer, row, col)
################################################
res = apply_rule(n, (row, col), t)
value_iter_tracker[res]+=1
array[t][row][col] = res
################################################
print(value_iter_tracker)
########################################################
#Now we need to add the results of the iteration dictionary to the corresponding
#lists in order to eventually export to the csv
Result_0.append(value_iter_tracker[0])
Result_1.append(value_iter_tracker[1])
Result_2.append(value_iter_tracker[2])
Result_3.append(value_iter_tracker[3])
########################################################
############################################################
#function call to export lists to a csv:
timesteps_result = list(range(1, timesteps))
timesteps_result = ["Time Step"] + timesteps_result
#If you don't understand what is going on here, put print statement and/or read docs
vals = zip(timesteps_result, Result_0, Result_1, Result_2, Result_3)
write_to_csv_file(list(vals))
############################################################
return array
################################################################################
#THIS CODE IS FROM:
#https://stackoverflow.com/questions/14037540/writing-a-python-list-of-lists-to-a-csv-file
import pandas as pd
def write_to_csv_file(data):
data = [list(x) for x in data]
my_df = pd.DataFrame(data)
my_df.to_csv('output1.csv', index=False, header=False)
################################################################################
def ca_reaction_diffusion(neighbourhood, c, t):
center_cell = neighbourhood[1][1]
total = np.sum(neighbourhood==1)
if total >= theta and center_cell==0:
return 1
elif center_cell == 1:
return 2
elif center_cell == 2:
return 3
elif center_cell == 3:
return 0
else:
return 0
# Initial condition
cellular_automaton = init_simple2d(size, size, val=0, dtype=int)
# Excitable initial cells
cellular_automaton[:, [1,2], [1,1]] = 1
# The evolution
cellular_automaton = evolve2d(cellular_automaton,
timesteps=time_steps,
neighbourhood='Moore',
apply_rule=ca_reaction_diffusion)
animation=plot2d_animate(cellular_automaton)
I have left comments that should clarify the changes that I made. Essentially, when you call the evolve2d function, a csv file called "output1.csv" is created with the timestep results. I used the pandas package to write the data into a csv but other methods could have been used as well. I will leave it to you to take advantage of the changes that I made for your use. Hope this helps.
The code is being executed but the output is not shown nor the variables are created
import numpy as np
def magicsquares():
n=input('enter the order of squares')
n=int(n)
m=np.zeros((n,n))
s=n*(n**2+1)/2 #sum of each row or diagonal
p=int(n/2)
q=(n-1)
for i in range(n**2):
m[p][q]=1 #assigning postion of 1
P=p-1
Q=q+1
if(i>=2): #assigning remaining positions
if(P==-1):
P=n-1
if(Q==n):
Q=0
there is not output showing because you are just declaring the function but not calling the function and there is no print/return inside the function. Here is a solution which you can use to see the output and work on:
import numpy as np
def magicsquares():
n = input('enter the order of squares')
n = int(n)
m = np.zeros((n, n))
s = n*(n**2+1)/2 # sum of each row or diagonal
p = int(n/2)
q = (n-1)
for i in range(n**2):
m[p][q] = 1 # assigning postion of 1
P = p-1
Q = q+1
if i >= 2: # assigning remaining positions
if P == -1:
P = n-1
if Q == n:
Q = 0
print(m)
magicsquares()
It is not the ultimate solution to find magic_square. It's just an updated version of your code so that you can see the outputs and work on.
I need to write a function which:
Receives - two numpy.array objects
Returns - the floating-point dot product of the two input
numpy arrays
Not allowed to use:
numpy.dot()
loops of any kind
Any suggestions?
A possible solution makes use of recursion
import numpy as np
def multiplier (first_vector, second_vector, size, index, total):
if index < size:
addendum = first_vector[index]*second_vector[index]
total = total + addendum
index = index + 1
# ongoing job
if index < size:
multiplier(first_vector, second_vector, size, index, total)
# job done
else:
print("dot product = " + str(total))
def main():
a = np.array([1.5, 2, 3.7])
b = np.array([3, 4.3, 5])
print(a, b)
i = 0
total_sum = 0
# check needed if the arrays are not hardcoded
if a.size == b.size:
multiplier(a, b, a.size, i, total_sum)
else:
print("impossible dot product for arrays with different size")
if __name__== "__main__":
main()
Probably considered cheating, but Python 3.5 added a matrix multiply operator that numpy uses to compute the dot product without actually calling np.dot:
>>> arr1 = np.array([1,2,3])
>>> arr2 = np.array([3,4,5])
>>> arr1 # arr2
26
Problem solved!