Using python extract from list of string - python-3.x

I have list of string from which i want to extract channel number value with low SIG value.
Below is mt list name "Outlines"
Flags: A - active, P - privacy, R - routeros-network, N - nstreme, T - tdma,
W - wds, B - bridge
ADDRESS SSID CHANNEL SIG NF SNR RADIO-... ROUTER...
AP 20:B5:C6:F1:B6:B0 INAPHDBD... 5815/2... -78 -60 -18
52:4F:54:43:4F:44 P2MPRWXX... 5835/20/a -80 -102 22
A 52:41:44:57:49:4E 5835/20/a -86 -102 16
APR B 4C:5E:0C:BF:EE:6E iBw 5865/2... -75 -102 27 4C5E0C... 6.38.5
A 00:19:70:2C:FD:82 TR6SL5 5835/20/a -86 -102 16
20:B5:C6:F0:E6:F7 5855/20/a -58 -103 45
Below is the code i tried, but didnt know how i can iterate when line starts without blank or any other random value.
RSSI = ''
Myindex = [l for l, elem in enumerate (outlines) if 'AP' in elem]
#Myindex = [elem for elem in outlines if 'AP' in elem]
#RSSI =int('nan')
for ind in Myindex:
newchannel = " ".join(outlines[ind].split()).split(' ')[3]
newRSSI = " ".join(outlines[ind].split()).split(' ')[4]
if RSSI < newRSSI or RSSI == '':
RSSI =newRSSI
channel = newchannel.split('/')[0]
print(channel)

You can get the positions in each line you need to parse from the line that contains 'SIG' and 'CHANNEL'.
You can loop all lines, extract the positions and convert the SIG to a number and put it into a dictionary of lists of CHANNELS (if 2 have the same SIG and both are lowest).
You can proceed to work with the dictionary data:
t = """Flags: A - active, P - privacy, R - routeros-network, N - nstreme, T - tdma,
W - wds, B - bridge
ADDRESS SSID CHANNEL SIG NF SNR RADIO-... ROUTER...
AP 20:B5:C6:F1:B6:B0 INAPHDBD... 5815/2... -78 -60 -18
52:4F:54:43:4F:44 P2MPRWXX... 5835/20/a -80 -102 22
A 52:41:44:57:49:4E 5835/20/a -86 -102 16
APR B 4C:5E:0C:BF:EE:6E iBw 5865/2... -75 -102 27 4C5E0C... 6.38.5
A 00:19:70:2C:FD:82 TR6SL5 5835/20/a -86 -102 16
20:B5:C6:F0:E6:F7 5855/20/a -58 -103 45
"""
sigstart = 0
channelstart = 0
data = {}
for line in t.split("\n"):
# find position of SIG and CHANNEL, ignore everything until we have that
if sigstart == 0:
try:
sigstart = line.index("SIG")
channelstart = line.index("CHANNEL")
except ValueError:
pass
continue
# parse line if long enough and has AP in it
if len(line)>= sigstart+3 and "AP" in line:
sig = int(line[sigstart:sigstart+3].strip())
channel = line[channelstart:sigstart].strip()
# add to dictionary - could use defaultdict if perfomance is an issue
# but for the example this works just fine
data.setdefault(sig,[]).append(channel)
# output all parsed data, sorted:
for sig,channel in sorted(data.items(), key = lambda x:x[0]):
print(sig, ', '.join(c.split("/")[0] for c in channel))
Output:
-78 5815 # lowerst, only 1 item
-75 5865
The very lowest would be sorted(data.items(), key = lambda x:x[0])[0].

I figure out to extract the channel value having lowest SIG value using regex, regex are very power to use specially for a nasty data(list of strings).
i used regex101 . com which makes life easier. below is the code i tried.
import re
outlines = """
Flags: A - active, P - privacy, R - routeros-network, N - nstreme, T - tdma,
W - wds, B - bridge
ADDRESS SSID CHANNEL SIG NF SNR RADIO-... ROUTER...
AP 20:b5:C6:F1:B6:B0 INAPHDBD... 5815/2... -78 -60 -18
52:4F:54:43:4F:44 P2MPRWXX... 5835/20/a -80 -102 22
A 52:41:44:57:49:4E 5835/20/a -86 -102 16
APR B 4C:5E:0C:BF:EE:6E iBw 5865/2... -75 -102 27 4C5E0C... 6.38.5
A 00:19:70:2C:FD:82 TR6SL5 5835/20/a -86 -102 16
20:b5:C6:F0:E6:F7 5855/20/a -58 -103 45
"""
regex = ".* ([0-9]+)\/[^ ]* ([-+0-9]*)[ ,]*"
oldsig =''
if __name__ == '__main__':
​
for line in outlines:
s = re.search(regex, line.replace('\r\n',''))
​
if s:
channel = s.group(1)
sig = s.group(2)
if oldsig < sig or oldsig=='':
oldsig = sig
nchannel1 = channel
print(nchannel1)
Which gives output
5835

Related

Obtain decimal netmask from prefix length python 3.x

I created this code because I was not able to find any functional that accomplishes my requirement.
If you can reduce it will be better.
Just enter de prefix lenght from 1 to 32 and you will get the decimal mask.
This code help me with my scripts for cisco.
import math
#Netmask octets
octet1 = [0,0,0,0,0,0,0,0]
octet2 = [0,0,0,0,0,0,0,0]
octet3 = [0,0,0,0,0,0,0,0]
octet4 = [0,0,0,0,0,0,0,0]
#POW list
pow_list = [7,6,5,4,3,2,1,0]
#Introduce prefix lenght
mask = int(input("Introduce the prefix lenght: "))
#According to the number of bits we will change the array elements from 0 to 1
while mask >= 25 and mask <= 32:
octet4[mask-25] = 1
mask -= 1
while mask >= 17 and mask <= 24:
octet3[mask-17] = 1
mask -= 1
while mask >= 9 and mask <= 16:
octet2[mask-9] = 1
mask -= 1
while mask >= 1 and mask <= 8:
octet1[mask-1] = 1
mask -= 1
#Obtain the number of ones
ones1 = octet1.count(1)
ones2 = octet2.count(1)
ones3 = octet3.count(1)
ones4 = octet4.count(1)
#Summary and reuslt of each octet.
sum1 = 0
for i in range(0,ones1):
sum1 = sum1 + math.pow(2,pow_list[i])
sum1 = int(sum1)
sum2 = 0
for i in range(0,ones2):
sum2 = sum2 + math.pow(2,pow_list[i])
sum2 = int(sum2)
sum3 = 0
for i in range(0,ones3):
sum3 = sum3 + math.pow(2,pow_list[i])
sum3 = int(sum3)
sum4 = 0
for i in range(0,ones4):
sum4 = sum4 + math.pow(2,pow_list[i])
sum4 = int(sum4)
#Join the results with a "."
decimal_netmask = str(sum1) + "." + str(sum2) + "." + str(sum3) + "." + str(sum4)
#Result
print("Decimal netmask is: "+ decimal_netmask)
Result:
Introduce the prefix lenght: 23
Decimal netmask is: 255.255.254.0
As you are probably doing more than just converting CIDR to netmask, I recommend checking out the built-in library ipaddress
from ipaddress import ip_network
cidr = input("Introduce the prefix length: ")
decimal_netmask = str(ip_network(f'0.0.0.0/{cidr}').netmask)
You can simplify your code by computing the overall mask value as an integer using the formula:
mask = 2**32 - 2**(32-prefix_length)
Then you can compute the 4 8-bit parts of the mask (by shifting and masking), appending the results to a list and then finally joining each element of the list with .:
def decimal_netmask(prefix_length):
mask = 2**32 - 2**(32-prefix_length)
octets = []
for _ in range(4):
octets.append(str(mask & 255))
mask >>= 8
return '.'.join(reversed(octets))
for pl in range(33):
print(f'{pl:3d}\t{decimal_netmask(pl)}')
Output:
0 0.0.0.0
1 128.0.0.0
2 192.0.0.0
3 224.0.0.0
4 240.0.0.0
5 248.0.0.0
6 252.0.0.0
7 254.0.0.0
8 255.0.0.0
9 255.128.0.0
10 255.192.0.0
11 255.224.0.0
12 255.240.0.0
13 255.248.0.0
14 255.252.0.0
15 255.254.0.0
16 255.255.0.0
17 255.255.128.0
18 255.255.192.0
19 255.255.224.0
20 255.255.240.0
21 255.255.248.0
22 255.255.252.0
23 255.255.254.0
24 255.255.255.0
25 255.255.255.128
26 255.255.255.192
27 255.255.255.224
28 255.255.255.240
29 255.255.255.248
30 255.255.255.252
31 255.255.255.254
32 255.255.255.255

" ArityMismatch: Adding expressions with non-matching form arguments () vs ('v_1',) " using FEniCS

I want to solve a continuum mechanics problem thanks to FEniCS. I apply pressure and take into account the weight. But when I add the thermoelasticity component, it doesn't work anymore.
Here is my code :
from dolfin import *
from fenics import *
from ufl import nabla_div
from ufl import as_tensor
import matplotlib.pyplot as plt
import numpy as np
E = Constant(100*10**9)
nu = Constant(0.3)
Lg = 0.01; W = 0.2
mu = E/(2+2*nu)
rho = Constant(2200)
delta = W/Lg
gamma = 0.4*delta**2
beta = 8
lambda_ = (E*nu)/((1+nu)*(1-2*nu))
alpha = 1.2*(10**(-8))
deltaT = Constant(50)
Kt = E*alph*deltaT/(1-2*nu)
g = 9.81
tol = 1E-14
# Create mesh and define function space
mesh = RectangleMesh(Point(-2., 0.),Point(2., 10.), 80, 200)
V = VectorFunctionSpace(mesh, "P", 1)
# Define boundary condition
def clamped_boundary(x, on_boundary):
return on_boundary and x[1] < tol
class UpFace(SubDomain):
def inside(self, x, on_boundary):
return on_boundary and (x[1] > 10 - tol)
ueN = UpFace()
boundaries = MeshFunction("size_t", mesh, mesh.topology().dim()-1, 0)
ueN.mark(boundaries, 1)
ds = Measure("ds", domain=mesh, subdomain_data=boundaries)
bc = DirichletBC(V, Constant((0, 0)), clamped_boundary)
def epsilon(u):
return 0.5*(nabla_grad(u) + nabla_grad(u).T)
def sigma(u):
return (lambda_*nabla_div(u) - Kt)*Identity(d) + (2*mu)*epsilon(u)
# Define variational problem
u = TrialFunction(V)
d = u.geometric_dimension() # space dimension
v = TestFunction(V)
f = Constant((0,-rho*g))
T = Constant((0, 0))
Pr = Constant((0, -2*10**9))
a = inner(sigma(u), epsilon(v))*dx
L = dot(f, v)*dx + dot(T, v)*ds + dot(Pr,v)*ds(1)
# Compute solution
u = Function(V)
solve(a == L, u, bc)
# Plot solution
plot(u, mode="displacement", color= "red")
plt.colorbar(plot(u))
I get this error message :
---------------------------------------------------------------------------
ArityMismatch Traceback (most recent call last)
<ipython-input-54-805d7c5b704f> in <module>
17 # Compute solution
18 u = Function(V)
---> 19 solve(a == L, u, bc)
20
21 # Plot solution
/usr/lib/python3/dist-packages/dolfin/fem/solving.py in solve(*args, **kwargs)
218 # tolerance)
219 elif isinstance(args[0], ufl.classes.Equation):
--> 220 _solve_varproblem(*args, **kwargs)
221
222 # Default case, just call the wrapped C++ solve function
/usr/lib/python3/dist-packages/dolfin/fem/solving.py in _solve_varproblem(*args, **kwargs)
240 # Create problem
241 problem = LinearVariationalProblem(eq.lhs, eq.rhs, u, bcs,
--> 242 form_compiler_parameters=form_compiler_parameters)
243
244 # Create solver and call solve
/usr/lib/python3/dist-packages/dolfin/fem/problem.py in __init__(self, a, L, u, bcs, form_compiler_parameters)
54 else:
55 L = Form(L, form_compiler_parameters=form_compiler_parameters)
---> 56 a = Form(a, form_compiler_parameters=form_compiler_parameters)
57
58 # Initialize C++ base class
/usr/lib/python3/dist-packages/dolfin/fem/form.py in __init__(self, form, **kwargs)
42
43 ufc_form = ffc_jit(form, form_compiler_parameters=form_compiler_parameters,
---> 44 mpi_comm=mesh.mpi_comm())
45 ufc_form = cpp.fem.make_ufc_form(ufc_form[0])
46
/usr/lib/python3/dist-packages/dolfin/jit/jit.py in mpi_jit(*args, **kwargs)
45 # Just call JIT compiler when running in serial
46 if MPI.size(mpi_comm) == 1:
---> 47 return local_jit(*args, **kwargs)
48
49 # Default status (0 == ok, 1 == fail)
/usr/lib/python3/dist-packages/dolfin/jit/jit.py in ffc_jit(ufl_form, form_compiler_parameters)
95 p.update(dict(parameters["form_compiler"]))
96 p.update(form_compiler_parameters or {})
---> 97 return ffc.jit(ufl_form, parameters=p)
98
99
/usr/lib/python3/dist-packages/ffc/jitcompiler.py in jit(ufl_object, parameters, indirect)
215
216 # Inspect cache and generate+build if necessary
--> 217 module = jit_build(ufl_object, module_name, parameters)
218
219 # Raise exception on failure to build or import module
/usr/lib/python3/dist-packages/ffc/jitcompiler.py in jit_build(ufl_object, module_name, parameters)
131 name=module_name,
132 params=params,
--> 133 generate=jit_generate)
134 return module
135
/usr/lib/python3/dist-packages/dijitso/jit.py in jit(jitable, name, params, generate, send, receive, wait)
163 elif generate:
164 # 1) Generate source code
--> 165 header, source, dependencies = generate(jitable, name, signature, params["generator"])
166 # Ensure we got unicode from generate
167 header = as_unicode(header)
/usr/lib/python3/dist-packages/ffc/jitcompiler.py in jit_generate(ufl_object, module_name, signature, parameters)
64
65 code_h, code_c, dependent_ufl_objects = compile_object(ufl_object,
---> 66 prefix=module_name, parameters=parameters, jit=True)
67
68 # Jit compile dependent objects separately,
/usr/lib/python3/dist-packages/ffc/compiler.py in compile_form(forms, object_names, prefix, parameters, jit)
141 """This function generates UFC code for a given UFL form or list of UFL forms."""
142 return compile_ufl_objects(forms, "form", object_names,
--> 143 prefix, parameters, jit)
144
145
/usr/lib/python3/dist-packages/ffc/compiler.py in compile_ufl_objects(ufl_objects, kind, object_names, prefix, parameters, jit)
183 # Stage 1: analysis
184 cpu_time = time()
--> 185 analysis = analyze_ufl_objects(ufl_objects, kind, parameters)
186 _print_timing(1, time() - cpu_time)
187
/usr/lib/python3/dist-packages/ffc/analysis.py in analyze_ufl_objects(ufl_objects, kind, parameters)
88 # Analyze forms
89 form_datas = tuple(_analyze_form(form, parameters)
---> 90 for form in forms)
91
92 # Extract unique elements accross all forms
/usr/lib/python3/dist-packages/ffc/analysis.py in <genexpr>(.0)
88 # Analyze forms
89 form_datas = tuple(_analyze_form(form, parameters)
---> 90 for form in forms)
91
92 # Extract unique elements accross all forms
/usr/lib/python3/dist-packages/ffc/analysis.py in _analyze_form(form, parameters)
172 do_apply_geometry_lowering=True,
173 preserve_geometry_types=(Jacobian,),
--> 174 do_apply_restrictions=True)
175 elif r == "tsfc":
176 try:
/usr/lib/python3/dist-packages/ufl/algorithms/compute_form_data.py in compute_form_data(form, do_apply_function_pullbacks, do_apply_integral_scaling, do_apply_geometry_lowering, preserve_geometry_types, do_apply_default_restrictions, do_apply_restrictions, do_estimate_degrees, do_append_everywhere_integrals, complex_mode)
416 preprocessed_form = remove_complex_nodes(preprocessed_form)
417
--> 418 check_form_arity(preprocessed_form, self.original_form.arguments(), complex_mode) # Currently testing how fast this is
419
420 # TODO: This member is used by unit tests, change the tests to
/usr/lib/python3/dist-packages/ufl/algorithms/check_arities.py in check_form_arity(form, arguments, complex_mode)
175 def check_form_arity(form, arguments, complex_mode=False):
176 for itg in form.integrals():
--> 177 check_integrand_arity(itg.integrand(), arguments, complex_mode)
/usr/lib/python3/dist-packages/ufl/algorithms/check_arities.py in check_integrand_arity(expr, arguments, complex_mode)
157 key=lambda x: (x.number(), x.part())))
158 rules = ArityChecker(arguments)
--> 159 arg_tuples = map_expr_dag(rules, expr, compress=False)
160 args = tuple(a[0] for a in arg_tuples)
161 if args != arguments:
/usr/lib/python3/dist-packages/ufl/corealg/map_dag.py in map_expr_dag(function, expression, compress)
35 Return the result of the final function call.
36 """
---> 37 result, = map_expr_dags(function, [expression], compress=compress)
38 return result
39
/usr/lib/python3/dist-packages/ufl/corealg/map_dag.py in map_expr_dags(function, expressions, compress)
84 r = handlers[v._ufl_typecode_](v)
85 else:
---> 86 r = handlers[v._ufl_typecode_](v, *[vcache[u] for u in v.ufl_operands])
87
88 # Optionally check if r is in rcache, a memory optimization
/usr/lib/python3/dist-packages/ufl/algorithms/check_arities.py in sum(self, o, a, b)
46 def sum(self, o, a, b):
47 if a != b:
---> 48 raise ArityMismatch("Adding expressions with non-matching form arguments {0} vs {1}.".format(_afmt(a), _afmt(b)))
49 return a
50
ArityMismatch: Adding expressions with non-matching form arguments () vs ('v_1',).
When I write this (I remove the Kt from sigma(u)):
def sigma(u):
return (lambda_*nabla_div(u))*Identity(d) + (2*mu)*epsilon(u)
It works perfectly.
In this page (Click here), they try to plot the same kind problem and it works on my computer.
Do you know how to fix it ?
I had exactly the same question and a colleague of mine did figure it out for me. As there is no answer given here, I will try to leave some directions to guide others to the solution. I have not a lot of expertise yet, so please consider that my use of terminology might be a little bit off.
The error of fenics somewhat mislead me into thinking the error is in the definition of the stress term sigma. It is not exactly there. The right handside and the left handside in the solve function are not defined correctly (also shown in the very top of the error code). The term kT*Identity(d) in the stress function sigma, is not dependent on the trialfunction u. It is just multiplied by the testfunction v later (epsilon(v)). Therefore it has to go into the L of the equation of the solver.
Beneath the Link that you shared, the scipt uses the rhs and lhs function to correctly split the equation into a and L.

Overflow when unpacking long - Pytorch

I am running the following code
import torch
from __future__ import print_function
x = torch.empty(5, 3)
print(x)
on an Ubuntu machine in CPU mode, which gives me following error, what would be the reason and how to fix
x = torch.empty(5, 3)
----> print(x)
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __repr__(self)
55 # characters to replace unicode characters with.
56 if sys.version_info > (3,):
---> 57 return torch._tensor_str._str(self)
58 else:
59 if hasattr(sys.stdout, 'encoding'):
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)
216 suffix = ', dtype=' + str(self.dtype) + suffix
217
--> 218 fmt, scale, sz = _number_format(self)
219 if scale != 1:
220 prefix = prefix + SCALE_FORMAT.format(scale) + ' ' * indent
/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _number_format(tensor, min_sz)
94 # TODO: use fmod?
95 for value in tensor:
---> 96 if value != math.ceil(value.item()):
97 int_mode = False
98 break
RuntimeError: Overflow when unpacking long
Since, torch.empty() gives uninitialized memory, so you may or may not get a large value from it. Try
x = torch.rand(5, 3)
print(x)
this would give the response.

How do I rightly code linear regression with gradient descent in Python?

import pandas as pd
import matplotlib.pyplot as plt
# I'm trying to code the utter basic func of LinearRegression
# from sklearn.linear_model import LinearRegression
dataframe = pd.read_fwf('brain_body.txt') # link given below
x_values = dataframe[['Brain']]
y_values = dataframe[['Body']]
lr = LinearRegression(0.0001, 10) # sending learning_rate and iterations
lr.fit(x_values, y_values)
# commenting out because the values are insane
# plt.scatter(x_values, y_values)
# plt.plot(x_values, clf.predict(x_values))
# plt.show()
Link to brain_body.txt
Here's the class I've written
class LinearRegression:
def __init__(self, learning_rate, iterations):
self.b = 0 # b as in y=mx+b
self.m = 0 # m as in y=mx+b
self.learning_rate = learning_rate
self.iterations = iterations
def get_y(self, x):
return self.m * float(x) + self.b
def step_gradient(self, x_values, y_values):
print()
print("Values before: m =", self.m, " b =", self.b)
m_gradient = 0
b_gradient = 0
N = float(len(x_values.ix[:, 0]))
print('%11s' % "d(m)", '%11s' % "m_gradient", '%11s' % "d(b)", '%11s' % "b_gradient")
for i in range(int(N)):
x = x_values.iloc[i][0]
y = y_values.iloc[i][0]
# EDIT: I missed a * -1 here
# But that wouldn't just fix everything, adjusting learning rate does
pm = (y - self.get_y(x)) * x # partial derivative of m
pb = (y - self.get_y(x)) * -1 # partial derivative of b
m_gradient += pm * 2 / N
b_gradient += pb * 2 / N
print('%11s' % pm, '%11s' % m_gradient, '%11s' % pb, '%11s' % b_gradient)
self.m -= self.learning_rate * m_gradient # adjust current m
self.b -= self.learning_rate * b_gradient # adjust current b
print("Values after: m =", self.m, " b =", self.b)
print()
def fit(self, x_values, y_values): # equivalent to train_model
for i in range(self.iterations):
self.step_gradient(x_values, y_values)
return
def predict(self, x_values): # equivalent to get_output
predictions = []
for x in x_values.ix[:, 0]:
predictions.append(self.get_y(x))
return predictions
I watched Siraj Raval's How to do Linear Regression the right way and followed almost the same way he did. I did learn what partial derivatives and gradient descents are, but I do not what the values of partial derivatives be (or to guess them). And the numbers are going like crazy in the first iteration itself:
Values before: m = 0 b = 0
d(m) m_gradient d(b) b_gradient
150.6325 4.85911290323 -44.5 -1.43548387097
7.44 5.09911290323 -15.5 -1.93548387097
10.935 5.45185483871 -8.1 -2.19677419355
196695.0 6350.45185484 -423.0 -15.8419354839
4341.435 6490.49814516 -119.5 -19.6967741935
3180.9 6593.10782258 -115.0 -23.4064516129
1456.306 6640.08543548 -98.2 -26.5741935484
5.72 6640.26995161 -5.5 -26.7516129032
243.02 6648.10930645 -58.0 -28.6225806452
2.72 6648.19704839 -6.4 -28.8290322581
0.404 6648.21008065 -4.0 -28.9580645161
5.244 6648.37924194 -5.7 -29.1419354839
6.6 6648.59214516 -6.6 -29.3548387097
0.0007 6648.59216774 -0.14 -29.3593548387
0.06 6648.59410323 -1.0 -29.3916129032
37.8 6649.81345806 -10.8 -29.74
24.6 6650.60700645 -12.3 -30.1367741935
10.71 6650.95249032 -6.3 -30.34
11723841.0 384839.371845 -4603.0 -178.823870968
0.0069 384839.372068 -0.3 -178.833548387
78394.9 387368.23981 -419.0 -192.349677419
341255.0 398376.465616 -655.0 -213.478709677
2.7475 398376.554245 -3.5 -213.591612903
1150.0 398413.651019 -115.0 -217.301290323
84.48 398416.376181 -25.6 -218.127096774
1.0 398416.408439 -5.0 -218.288387097
24.675 398417.204406 -17.5 -218.852903226
359720.0 410021.075374 -680.0 -240.788387097
84042.0 412732.107632 -406.0 -253.88516129
27625.0 413623.236665 -325.0 -264.369032258
9.225 413623.534245 -12.3 -264.765806452
81840.0 416263.534245 -1320.0 -307.346451613
38007648.0 1642316.69554 -5712.0 -491.604516129
13.65 1642317.13586 -3.9 -491.730322581
1217.2 1642356.40037 -179.0 -497.504516129
1960.0 1642419.62618 -56.0 -499.310967742
68.85 1642421.84715 -17.0 -499.859354839
0.12 1642421.85102 -1.0 -499.891612903
0.0092 1642421.85132 -0.4 -499.904516129
0.0025 1642421.8514 -0.25 -499.912580645
17.5 1642422.41591 -12.5 -500.315806452
122500.0 1646374.02882 -490.0 -516.122258065
30.25 1646375.00462 -12.1 -516.512580645
9712.5 1646688.31107 -175.0 -522.157741935
15700.0 1647194.76269 -157.0 -527.222258065
22950.4 1647935.09817 -440.0 -541.415806452
1893.725 1647996.18607 -179.5 -547.206129032
1.32 1647996.22865 -2.4 -547.283548387
4860.0 1648153.00285 -81.0 -549.896451613
75.6 1648155.44156 -21.0 -550.573870968
168.0896 1648160.8638 -39.2 -551.838387097
0.532 1648160.88096 -1.9 -551.899677419
0.09 1648160.88387 -1.2 -551.938387097
0.366 1648160.89567 -3.0 -552.03516129
0.01584 1648160.89619 -0.33 -552.045806452
34560.0 1649275.73489 -180.0 -557.852258065
75.0 1649278.15425 -25.0 -558.658709677
27040.0 1650150.41231 -169.0 -564.110322581
2.34 1650150.4878 -2.6 -564.194193548
18.468 1650151.08354 -11.4 -564.561935484
0.26 1650151.09193 -2.5 -564.642580645
213.444 1650157.97722 -50.4 -566.268387097
Values after: m = -165.015797722 b = 0.0566268387097
Values after 10 iteration: m = -1.76899770934e+22 b = 4.21166966984e+18
How do I rightly do LinearRegression from scratch?
This might not be a true answer as it's using R (I could probably figure this out in python, but it would take me longer). I think your issue is in the size of your learning_rate. I'm taking this machine learning class at the moment and so I'm familiar with what you're doing and attempted to implement it myself. Here was my code:
library(ggplot2)
## create test data
data <- data.frame(x = 1:10, y = 1:10)
n <- nrow(data)
## initialize values
m <- 0
b <- 0
alpha <- 0.01
iters <- 100
results <- data.frame(i = 1:iters,
pm = 1:iters,
pb = 1:iters,
m = 1:iters,
b = 1:iters)
for (i in 1:iters) {
y_hat <- (m * data$x) + b
pm <- (1/n) * sum((y_hat - data$y) * data$x)
pb <- (1/n) * sum(y_hat - data$y)
m <- m - (alpha * pm)
b <- b - (alpha * pb)
## uncomment if you want; shows "animated" change
## p <- ggplot(data, aes(x = x, y = y)) + geom_point()
## p <- p + geom_abline(intercept = b, slope = m)
## print(p)
## this turned out to be key for looking at output
results[i, 2:5] <- c(pm, pb, m, b)
}
Now, note the end of results with a big alpha, 0.1:
> tail(results)
i pm pb m b
95 95 -2.864612e+45 -4.114745e+44 2.135518e+44 3.067470e+43
96 96 8.390457e+45 1.205210e+45 -6.254938e+44 -8.984628e+43
97 97 -2.457567e+46 -3.530062e+45 1.832073e+45 2.631600e+44
98 98 7.198218e+46 1.033956e+46 -5.366146e+45 -7.707961e+44
99 99 -2.108360e+47 -3.028460e+46 1.571745e+46 2.257664e+45
100 100 6.175391e+47 8.870365e+46 -4.603646e+46 -6.612702e+45
See how m and b are flip flopping? The learning rate alpha is so high that alpha * derivative are jumping over the minima! In the linked class this is shown in the gradient descent videos, but the concept is the same as this image I found:
Look at results using alpha = 0.01:
> tail(results)
i pm pb m b
95 95 -0.003483741 0.02425319 0.9834438 0.1152615
96 96 -0.003476426 0.02420226 0.9834785 0.1150195
97 97 -0.003469127 0.02415144 0.9835132 0.1147780
98 98 -0.003461842 0.02410073 0.9835478 0.1145370
99 99 -0.003454573 0.02405012 0.9835824 0.1142965
100 100 -0.003447319 0.02399962 0.9836169 0.1140565
It's slow, but we're honing in on m = 1 and b = 0 as expected. With your real data, I had a similar issue. The main code body is the same, with this replacing the data <- data.frame() line at the beginning:
data <- read.table(file = "https://raw.githubusercontent.com/llSourcell/linear_regression_demo/master/brain_body.txt",
header = T, sep = "", stringsAsFactors = F)
names(data) <- c("y", "x")
Everything else is the same, except that I played with alpha and iters. Here's what I found!
## your learning rate; diverging/flip-flopping
## alpha <- 0.0001
> tail(results)
i pm pb m b
95 95 -3.842565e+190 -1.167811e+187 3.801319e+186 1.155276e+183
96 96 3.541406e+192 1.076285e+189 -3.503393e+188 -1.064732e+185
97 97 -3.263851e+194 -9.919315e+190 3.228817e+190 9.812842e+186
98 98 3.008048e+196 9.141894e+192 -2.975760e+192 -9.043766e+188
99 99 -2.772294e+198 -8.425404e+194 2.742537e+194 8.334966e+190
100 100 2.555018e+200 7.765068e+196 -2.527592e+196 -7.681718e+192
## 1/10 as big; still diverging!
## alpha <- 0.00001
> tail(results)
i pm pb m b
95 95 -2.453089e+92 -7.455293e+88 2.189776e+87 6.655047e+83
96 96 2.040052e+93 6.200012e+89 -1.821074e+88 -5.534508e+84
97 97 -1.696559e+94 -5.156089e+90 1.514452e+89 4.602638e+85
98 98 1.410902e+95 4.287936e+91 -1.259457e+90 -3.827672e+86
99 99 -1.173342e+96 -3.565957e+92 1.047397e+91 3.183190e+87
100 100 9.757815e+96 2.965541e+93 -8.710418e+91 -2.647222e+88
## even smaller; that's better!
## alpha <- 0.000001
> tail(results)
i pm pb m b
95 95 -0.01579109 51.95899 0.8856351 -0.004667159
96 96 -0.01579107 51.95894 0.8856352 -0.004719118
97 97 -0.01579106 51.95889 0.8856352 -0.004771077
98 98 -0.01579104 51.95885 0.8856352 -0.004823036
99 99 -0.01579103 51.95880 0.8856352 -0.004874995
100 100 -0.01579102 51.95875 0.8856352 -0.004926953
With this final result, I plotted the results which look reasonable?
p <- ggplot(data, aes(x = x, y = y)) + geom_point()
p <- p + geom_abline(intercept = b, slope = m)
print(p)
So, to wrap up:
I didn't verify/check your python code
I did implement my understanding of gradient descent in R and try with a test to verify behavior
I re-tried this with your actual data to find it appears to work
thus, my recommendation would be to re-try your method with simplified data (sounds like you already might have) and then look at the initial steps with a very small learning rate to see if that fixes it. If not, there may still be something wrong with your math?
Hope that helps!

Python3 Extracting lines between first instance of two markers

I imported a text file from URL and want to process it. The file looks as below. There are two instances of " innings " and "Extras ". I want to extract lines between the FIRST instance of " innings " and FIRST instance of "Extras ". The code that I wrote extracts ALL instances. How do I resolve this?
Toss: Sri Lanka Umpires: M Erasmus (South Africa) and NJ Llong
(England) TV umpire: S Ravi (India) Match referee: DC Boon
(Australia) Reserve umpire: SD Fry (Australia) Player of the
match: CJ Anderson New Zealand innings (50 overs maximum)
R M B 4 6 MJ Guptill c Sangakkara b Lakmal
49 94 62 5 0 CJ Anderson c Lakmal b
Kulasekara 75 77 46 8 2
+L Ronchi not out 29 29 19 4 0
Extras (lb 2, w 8, nb 3) 13 Total (6 wickets, 50 overs, 226 mins) 331
Sri Lanka innings (target: 332 runs from 50 overs) R
M B 4 6 HDRL Thirimanne b Boult
65 90 60 8 0 RAS Lakmal not out
7 21 17 0 0
Extras (w 10, nb 1) 11 Total (all out, 46.1 overs, 210 mins) 233
Here is my code:
flag = 1
for line in data:
if " innings " in line:
flag = 0
print('')
if line.startswith("Extras "):
flag = 1
print('')
if not flag and not " innings " in line:
print(line)
Your program must stop on the first occurrence of Extras:
active = False # A variable `flag` is not very precisely named,
# better call it `active`, make it boolean
# and flip the values
for line in data:
if " innings " in line:
active = True # now we want to do things
print('')
continue # but not in this loop
if line.startswith("Extras "):
print('')
break # now we're done!
# alternative Method:
# active = False
if active:
print(line)
If you want to store all occurrences:
active = False
stored = []
for line in data:
if " innings " in line:
tmp = []
active = True # now we want to do things
continue # but not in this loop
if line.startswith("Extras "):
stored.append(tmp)
active = False
continue
if active:
tmp.append(line)
You'll end up with a list of lists of lines for further processing.

Resources