Not giving nearest distance from distance matrix in google OR solver - python-3.x

I am using google or tools to get the root from distance matrix but my constrains is to start from
nearest distance but solver is giving unexpected route. Below my distance matrix. My expected output is 1 to 10 could in sequence based on distance.
data['distance_matrix']=[[1000. 451.13 508.64 543.41 577.64 611.88 646.12 672.58 689.2
1231.78 1246.69]
[ 451.13 1000. 484.56 519.33 553.56 587.8 622.04 648.5 665.12
1207.7 1222.61]
[ 508.64 460.8 1000. 485.58 519.81 554.05 588.29 614.75 631.37
1173.95 1188.86]
[ 543.41 483.77 473.78 1000. 496.84 531.08 565.32 591.78 608.4
1150.98 1165.89]
[ 577.64 506.5 496.51 485.34 1000. 508.35 542.59 569.05 585.67
1128.25 1143.16]
[ 611.88 529.23 519.24 508.07 496.84 1000. 519.86 546.32 562.94
1105.52 1120.43]
[ 646.12 551.96 541.97 530.8 519.57 508.35 1000. 523.59 540.21
1082.79 1097.7 ]
[ 672.58 571.36 561.37 550.2 538.97 527.75 516.53 1000. 520.81
1063.39 1078.3 ]
[ 689.2 587.98 577.99 566.82 555.59 544.37 533.15 520.81 1000.
1046.77 1061.68]
[1231.78 1154.62 1144.63 1133.46 1122.23 1111.01 1099.79 1087.45 1070.83
1000. 495.04]
[1246.69 1171.32 1161.33 1150.16 1138.93 1127.71 1116.49 1104.15 1087.53
496.83 10. ]]
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']), data['num_vehicles'], data['depot'])
routing = pywrapcp.RoutingModel(manager)
...
transit_callback_index = routing.RegisterTransitCallback(distance_callback)
routing.SetArcCostEvaluatorOfAllVehicles(transit_callback_index)
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.PATH_CHEAPEST_ARC)
solution= routing.SolveWithParameters(search_parameters)
... Is there any way to add a constraint to solver to get route based on distance value order.if distance value 4, 3, 2 then my route point should be 4, 3, 2. ?


Do you mean for all i if j is nextVar(i) then transit(i-1, i) >= transit(i, j) ?
solver = routing.solver()
for index in range(routing.size()):
if routing.IsEnd(index):
continue
node = manager.IndexToNode(index)
for next_index in range(routing.size()):
if routing.IsEnd(next_index) or routing.IsStart(next_index):
continue
next_node = manager.IndexToNode(next_index)
first_cond = routing.NextVar(index) == next_index
for next_next_index in range(routing.size()):
if routing.IsStart(next_next_index):
continue
next_next_node = manager.IndexToNode(next_next_index)
second_cond = routing.NextVar(next_index) == next_next_index
sovler.Add(first_cond * second_cond * distance_matrix[node][next_node] >= first_cond * second_cond * distance_matrix[next_node][next_next_node])

Related

Numerically integrate a function with an array as upper limit

I'm trying to integrate numerically a 1/H(z)dz, where the function H(z) is given by
def Hz_th(theta,z):
aa,H0,omegam0=theta
return np.sqrt(H0**2*(1 + z)**3*omegam0 + 3*H0**2*(1 + z)**3*aa*omegam0 + (1 + z)**3*np.sqrt((4*H0**4*(1 - omegam0))/(1 + z)**6 + (-(H0**2*omegam0) - 3*H0**2*aa*omegam0)**2))/np.sqrt(2)
Where z is an array
z = np.array([0.50349, 0.4952, 0.6782, 0.8672, 0.7992, 0.37129, 0.35568, 0.28391, 0.46691, 0.94791, 0.69391, 0.89791, 0.78991, 0.59091, 0.2102, 0.8592, 0.5592, 0.9972, 0.5834, 0.7972, 0.9142, 0.94921, 0.7342, 0.7662, 0.9842, 0.7012, 0.83716, 0.37016, 0.35816, 0.52216, 0.70116, 0.74526, 0.59116, 0.62039, 1.2024, 0.6104, 0.35819, 0.45138, 0.55239, 0.75039, 0.82238, 0.95039, 0.34039, 0.8104, 0.61191, 0.54891, 1.02991, 0.42291, 0.70191, 0.68591, 0.93291, 0.74991, 0.46891, 0.95991, 0.8412, 0.5632, 0.8292, 0.57921, 0.8652, 0.8592, 0.58621, 0.4892, 0.68921, 0.32416, 0.18516, 0.68116, 0.92116, 0.42816, 0.69916, 0.57516, 0.51116, 0.64116, 0.35116, 0.34916, 0.35486, 0.87116, 0.92516, 0.4194, 0.89038, 0.48038, 0.58038, 0.80538, 0.66438, 0.74538, 0.7364, 0.37039, 0.85039, 0.21938, 0.64739, 0.60038, 0.76039, 0.47039, 0.49791, 0.50791, 0.58791, 0.69992, 0.76191, 0.78891, 0.40591, 0.80892, 0.58392, 0.72091, 0.83791, 0.80691, 0.50016, 0.53316, 0.82116, 0.12516, 0.44316, 0.68339, 0.72639, 0.40439, 0.69038, 0.57639, 0.72039, 0.55091, 0.8489, 0.43591, 0.30191, 0.58119, 0.34621, 0.3312, 0.44939, 0.2916, 0.46109, 0.63291, 0.26891, 0.92591, 0.62591, 0.60891, 0.57891, 0.7202, 0.7672, 0.3682, 0.8492, 0.77721, 0.58421, 0.5892, 0.7692, 0.5142, 0.63821, 0.43521, 0.58421, 0.66259, 0.62116, 0.41616, 0.45117, 0.73316, 0.74116, 0.34916, 0.51416, 0.64339, 0.4704, 0.26338, 0.7424, 0.91039, 0.3374, 0.98339, 0.81338, 0.8174, 0.96038, 0.75639, 0.87891, 0.80991, 0.62791, 0.93491, 0.69791, 0.76791, 0.86491, 0.51491, 0.47091, 0.92891, 0.6312, 0.6162, 0.5652, 0.55819, 0.7362, 0.2622, 0.7622, 0.8592, 0.5592, 0.48016, 0.92116, 0.47516, 0.89216, 0.63116, 0.73516, 0.41816, 0.53516, 0.73416, 0.60916, 0.70116, 0.98216, 0.75816, 0.64339, 0.51539, 0.80538, 0.80039, 0.71538, 0.76639, 0.90139, 0.71839, 0.57938, 0.96038, 0.64839, 0.93639, 0.46138, 0.67039, 0.24639, 0.63892, 0.53591, 0.77391, 0.37091, 0.69992, 0.37391, 0.85391, 0.60392, 0.53491, 0.40091, 0.37191, 0.84091, 0.53216, 0.93116, 0.55316, 0.84116, 0.72739, 0.44238, 0.2824, 0.5194, 1.06039, 0.26838, 0.2504, 0.69892, 0.73091, 0.99891, 0.40991, 0.12694, 0.25609, 0.08848, 0.11797, 0.18322, 0.1407, 0.15186, 0.09401, 0.28662, 0.26349, 0.1431, 0.16035, 0.12303, 0.12608, 0.17315, 0.16431, 0.1599, 0.24448, 0.24835, 0.27544, 0.33051, 0.33103, 0.16059, 0.21887, 0.11991, 0.17839, 0.27923, 0.18581, 0.06641, 0.258, 0.2798, 0.26576, 0.185, 0.05715, 0.3264, 0.10886, 0.20323, 0.37986, 0.2116, 0.21353, 0.14409, 0.25249, 0.25672, 0.21885, 0.07883, 0.08282, 0.17028, 0.2367, 0.30569, 0.08254, 0.18891, 0.08141, 0.25655, 0.12622, 0.10264, 0.12455, 0.37046, 0.20215, 0.1799, 0.1618, 0.22669, 0.29683, 0.25029, 0.14882, 0.14729, 0.21513, 0.14478, 0.13243, 0.21975, 0.38492, 0.174, 0.17789, 0.29936, 0.11845, 0.17653, 0.24552, 0.19611, 0.24852, 0.30632, 0.24637, 0.13514, 0.21054, 0.2363, 0.17851, 0.22068, 0.31494, 0.18443, 0.26129, 0.18123, 0.38048, 0.18481, 0.20847, 0.17706, 0.31582, 0.24973, 0.20368, 0.12336, 0.20318, 0.15339, 0.33439, 0.1958, 0.24973, 0.15509, 0.15596, 0.15837, 0.30938, 0.24857, 0.18507, 0.18376, 0.18079, 0.17696, 0.17768, 0.0718, 0.13811, 0.1696, 0.24555, 0.14387, 0.25191, 0.03743, 0.27749, 0.28746, 0.19867, 0.15999, 0.21075, 0.23775, 0.15517, 0.09391, 0.15271, 0.25722, 0.21226, 0.2454, 0.13696, 0.32569, 0.10638, 0.40127, 0.18931, 0.19726, 0.12837, 0.12262, 0.12715, 0.21323, 0.21099, 0.26856, 0.20453, 0.2701, 0.24185, 0.29114, 0.35346, 0.28543, 0.21955, 0.17558, 0.23542, 0.15325, 0.21005, 0.29135, 0.31376, 0.13687, 0.29567, 0.30394, 0.20557, 0.1219, 0.17428, 0.24883, 0.24558, 0.13833, 0.24315, 0.24267, 0.31468, 0.27573, 0.30041, 0.21728, 0.20335, 0.19025, 0.29021, 0.22786, 0.31065, 0.13729, 0.08784, 0.08504, 0.19009, 0.09227, 0.12903, 0.26166, 0.26162, 0.19422, 0.14703, 0.21179, 0.17958, 0.19029, 0.3388, 0.11741, 0.2889, 0.26405, 0.24961, 0.22967, 0.08543, 0.27656, 0.36193, 0.30021, 0.11635, 0.15492, 0.25037, 0.25174, 0.12928, 0.30929, 0.29888, 0.27091, 0.29353, 0.18979, 0.12376, 0.31288, 0.30915, 0.20267, 0.21163, 0.1789, 0.30354, 0.21457, 0.32045, 0.21835, 0.07523, 0.2576, 0.18468, 0.11628, 0.06481, 0.10288, 0.04437, 0.16595, 0.16817, 0.17063, 0.12042, 0.13868, 0.2338, 0.24643, 0.22286, 0.17915, 0.19718, 0.22916, 0.26451, 0.10337, 0.21338, 0.15653, 0.2434, 0.23652, 0.20852, 0.25104, 0.22534, 0.28742, 0.32984, 0.24603, 0.11966, 0.32931, 0.21082, 0.20308, 0.28436, 0.18037, 0.2607, 0.14792, 0.32793, 0.24206, 0.22334, 0.1828, 0.15304, 0.13818, 0.20735, 0.20313, 0.19801, 0.22256, 0.2734, 0.18651, 0.15935, 0.21935, 0.17373, 0.24436, 0.13353, 0.28332, 0.17383, 0.19654, 0.36952, 0.20548, 0.17357, 0.12688, 0.28434, 0.19392, 0.10018, 0.37425, 0.11771, 0.19176, 0.23684, 0.15899, 0.07877, 0.13864, 0.23381, 0.15689, 0.08806, 0.13045, 0.18203, 0.25625, 0.21674, 0.14533, 0.13639, 0.20515, 0.36976, 0.1294, 0.26687, 0.27822, 0.17483, 0.11364, 0.11242, 0.15317, 0.19671, 0.18914, 0.17625, 0.21757, 0.18563, 0.16356, 0.25721, 0.12164, 0.16455, 0.19862, 0.26821, 0.40128, 0.14581, 0.05573, 0.22452, 0.28655, 0.18388, 0.10351, 0.33136, 0.21865, 0.26357, 0.16284, 0.10683, 0.16477, 0.1075, 0.12899, 0.05948, 0.04093, 0.01705, 0.01531, 0.01472, 0.02673, 0.01732, 0.04437, 0.01567, 0.02477, 0.038, 0.0299, 0.01012, 0.0258, 0.01038, 0.03791, 0.03507, 0.01268, 0.02525, 0.01908, 0.03489, 0.0295, 0.03416, 0.04067, 0.07515, 0.05581, 0.02529, 0.03501, 0.02846, 0.0328, 0.03378, 0.04597, 0.04341, 0.05684, 0.02024, 0.03191, 0.0415, 0.02639, 0.02651, 0.02689, 0.02349, 0.05067, 0.04093, 0.06877, 0.01754, 0.02557, 0.02189, 0.03155, 0.06845, 0.02249, 0.04949, 0.03161, 0.03434, 0.04486, 0.01531, 0.03048, 0.03837, 0.03682, 0.0221, 0.01402, 0.0589, 0.05895, 0.06888, 0.0204, 0.02894, 0.02384, 0.01451, 0.03292, 0.06455, 0.01663, 0.05359, 0.03163, 0.02751, 0.02559, 0.02401, 0.03713, 0.03408, 0.02454, 0.01697, 0.01629, 0.03184, 0.037, 0.02932, 0.0174, 0.03578, 0.02845, 0.05035, 0.04269, 0.03212, 0.02752, 0.03049, 0.01724, 0.07063, 0.05344, 0.02878, 0.03518, 0.03348, 0.04651, 0.06119, 0.04714, 0.0212, 0.04028, 0.03863, 0.03314, 0.02414, 0.02837, 0.01043, 0.02038, 0.0471, 0.03984, 0.0122, 0.01271, 0.01763, 0.02171, 0.01572, 0.01972, 0.01678, 0.01226, 0.05322, 0.03055, 0.03001, 0.01506, 0.0153, 0.04713, 0.03411, 0.0152, 0.01504, 0.02763, 0.03359, 0.01447, 0.02547, 0.01082, 0.01741, 0.02031, 0.04819, 0.02223, 0.02811, 0.05458, 0.05721, 0.02091, 0.01454, 0.0431, 0.03273, 0.03399, 0.01382, 0.05092, 0.01752, 0.02482, 0.0288, 0.01568, 0.02197, 0.03426, 0.01698, 0.02752, 0.03438, 0.01318, 0.01726, 0.01209, 0.05418, 0.0197, 0.01662, 0.02398, 0.03144, 0.01291, 0.03827, 0.01532, 0.02643, 0.0202, 0.0373, 0.02348, 0.01635, 0.02275, 0.2305, 0.3812, 0.2534, 0.1526, 0.4365, 0.1023, 0.3712, 0.4661, 0.1959, 0.3112, 0.0322, 0.1098, 0.2255, 0.2604, 0.2253, 0.4474, 0.4399, 0.3374, 0.5283, 0.5761, 0.3876, 0.36, 0.347, 0.4391, 0.3492, 0.4305, 0.5088, 0.2888, 0.1727, 0.318, 0.2752, 0.103, 0.2049, 0.2647, 0.2348, 0.2145, 0.4276, 0.4019, 0.3218, 0.4512, 0.1612, 0.3512, 0.2492, 0.0822, 0.1056, 0.3327, 0.1022, 0.2928, 0.1326, 0.2707, 0.1004, 0.3536, 0.4235, 0.3001, 0.1476, 0.2288, 0.1388, 0.1436, 0.0919, 0.1568, 0.2388, 0.1496, 0.5078, 0.2528, 0.2842, 0.2368, 0.3412, 0.2641, 0.349, 0.3212, 0.1352, 0.3065, 0.2335, 0.3006, 0.3676, 0.1692, 0.07976, 0.1806, 0.1806, 0.2307, 0.04582, 0.2212, 0.2006, 0.0755, 0.02627, 0.1912, 0.1777, 0.1852, 0.3491, 0.2712, 0.2921, 0.3704, 0.0963, 0.1808, 0.2748, 0.2454, 0.3618, 0.2318, 0.1466, 0.1592, 0.4198, 0.3192, 0.5196, 0.1506, 0.3026, 0.2106, 0.2396, 0.3316, 0.2006, 0.3606, 0.2426, 0.2212, 0.1206, 0.3412, 0.2504, 0.3804, 0.3106, 0.1994, 0.24, 0.5027, 0.3701, 0.27, 0.2631, 0.268, 0.308, 0.25, 0.2804, 0.3148, 0.3288, 0.3297, 0.2588, 0.2246, 0.2102, 0.3878, 0.3068, 0.3652, 0.2488, 0.2988, 0.1238, 0.3312, 0.3826, 0.2406, 0.1712, 0.2088, 0.2896, 0.2457, 0.1361, 0.1505, 0.1187, 0.2448, 0.3503, 0.6192, 0.2196, 0.0994, 0.4098, 0.5025, 0.3815, 0.2797, 0.2307, 0.4269, 0.5501, 0.3253, 0.3509, 0.3283, 0.4504, 0.06441, 0.06559, 0.3185, 0.1453, 0.1396, 0.4294, 0.2166, 0.3666, 0.3806, 0.4206, 0.1996, 0.5106, 0.4267, 0.3156, 0.5779, 0.4812, 0.3922, 0.1735, 0.3302, 0.2996, 0.2296, 0.2927, 0.2085, 0.2749, 0.3032, 0.359, 0.36, 0.349, 0.6304, 0.3207, 0.51, 0.3669, 0.1384, 0.1792, 0.1717, 0.2708, 0.3473, 0.4796, 0.3192, 0.5192, 0.3802, 0.3712, 0.1808, 0.2236, 0.1412, 0.3055, 0.4045, 0.2856, 0.3012, 0.2637, 0.5006, 0.3051, 0.3857, 0.1985, 0.0906, 0.4774, 0.3312, 0.4712, 0.3306, 0.2607, 0.07141, 0.1812, 0.3379, 0.1221, 0.2501, 0.1604, 0.2988, 0.3892, 0.2928, 0.3288, 0.3056, 0.2858, 0.4292, 0.1792, 0.1896, 0.2488, 0.4096, 0.2416, 0.3376, 0.0706, 0.3456, 0.3306, 0.05495, 0.4231, 0.1072, 0.2854, 0.3254, 0.3347, 0.2907, 0.4404, 0.2411, 0.4607, 0.1194, 0.545, 0.4114, 0.2688, 0.04976, 0.4118, 0.07369, 0.3188, 0.4198, 0.4208, 0.4392, 0.4804, 0.1998, 0.3018, 0.2902, 0.3092, 0.2956, 0.3386, 0.3782, 0.0896, 0.4507, 0.4604, 1.206, 1.33, 1.54, 1.55, 1.7, 1.8, 2.26, 1.914, 1.3, 1.34, 01.02, 0.735, 1.12, 1.23, 1.23, 0.854, 1.37, 0.9752, 0.97, 0.74, 1.39, 1.305, 0.935, 1.014, 1.315, 1.092])
Furtherly I need to use the integral for MCMC sampling of free parameters, contained in theta. I tried the following code to perform the numerical integration over an array
def MCMCf(theta,z):
aa,H0,omegam0=theta
return np.vectorize(lambda x: 1.0/Hz_th(theta,z))
def MCMCfint(theta,z):
aa,H0,omegam0=theta
return np.vectorize(integrate.quad(MCMCf(theta,z), 0, z))
But it does not work for me. Maybe someone could give an advice? I need to keep the functional form, because I could not provide the values of free parameters, they will be constrained with emcee.
There is a full traceback, when I'm trtying to plot the result with random true values as free parameters
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/tmp/ipykernel_17393/1708689059.py in <module>
7 return np.vectorize(integrate.quad(MCMCf(aa,H0,omegam0,z), 0, z))
8 z=z_values
----> 9 plt.plot(z,MCMCfint(aa_true,H0_true,omegam0_true,z))
/tmp/ipykernel_17393/1708689059.py in MCMCfint(aa, H0, omegam0, z)
5
6 def MCMCfint(aa,H0,omegam0,z):
----> 7 return np.vectorize(integrate.quad(MCMCf(aa,H0,omegam0,z), 0, z))
8 z=z_values
9 plt.plot(z,MCMCfint(aa_true,H0_true,omegam0_true,z))
~/.local/lib/python3.8/site-packages/scipy/integrate/quadpack.py in quad(func, a, b, args, full_output, epsabs, epsrel, limit, points, weight, wvar, wopts, maxp1, limlst)
346
347 # check the limits of integration: \int_a^b, expect a < b
--> 348 flip, a, b = b < a, min(a, b), max(a, b)
349
350 if weight is None:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

import scipy.special as spec
from scipy.integrate
import quad
import numpy as np
array = np.array([0, 1, 2, 3, 4, 5]) # array[i] >= 0
for all i in [0, len(array) - 1].For example, array = np.array([1, 2, 3, 4, 5])
constant = 3 # >= 0. For example, constant = 3
a = array
b = constant * a # multiply numpy array by a scalar(it is possible with numpy
arrays(: ) c = b * a / constant
e3 = spec.expn(3, c)
print('Output:', e3) #
return a ndarray with the value of the integral e3 en each point
specified in c(numpy array)
# Output: [1.09691967e-01 2.76136095e-03 1.04786279e-05 5.96855753e-09 4.97790975e-13]
# For evaluate the integral e3 from 0 to a[len(a) - 1]:
result1 = e3[len(a) - 1] - e3[0] print('Output:', result1) # Output: -0.499999999950223
# Other thing is the integral of e3 from 0 to a[len(a) - 1].You can use
quad(
return the value and an estimted error) # See https:
result2 = quad(lambda x: spec.expn(3, x), 0, c[len(a) - 1])
print('Output:', result2) # Output: (0.3333333333328521, 4.440892098500626e-16)

How to find the shortest distance between two line segments capturing the sign values with python

I have a pandas dataframe of the form:
benchmark_x benchmark_y ref_point_x ref_point_y
0 525039.140 175445.518 525039.145 175445.539
1 525039.022 175445.542 525039.032 175445.568
2 525038.944 175445.558 525038.954 175445.588
3 525038.855 175445.576 525038.859 175445.576
4 525038.797 175445.587 525038.794 175445.559
5 525038.689 175445.609 525038.679 175445.551
6 525038.551 175445.637 525038.544 175445.577
7 525038.473 175445.653 525038.459 175445.594
8 525038.385 175445.670 525038.374 175445.610
9 525038.306 175445.686 525038.289 175445.626
I am trying to find the shortest distance from the line to the benchmark such that if the line is above the benchmark the distance is positive and if it is below the benchmark the distance is negative. See image below:
I used the KDTree from scipy like so:
from scipy.spatial import KDTree
tree=KDTree(df[["benchmark_x", "benchmark_y"]])
test = df.apply(lambda row: tree.query(row[["ref_point_x", "ref_point_y"]]), axis=1)
test=test.apply(pd.Series, index=["distance", "index"])
This seems to work except that it fails to capture the negative values as a result that the line is below the benchmark.

# recreating your example
columns = "benchmark_x benchmark_y ref_point_x ref_point_y".split(" ")
data = """525039.140 175445.518 525039.145 175445.539
525039.022 175445.542 525039.032 175445.568
525038.944 175445.558 525038.954 175445.588
525038.855 175445.576 525038.859 175445.576
525038.797 175445.587 525038.794 175445.559
525038.689 175445.609 525038.679 175445.551
525038.551 175445.637 525038.544 175445.577
525038.473 175445.653 525038.459 175445.594
525038.385 175445.670 525038.374 175445.610
525038.306 175445.686 525038.289 175445.626"""
data = [float(x) for x in data.replace("\n"," ").split(" ") if len(x)>0]
arr = np.array(data).reshape(-1,4)
df = pd.DataFrame(arr, columns=columns)
# adding your two new columns to the df
from scipy.spatial import KDTree
tree=KDTree(df[["benchmark_x", "benchmark_y"]])
df["distance"], df["index"] = tree.query(df[["ref_point_x", "ref_point_y"]])
Now to compare if one line is above the other or not, we have to evaluate y at the same x position. Therefore we need to interpolate the y points for the x positions of the other line.
df = df.sort_values("ref_point_x") # sorting is required for interpolation
xy_refpoint = df[["ref_point_x", "ref_point_y"]].values
df["ref_point_y_at_benchmark_x"] = np.interp(df["benchmark_x"], xy_refpoint[:,0], xy_refpoint[:,1])
And finally your criterium can be evaluated and applied:
df["distance"] = np.where(df["ref_point_y_at_benchmark_x"] < df["benchmark_y"], -df["distance"], df["distance"])
# or change the < to <,>,<=,>= as you wish

Prime Integer - Add Smallest Value - time limit exceeding python

The program must accept N integers as the input. For each integer value X among the N integers, the program must print the smallest values that can be added to the integer X (possibly zero) so that the integer becomes a prime
Boundary :
1<=N<=2,00,000
2<=Each Int<=10^6
TimeLimit : 100 ms
My Code :
def isPrime(n) :
if (n <= 1) :
return False
if (n <= 3) :
return True
if (n % 2 == 0 or n % 3 == 0) :
return False
i = 5
while(i * i <= n) :
if (n % i == 0 or n % (i + 2) == 0) :
return False
i = i + 6
return True
def summ(g):
for i in range(20):
if isPrime(g+i):
return i
n = int(input())
print(*(summ(i) for i in list(map(int,input().split()))))
Example :
Input :
7
89 54 36 74 44 19 12
Output:
0 5 1 5 3 0 1
My code has no error, and i have used the optimised way to find whether a integer is prime or not but still i get time limit exceed for N == 100,000 Is there any way to reduce the time limit to 100 ms
The worst case scenario is
n=200,000
[169988', '865748', '818394', '160754', '532538', '44700', '607214', '404390', '405894', '954978', '115734', '871022', '762398', '744128', '394638', '981440', '966030', '515540', '859578', '116240', '534858', '263984', '159408', '333930', '310728', '949262', '580734', '40032', '289244', '130052', '921200', '452534', '58632', '689964', '250920', '126224', '845184', '806088', '867734', '662370', '204668', '3728', '193164', '187964', '477260', '365138', '21194', '334760', '46272', '401244', '239934', '670490', '894330', '408770', '766068', '129968', '702828', '354464', '396834', '905340', '586770', '543860', '923388', '197610', '809834', '862344', '194204', '609278', '7874', '411158', '701220', '78780', '525980', '949812', '58700', '829224', '885608', '38630', '507372', '866330', '12918', '292058', '143978', '801290', '97460', '746118', '625610', '48590', '182124', '127952', '545388', '922638', '268050', '801488', '613290', '276278', '387138', '664442', '519588', '418812', '536532', '755570', '283832', '92004', '66890', '724520', '11594', '134742', '542684', '169910', '37022', '689762', '633878', '640968', '277550', '839162', '953150', '742370', '219954', '271058', '846722', '22398', '237930', '637712', '46220', '868740', '9872', '573108', '399720', '114810', '262194', '122100', '104712', '494354', '791804', '279110', '484544', '89460', '797682', '875298', '740522', '218510', '188954', '337794', '375158', '296160', '172604', '871958', '451362', '746414', '757344', '657894', '60734', '485732', '213650', '206300', '716858', '101064', '118820', '185750', '332264', '93288', '363590', '555828', '120284', '814008', '954912', '513698', '795002', '384344', '116960', '866738', '926660', '998528', '170384', '569244', '695504', '77864', '469070', '38784', '47388', '358484', '533738', '966432', '126714', '440822', '964040', '808958', '698388', '157428', '338412', '332208', '976514', '25794', '251540', '399872', '765468', '331142', '454722', '263958', '51614', '694080', '422268', '14754', '661268', '479798', '647402', '198842', '896112', '346562', '72354', '106950', '516234', '313154', '60728', '244640', '345732', '215768', '75980', '349110', '589140', '950690', '476888', '246540', '150908', '291104', '42768', '82470', '751278', '366678', '806904', '741080', '356094', '206828', '850388', '270338', '649218', '10274', '48884', '935814', '978750', '892628', '212678', '163212', '133650', '883974', '665294', '806762', '876824', '8838', '697262', '403434', '114408', '382862', '351438', '969342', '79802', '645000', '113028', '349338', '354150', '384548', '20564', '465644', '721452', '607418', '575250', '940032', '463890', '579282', '862992', '847664', '896634', '828408', '847202', '443940', '879114', '418512', '997202', '690104', '770184', '594750', '851034', '447108', '640650', '271218', '990762', '493920', '594642', '852830', '191562', '794994', '117390', '672902', '978018', '59772', '842450', '153818', '139362', '712302', '437862', '263724', '201894', '575220', '481640', '242010', '466730', '997370', '128450', '174222', '109904', '455970', '564498', '534740', '340802', '358074', '796152', '97022', '630024', '221874', '950738', '730592', '907394', '566988', '580982', '155502', '425520', '908808', '199374', '491498', '50232', '107468', '857322', '728628', '477410', '430290', '643458', '642582', '612714', '705560', '834470', '775344', '55734', '782088', '13800', '432344', '146778', '518412', '450600', '163322', '854958', '30894', '872090', '855734', '304212', '535124', '68438', '734480', '532752', '547038', '366522', '424268', '644364', '152658', '17402', '932750', '465008', '807540', '670730', '205398', '988850', '779900', '231968', '746024', '678612', '319682', '357564', '521982', '363902', '487692', '880730', '340958', '807338', '540612', '38012', '21912', '657500', '836570', '647430', '884294', '470078', '875114', '266354', '721058', '853970', '265872', '795102', '788078', '451580', '169952', '997268', '147648', '185768', '286802', '408744', '459230', '33108', '490004', '27998', '123050', '153702', '151142', '543828', '18798', '498104', '459842', '895314', '119298', '787712', '197424', '918752', '82268', '443732', '839430', '134334', '570392', '16964', '537242', '149112', '500180', '531912', '556944', '942860', '892862', '518690', '330612', '904088', '746678', '310232', '876374', '345908', '889280', '619188', '808994', '658212', '248888', '135572', '55890', '741342', '512538', '55932', '251622', '12228', '687458', '48342', '310364', '345228', '436158', '72720', '634688', '179990', '527942', '939924', '68042', '902502', '295334', '9414', '203354', '758112', '134918', '661482', '948174', '476082', '75330', '94874', '825108', '78008', '833048', '647588', '951278', '857714', '970538', '248628', '141444', '477768', '464814', '514278', '53234', '866714', '455514', '188300', '90620', '206624', '259682', '13760', '77154', '272772', '58604', '358880', '522080', '914240', '644790', '1700', '381570', '955278', '429242', '713192', '745934', '961100', '480584', '352250', '658488', '929808', '224430', '52584', '915974', '147180', '953112', '585792', '212058', '915144', '953502', '756704', '800144', '143138', '96780', '303890', '557282', '999050', '680658', '446934', '304962', '211868', '308928', '42452', '578484', '853320', '671358', '780050', '196908', '983930', '950810', '65124', '489794', '391250', '708284', '351078', '34920', '116868', '397634', '19604', '867620', '547770', '746792', '503624', '245630', '62202', '222008', '207470', '674162', '136538', '130100', '452042', '384488', '432662', '280352', '570192', '905652', '529808', '33410', '482622', '992372', '28278', '196202', '170778', '874772', '80538', '450842', '532452', '164430', '411992', '456960', '32942', '934002', '565248', '973760', '469770', '280298', '21384', '807474', '254520', '235242', '633938', '873404', '687180', '305472', '377844', '110850', '289848', '969444', '812138', '558422', '334932', '630920', '276444', '483140', '951362', '720092', '286814', '503054', '727668', '587520', '307512', '448598', '332952', '7322', '721014', '869900', '278690', '169244', '763494', '358314', '438828', '23628', '241982', '672044', '499854', '4458', '191510', '814328', '728874', '948090', '327344', '957332', '116868', '879170', '442208', '113024', '11312', '446190', '847934', '792108', '307578', '60938', '987660', '26628', '996600', '21882', '580170', '877940', '633470', '388692', '7938', '543608', '806372', '544938', '819150', '439788', '393630', '126962', '662714', '595962', '759180', '76718', '731118', '514890', '125222', '384438', '645684', '268884', '608118', '537374', '500258', '90584', '27774', '195992', '872090', '923712', '24692', '600360', '571854', '863120', '718304', '29790', '107508', '717428', '52070', '609374', '208254', '1230', '92684', '51488', '986114', '798228', '951080', '944310', '582458', '182058', '41958', '488922', '101604', '302010', '459182', '912344', '868874', '1298', '166350', '531638', '219282', '99000', '689412', '809604', '60398', '814008', '211658', '357672', '26628', '874824', '520380', '38288', '105608', '581730', '978474', '972684', '4598', '92762', '2162', '630912', '225242', '663654', '145220', '517824', '999102', '414312', '211502', '833310', '541712', '379724', '902600', '277494', '607038', '329432', '757364', '784628', '25170', '282428', '35324', '932664', '455402', '796260', '540588', '606302', '22434', '128340', '10464', '138684', '34614', '534342', '660120', '271068', '346440', '959370', '178118', '256020', '10772', '657708', '781044', '228678', '311882', '488', '510452', '327444', '875712', '647964', '869438', '121260', '579504', '532920', '938342', '540438', '784548', '260922', '186480', '653820', '163772', '130364', '959490', '810194', '812250', '375252', '168458', '476478', '838694', '257372', '311420', '615068', '961118', '950634', '702018', '408804', '111494', '591752', '467472', '599742', '374118', '509222', '283008', '91250', '362852', '718974', '580734', '755318', '920790', '908582', '72224', '936054', '698670', '892604', '510332', '171618', '688680', '299018', '864290', '426998', '663410', '710372', '521642', '43952', '389798', '753984', '107310', '81902', '915380', '883490', '275912', '70452', '702102', '170958', '479570', '269210', '871160', '158792', '473288', '345432', '758930', '848850', '725910', '557694', '515598', '30638', '571370', '189068', '294704', '793548', '745118', '832478', '615722', '156362', '854364', '688160', '363152', '631938', '236682', '460464', '36068', '269982', '634344', '348210', '249974', '324774', '705534', '79394', '122850', '694082', '346878', '704688', '880514', '77558', '647790', '39954', '635354', '407948', '3080', '131448', '365760', '533672', '923948', '657072', '565668', '324294', '105692', '762780', '342972', '591090', '752994', '815520', '216732', '99714', '114278', '480072', '632994', '684978', '666770', '997740', '820578', '293640', '411194', '446294', '550490', '933968', '443868', '13298', '214010', '46818', '182060', '383694', '955608', '515612', '717884', '1302', '739494', '572834', '264464', '947652', '565464', '496748', '574160', '651462', '186708', '957212', '464910', '638568', '22470', '238748', '830888', '239964', '94350', '830580', '561104', '403788', '28352', '433370', '943752', '160982', '262338', '641262', '404558', '720060', '171264', '972330', '380198', '351402', '287150', '633474', '338390', '33354', '534404', '169322', '372060', '692730', '52364', '652358', '74204', '419088', '499524', '513158', '648342', '718358', '294564', '335610', '8630', '652658', '408624', '339762', '786420', '73304', '379188', '527348', '148914', '430752', '644198', '200088', '957098', '340284', '278414', '162692', '883878', '335528', '694368', '192818', '96908', '754968', '222008', '653622', '552710', '721208', '300234', '592854', '382650', '336042', '445878', '148868', '217970', '489990', '9030', '443604', '829988', '224712', '187508', '1784', '735422', '600338', '52164', '182142', '582794', '412950', '447678', '869070', '597270', '172002', '625410', '894404', '769620', '53820', '103484', '852990', '332804', '895320', '361512', '331502', '187928', '771218', '66702', '580530', '725808', '530262', '828524', '814644', '877404', '621740', '592484', '442754', '791952', '186552', '9750', '50924', '742608', '794594', '241454', '379178', '177044', '43332', '702828', '426744', '302928', '349914', '656600', '311982', '254700', '376758', '85782', '893220', '337284', '308142', '443040', '615314', '95084', '893568', '127740', '306030', '390782', '90002', '286290', '155664', '723960', '650564', '64680', '482124', '700362', '806858', '217082', '672884', '173714', '489102', '174458', '432054', '241952', '240422', '439800', '611484', '937502', '286928', '371360', '473504', '961134', '10134', '412538', '456350', '945962', '673550', '862418', '426198', '809424', '152810', '931878', '698978', '243840', '473378', '630102', '665222', '351932', '538598', '819812', '53454', '526652', '91944', '473328', '505694', '93494', '511898', '996884', '100610', '787808', '946460', '589328', '317732', '617768', '730188', '554532', '309108', '818430', '170810', '316214', '17342', '855620', '933422', '987798', '939452', '42900', '185492', '970878', '403688', '2544', '157890', '269210', '536672', '9042', '305862', '615188', '143528', '867734', '16128', '182', '210462', '540804', '504390', '880872', '73422', '20232', '606380', '668154', '66404', '507140', '121622', '950334', '157680', '170844', '514562', '110588', '239384', '390114', '733374', '6222', '400754', '746414', '106020', '675974', '325518', '389570', '377790', '800874', '831168', '21408', '612612', '599244', '729614', '486768', '365852', '588738', '897008', '983318', '728994', '71414', '352742', '960738', '220404', '754302', '95604', '581558', '85082', '602040', '473444', '907470', '608498', '26634', '434510', '358230', '327402', '713564', '421350', '528392', '647838', '855672', '717668', '182048', '293484', '179820', '117530', '143528', '754532', '399044', '649262', '895244', '582018', '25764', '655718', '918890', '879798', '511172', '154098', '92204', '897882', '120504', '2790', '1500', '285674', '24152', '196728', '192588', '705644', '779082', '876722', '549644', '532404', '761928', '998540', '311792', '66960', '77478', '191232', '181838', '146678', '549944', '315698', '250920', '576882', '955334', '318824', '620492', '29412', '193608', '16092', '95582', '405818', '765104', '975900', '20850', '474788', '934598', '725520', '683358', '988902', '900624', '728948', '181362', '690120', '155382', '450422', '292460', '60590', '1784', '210128', '952670', '966212', '212124', '313344', '267498', '522000', '510050', '623478', '309092', '133968', '739202', '400314', '866400', '226484', '7220', '692250', '462444', '443190', '846578', '458070', '432924', '304430', '915642', '940760', '402342', '463680', '932022', '508302', '989798', '74288', '360008', '138060', '257540', '80898', '555684', '12792', '678578', '147254', '59264', '16488', '171518', '275340', '5642', '446262', '509522', '279680', '750312', '496314', '193800', '938808', '693878']
I can't copy all the 200,000 integers

Here's one possible solution. It generates a list of primes up to the maximum value of the input numbers, then uses a binary search to find the next largest prime relative to each input number, finally computing the difference between those values and the inputs:
from math import sqrt
def sieve(prime, n, primes):
for i in range(prime*prime, n+1, prime):
primes[i] = False
return primes
def listPrimes(n):
n = n * 2 # to allow for the largest input value not being prime
primes = [False, False] + [True] * (n - 1)
primes = sieve(2, n, primes)
q = int(sqrt(n))
for p in range(3, q+1, 2):
if primes[p]:
primes = sieve(p, n, primes)
return [p for p in range(n+1) if primes[p]]
def findNextPrime(n, primes):
i = len(primes) // 2
if primes[i] >= n:
if i == 0 or primes[i-1] < n:
return primes[i]
return findNextPrime(n, primes[:i])
return findNextPrime(n, primes[i:])
nums = [89, 54, 91, 74, 44, 19, 12]
primes = listPrimes(max(nums))
nextPrimes = [findNextPrime(n, primes) for n in nums]
delta = [np - p for p, np in zip(nums, nextPrimes)]
print(delta)
Output:
[0, 5, 1, 5, 3, 0, 1]
Update
As #PresidentJamesK.Polk points out in the comments, this can be made more efficient by pre-computing the differences between each number and the next prime. This can be done by not filtering the sieve and then traversing it in reverse order to compute the distance from each number to the next prime:
from math import sqrt
def sieve(prime, n, primes):
for i in range(prime*prime, n+1, prime):
primes[i] = False
return primes
def listPrimes(n):
n = n * 2 # to allow for the largest input value not being prime
primes = [False, False] + [True] * (n - 1)
primes = sieve(2, n, primes)
q = int(sqrt(n))
for p in range(3, q+1, 2):
if primes[p]:
primes = sieve(p, n, primes)
return primes
def primeDiffs(primes):
primediffs = []
i = len(primes) - 1
while not primes[i]:
i -= 1
for i in range(i, -1, -1):
if primes[i]:
dist = 0
else:
dist += 1
primediffs.append(dist)
return primediffs[::-1]
nums = [89, 54, 36, 74, 44, 19, 12]
primes = listPrimes(max(nums))
diffs = primeDiffs(primes)
delta = [diffs[n] for n in nums]
print(delta)
Output:
[0, 5, 1, 5, 3, 0, 1]
Note
For both cases, max(nums) should be computed as the numbers are input to save time.

Here's a quick way of doing it, it's faster than 100ms for each item but slower for the full set ( even sieving takes longer than 100ms, and that's a fast algorithm):
from sympy import isprime
import random
import numpy as np
def primes_sieve2(limit):
a = [True] * limit
a[0] = a[1] = False
for (i, isprime) in enumerate(a):
if isprime:
yield i
for n in range(i*i, limit, i):
a[n] = False
PRIMES=np.array(list(primes_sieve2(2750160)), 'uint32')
def create_random_array(limit=len(PRIMES)):
vv = np.zeros(len(PRIMES), 'uint32')
for x in range(limit-1):
vv[x] = random.randint(0,pow(10,6))
return vv
vv = create_random_array()
vvx = np.zeros(len(PRIMES), 'uint32')
count = 0
for xx in vv:
vvx[count] = PRIMES[np.argmax(PRIMES > xx)] - xx
count+=1
vvisprime = np.zeros(len(PRIMES), 'bool')
for x in range(len(PRIMES)):
vvisprime[x] = isprime(int(vvx[x]) + int(vv[x]))
In [49]: vvx
Out[49]: array([ 5, 10, 13, ..., 12, 2, 2], dtype=uint32)
In [48]: vvisprime
Out[48]: array([ True, True, True, ..., True, True, True])

scipy.optimize.minimize() not converging giving success=False

I recently tried to apply backpropagation algorithm in python, I tried fmin_tnc,bfgs but none of them actually worked, so please help me to figure out the problem.
def sigmoid(Z):
return 1/(1+np.exp(-Z))
def costFunction(nnparams,X,y,input_layer_size=400,hidden_layer_size=25,num_labels=10,lamda=1):
#input_layer_size=400; hidden_layer_size=25; num_labels=10; lamda=1;
Theta1=np.reshape(nnparams[0:hidden_layer_size*(input_layer_size+1)],(hidden_layer_size,(input_layer_size+1)))
Theta2=np.reshape(nnparams[(hidden_layer_size*(input_layer_size+1)):],(num_labels,hidden_layer_size+1))
m=X.shape[0]
J=0;
y=y.reshape(m,1)
Theta1_grad=np.zeros(Theta1.shape)
Theta2_grad=np.zeros(Theta2.shape)
X=np.concatenate([np.ones([m,1]),X],1)
a2=sigmoid(Theta1.dot(X.T));
a2=np.concatenate([np.ones([1,a2.shape[1]]),a2])
h=sigmoid(Theta2.dot(a2))
c=np.array(range(1,11))
y=y==c;
for i in range(y.shape[0]):
J=J+(-1/m)*np.sum(y[i,:]*np.log(h[:,i]) + (1-y[i,:])*np.log(1-h[:,i]) );
DEL2=np.zeros(Theta2.shape); DEL1=np.zeros(Theta1.shape);
for i in range(m):
z2=Theta1.dot(X[i,:].T);
a2=sigmoid(z2).reshape(-1,1);
a2=np.concatenate([np.ones([1,a2.shape[1]]),a2])
z3=Theta2.dot(a2);
# print('z3 shape',z3.shape)
a3=sigmoid(z3).reshape(-1,1);
# print('a3 shape = ',a3.shape)
delta3=(a3-y[i,:].T.reshape(-1,1));
# print('y shape ',y[i,:].T.shape)
delta2=((Theta2.T.dot(delta3)) * (a2 * (1-a2)));
# print('shapes = ',delta3.shape,a3.shape)
DEL2 = DEL2 + delta3.dot(a2.T);
DEL1 = DEL1 + (delta2[1,:])*(X[i,:]);
Theta1_grad=np.zeros(np.shape(Theta1));
Theta2_grad=np.zeros(np.shape(Theta2));
Theta1_grad[:,0]=(DEL1[:,0] * (1/m));
Theta1_grad[:,1:]=(DEL1[:,1:] * (1/m)) + (lamda/m)*(Theta1[:,1:]);
Theta2_grad[:,0]=(DEL2[:,0] * (1/m));
Theta2_grad[:,1:]=(DEL2[:,1:]*(1/m)) + (lamda/m)*(Theta2[:,1:]);
grad=np.concatenate([Theta1_grad.reshape(-1,1),Theta2_grad.reshape(-1,1)]);
return J,grad
This is how I called the function (op is scipy.optimize)
r2=op.minimize(fun=costFunction, x0=nnparams, args=(X, dataY.flatten()),
method='TNC', jac=True, options={'maxiter': 400})
r2 is like this
fun: 3.1045444063663266
jac: array([[-6.73218494e-04],
[-8.93179045e-05],
[-1.13786179e-04],
...,
[ 1.19577741e-03],
[ 5.79555099e-05],
[ 3.85717533e-03]])
message: 'Linear search failed'
nfev: 140
nit: 5
status: 4
success: False
x: array([-0.97996948, -0.44658952, -0.5689309 , ..., 0.03420931,
-0.58005183, -0.74322735])
Please help me to find correct way of minimizing this function, Thanks in advance

Finally Solved it, The problem was I used np.randn() to generate random Theta values which gives random values in a standard normal distribution, therefore as too many values were within the same range,therefore this lead to symmetricity in the theta values. Due to this symmetricity problem the optimization terminates in the middle of the process.
Simple solution was to use np.rand() (which provide uniform random distribution) instead of np.randn()

hell going on with stochastic gradient descent

I am working with multivariate linear regression and using stochastic gradient descent to optimize.
Working on this dataSet
http://archive.ics.uci.edu/ml/machine-learning-databases/abalone/
for every run all hyperParameters and all remaining things are same, epochs=200 and alpha=0.1
when I first run then I got final_cost=0.0591, when I run the program again keeping everything same I got final_cost=1.0056
, running again keeping everything same I got final_cost=0.8214
, running again final_cost=15.9591, running again final_cost=2.3162 and so on and on...
As you can see that keeping everything same and running, again and again, each time the final cost changes by large amount sometimes so large like from 0.8 to direct 15.9 , 0.05 to direct 1.00 and not only this the graph of final cost after every epoch within the same run is every zigzag unlike in batch GD in which the cost graph decreases smoothly.
I can't understand that why SGD is behaving so weirdly, different results in the different run.
I tried the same with batch GD and everything is absolutely fine and smooth as per expectations. In case of batch GD no matter how many times I run the same code the result is exactly the same every time.
But in the case of SGD, I literally cried,
class Abalone :
def __init__(self,df,epochs=200,miniBatchSize=250,alpha=0.1) :
self.df = df.dropna()
self.epochs = epochs
self.miniBatchSize = miniBatchSize
self.alpha = alpha
print("abalone created")
self.modelTheData()
def modelTheData(self) :
self.TOTAL_ATTR = len(self.df.columns) - 1
self.TOTAL_DATA_LENGTH = len(self.df.index)
self.df_trainingData =
df.drop(df.index[int(self.TOTAL_DATA_LENGTH * 0.6):])
self.TRAINING_DATA_SIZE = len(self.df_trainingData)
self.df_testingData =
df.drop(df.index[:int(self.TOTAL_DATA_LENGTH * 0.6)])
self.TESTING_DATA_SIZE = len(self.df_testingData)
self.miniBatchSize = int(self.TRAINING_DATA_SIZE / 10)
self.thetaVect = np.zeros((self.TOTAL_ATTR+1,1),dtype=float)
self.stochasticGradientDescent()
def stochasticGradientDescent(self) :
self.finalCostArr = np.array([])
startTime = time.time()
for i in range(self.epochs) :
self.df_trainingData =
self.df_trainingData.sample(frac=1).reset_index(drop=True)
miniBatches=[self.df_trainingData.loc[x:x+self.miniBatchSize-
((x+self.miniBatchSize)/(self.TRAINING_DATA_SIZE-1)),:]
for x in range(0,self.TRAINING_DATA_SIZE,self.miniBatchSize)]
self.epochCostArr = np.array([])
for j in miniBatches :
tempMat = j.values
self.actualValVect = tempMat[ : , self.TOTAL_ATTR:]
tempMat = tempMat[ : , :self.TOTAL_ATTR]
self.desMat = np.append(
np.ones((len(j.index),1),dtype=float) , tempMat , 1 )
del tempMat
self.trainData()
currCost = self.costEvaluation()
self.epochCostArr = np.append(self.epochCostArr,currCost)
self.finalCostArr = np.append(self.finalCostArr,
self.epochCostArr[len(miniBatches)-1])
endTime = time.time()
print(f"execution time : {endTime-startTime}")
self.graphEvaluation()
print(f"final cost :
{self.finalCostArr[len(self.finalCostArr)-1]}")
print(self.thetaVect)
def trainData(self) :
self.predictedValVect = self.predictResult()
diffVect = self.predictedValVect - self.actualValVect
partialDerivativeVect = np.matmul(self.desMat.T , diffVect)
self.thetaVect -=
(self.alpha/len(self.desMat))*partialDerivativeVect
def predictResult(self) :
return np.matmul(self.desMat,self.thetaVect)
def costEvaluation(self) :
cost = sum((self.predictedValVect - self.actualValVect)**2)
return cost / (2*len(self.actualValVect))
def graphEvaluation(self) :
plt.title("cost at end of all epochs")
x = range(len(self.epochCostArr))
y = self.epochCostArr
plt.plot(x,y)
plt.xlabel("iterations")
plt.ylabel("cost")
plt.show()
I kept epochs=200 and alpha=0.1 for all runs but I got a totally different result in each run.
The vector mentioned below is the theta vector where the first entry is the bias and remaining are weights
RUN 1 =>>
[[ 5.26020144]
[ -0.48787333]
[ 4.36479114]
[ 4.56848299]
[ 2.90299436]
[ 3.85349625]
[-10.61906207]
[ -0.93178027]
[ 8.79943389]]
final cost : 0.05917831328836957
RUN 2 =>>
[[ 5.18355814]
[ -0.56072668]
[ 4.32621647]
[ 4.58803884]
[ 2.89157598]
[ 3.7465471 ]
[-10.75751065]
[ -1.03302031]
[ 8.87559247]]
final cost: 1.0056239103948563
RUN 3 =>>
[[ 5.12836056]
[ -0.43672936]
[ 4.25664898]
[ 4.53397465]
[ 2.87847224]
[ 3.74693215]
[-10.73960775]
[ -1.00461585]
[ 8.85225402]]
final cost : 0.8214901206702101
RUN 4 =>>
[[ 5.38794798]
[ 0.23695412]
[ 4.43522951]
[ 4.66093372]
[ 2.9460605 ]
[ 4.13390252]
[-10.60071883]
[ -0.9230675 ]
[ 8.87229324]]
final cost: 15.959132174895712
RUN 5 =>>
[[ 5.19643132]
[ -0.76882106]
[ 4.35445135]
[ 4.58782119]
[ 2.8908931 ]
[ 3.63693031]
[-10.83291949]
[ -1.05709616]
[ 8.865904 ]]
final cost: 2.3162151072779804
I am unable to figure out what is going Wrong. Does SGD behave like this or I did some stupidity while transforming my code from batch GD to SGD. And if SGD behaves like this then how I get to know that how many times I have to rerun because I am not so lucky that every time in the first run I got such a small cost like 0.05 sometimes the first run gives cost around 10.5 sometimes 0.6 and maybe rerunning it a lot of times I got cost even smaller than 0.05.
when I approached the exact same problem with exact same code and hyperParameters just replacing the SGD function with normal batch GD I get the expected result i.e, after each iteration over the same data my cost is decreasing smoothly i.e., a monotonic decreasing function and no matter how many times I rerun the same program I got exactly the same result as this is very obvious.
"keeping everything same but using batch GD for epochs=20000 and alpha=0.1
I got final_cost=2.7474"
def BatchGradientDescent(self) :
self.costArr = np.array([])
startTime = time.time()
for i in range(self.epochs) :
tempMat = self.df_trainingData.values
self.actualValVect = tempMat[ : , self.TOTAL_ATTR:]
tempMat = tempMat[ : , :self.TOTAL_ATTR]
self.desMat = np.append( np.ones((self.TRAINING_DATA_SIZE,1),dtype=float) , tempMat , 1 )
del tempMat
self.trainData()
if i%100 == 0 :
currCost = self.costEvaluation()
self.costArr = np.append(self.costArr,currCost)
endTime = time.time()
print(f"execution time : {endTime - startTime} seconds")
self.graphEvaluation()
print(self.thetaVect)
print(f"final cost : {self.costArr[len(self.costArr)-1]}")
SomeBody help me figure out What actually is going on. Every opinion/solution is big revenue for me in this new field :)

You missed the most important and only difference between GD ("Gradient Descent") and SGD ("Stochastic Gradient Descent").
Stochasticity - Literally means "the quality of lacking any predictable order or plan". Meaning randomness.
Which means that while in the GD algorithm, the order of the samples in each epoch remains constant, in SGD the order is randomly shuffled at the beginning of every epochs.
So every run of GD with the same initialization and hyperparameters will produce the exact same results, while SGD will most defiantly not (as you have experienced).
The reason for using stochasticity is to prevent the model from memorizing the training samples (which will results in overfitting, where accuracy on the training set will be high but accuracy on unseen samples will be bad).
Now regarding to the big differences in final cost values between runs at your case, my guess is that your learning rate is too high. You can use a lower constant value, or better yet, use a decaying learning rate (which gets lower as epochs get higher).

Resources