How to use custom mean, median, mode functions with array of 2500 in python? - python-3.x

So I am trying to solve mean, median and mode challenge on Hackerrank. I defined 3 functions to calculate mean, median and mode for a given array with length between 10 and 2500, inclusive.
I get an error with an array of 2500 integers, not sure why. I looked into python documentation and found no mentions of max length for lists. I know I can use statistics module but trying the hard way and being stubborn I guess. Any help and criticism is appreciated regarding my code. Please be honest and brutal if need be. Thanks
N = int(input())
var_list = [int(x) for x in input().split()]
def mean(sample_list):
mean = sum(sample_list)/N
print(mean)
return
def median(sample_list):
sorted_list = sorted(sample_list)
if N%2 != 0:
median = sorted_list[(N//2)]
else:
median = (sorted_list[N//2] + sorted_list[(N//2)-1])/2
print(median)
return
def mode(sample_list):
sorted_list = sorted(sample_list)
mode = min(sorted_list)
max_count = sorted_list.count(mode)
for i in sorted_list:
if (i <= mode) and (sorted_list.count(i) >= max_count):
mode = i
print(mode)
return
mean(var_list)
median(var_list)
mode(var_list)
Compiler Message
Wrong Answer
Input (stdin)
2500
19325 74348 68955 98497 26622 32516 97390 64601 64410 10205 5173 25044 23966 60492 71098 13852 27371 40577 74997 42548 95799 26783 51505 25284 49987 99134 33865 25198 24497 19837 53534 44961 93979 76075 57999 93564 71865 90141 5736 54600 58914 72031 78758 30015 21729 57992 35083 33079 6932 96145 73623 55226 18447 15526 41033 46267 52486 64081 3705 51675 97470 64777 31060 90341 55108 77695 16588 64492 21642 56200 48312 5279 15252 20428 57224 38086 19494 57178 49084 37239 32317 68884 98127 79085 77820 2664 37698 84039 63449 63987 20771 3946 862 1311 77463 19216 57974 73012 78016 9412 90919 40744 24322 68755 59072 57407 4026 15452 82125 91125 99024 49150 90465 62477 30556 39943 44421 68568 31056 66870 63203 43521 78523 58464 38319 30682 77207 86684 44876 81896 58623 24624 14808 73395 92533 4398 8767 72743 1999 6507 49353 81676 71188 78019 88429 68320 59395 95307 95770 32034 57015 26439 2878 40394 33748 41552 64939 49762 71841 40393 38293 48853 81628 52111 49934 74061 98537 83075 83920 42792 96943 3357 83393{-truncated-}
Download to view the full testcase
Expected Output
49921.5
49253.5
2184

Your issue seems to be that you are actually using standard list operations rather than calculating things on the fly, while looping through the data once (for the average). sum(sample_list) will almost surely give you something which exceeds the double-limit, i.a.w. it becomes really big.
Further reading
Calculating the mean, variance, skewness, and kurtosis on the fly
How do I determine the standard deviation (stddev) of a set of values?
Rolling variance algorithm
What is a good solution for calculating an average where the sum of all values exceeds a double's limits?
How do I determine the standard deviation (stddev) of a set of values?
How to efficiently compute average on the fly (moving average)?

I figured out that you forgot to change the max_count variable inside the if block. Probably that causes the wrong result. I tested the debugged version on my computer and they seem to work well when I compare their result with the scipy's built-in functions. The correct mode function should be
def mode(sample_list):
N = len(sample_list)
sorted_list = sorted(sample_list)
mode = min(sorted_list)
max_count = sorted_list.count(mode)
for i in sorted_list:
if (sorted_list.count(i) >= max_count):
mode = i
max_count = sorted_list.count(i)
print(mode)

I was busy with some stuff and now came back to completing this. I am happy to say that I have matured enough as a coder and solved this issue.
Here is the solution:
# Enter your code here. Read input from STDIN. Print output to STDOUT
# Input an array of numbers, convert it to integer array
n = int(input())
my_array = list(map(int, input().split()))
my_array.sort()
# Find mean
array_mean = sum(my_array) / n
print(array_mean)
# Find median
if (n%2) != 0:
array_median = my_array[n//2]
else:
array_median = (my_array[n//2 - 1] + my_array[n//2]) / 2
print(array_median)
# Find mode(I could do this using multimode method of statistics module for python 3.8)
def sort_second(array):
return array[1]
modes = [[i, my_array.count(i)] for i in my_array]
modes.sort(key = sort_second, reverse=True)
array_mode = modes[0][0]
print(array_mode)

Related

Generate interleavings of two strings in lexicographical order in Python

How to generate interleavings of two strings in lexicograhical order in python?
I was able to generate interleavings, but not in lexicographical order.
The input given was,
Input :
2
nkb gl
bn zh
Expected output:
Case #1:
glnkb
gnkbl
gnklb
gnlkb
ngkbl
ngklb
nglkb
nkbgl
nkgbl
nkglb
Case #2:
bnzh
bzhn
bznh
zbhn
zbnh
zhbn
This is my code --
def interleave(A,B,ans,m,n,idx):
if m == 0 and n == 0:
print("".join(ans))
return
if len(A)<=len(B):
if m!=0:
ans[idx]=A[0]
interleave(A[1:],B,ans,m-1,n,idx+1)
if n!=0:
ans[idx]=B[0]
interleave(A,B[1:],ans,m,n-1,idx+1)
else:
if n!=0:
ans[idx]=B[0]
interleave(A,B[1:],ans,m,n-1,idx+1)
if m!=0:
ans[idx]=A[0]
interleave(A[1:],B,ans,m-1,n,idx+1)
t=int(input())
count=0
for i in range(t):
count+=1
print("Case #%d:"%count)
A,B=input().split()
m,n=len(A),len(B)
ans=['']*(m+n)
idx=0
interleave(A,B,ans,m,n,idx)
Output of the code that I wrote was--
Case #1:
glnkb
gnlkb
gnkbl
gnklb
nkbgl
nkgbl
nkglb
nglkb
ngkbl
ngklb
Case #2:
bnzh
bznh
bzhn
zhbn
zbnh
zbhn
There is some problem in the logic. Please help me to figure it out.

Pan Tompkins Lowpass filter overflow

The Pan Tompkins algorithm1 for removing noise from an ECG/EKG is cited often. They use a low pass filter, followed by a high pass filter. The output of the high pass filter looks great. But (depending on starting conditions) the output of the low pass filter will continuously increase or decrease. Given enough time, your numbers will eventually get to a size that the programming language cannot handle and rollover. If I run this on an Arduino (which uses a variant of C), it rolls over on the order of 10 seconds. Not ideal. Is there a way to get rid of this bias? I've tried messing with initial conditions, but I'm fresh out of ideas. The advantage of this algorithm is that it's not very computationally intensive and will run comfortably on a modest microprocessor.
1 Pan, Jiapu; Tompkins, Willis J. (March 1985). "A Real-Time QRS Detection Algorithm". IEEE Transactions on Biomedical Engineering. BME-32 (3): 230–236.
Python code to illustrate problem. Uses numpy and matplotlib:
import numpy as np
import matplotlib.pyplot as plt
#low-pass filter
def lpf(x):
y = x.copy()
for n in range(len(x)):
if(n < 12):
continue
y[n,1] = 2*y[n-1,1] - y[n-2,1] + x[n,1] - 2*x[n-6,1] + x[n-12,1]
return y
#high-pass filter
def hpf(x):
y = x.copy()
for n in range(len(x)):
if(n < 32):
continue
y[n,1] = y[n-1,1] - x[n,1]/32 + x[n-16,1] - x[n-17,1] + x[n-32,1]/32
return y
ecg = np.loadtxt('ecg_data.csv', delimiter=',',skiprows=1)
plt.plot(ecg[:,0], ecg[:,1])
plt.title('Raw Data')
plt.grid(True)
plt.savefig('raw.png')
plt.show()
#Application of lpf
f1 = lpf(ecg)
plt.plot(f1[:,0], f1[:,1])
plt.title('After Pan-Tompkins LPF')
plt.xlabel('time')
plt.ylabel('mV')
plt.grid(True)
plt.savefig('lpf.png')
plt.show()
#Application of hpf
f2 = hpf(f1[16:,:])
print(f2[-300:-200,1])
plt.plot(f2[:-100,0], f2[:-100,1])
plt.title('After Pan-Tompkins LPF+HPF')
plt.xlabel('time')
plt.ylabel('mV')
plt.grid(True)
plt.savefig('hpf.png')
plt.show()
raw data in CSV format:
timestamp,ecg_measurement
96813044,2.2336266040
96816964,2.1798632144
96820892,2.1505377292
96824812,2.1603128910
96828732,2.1554253101
96832660,2.1163244247
96836580,2.0576734542
96840500,2.0381231307
96844420,2.0527858734
96848340,2.0674486160
96852252,2.0283479690
96856152,1.9648094177
96860056,1.9208210945
96863976,1.9159335136
96867912,1.9208210945
96871828,1.8768328666
96875756,1.7986314296
96879680,1.7448680400
96883584,1.7155425548
96887508,1.7057673931
96891436,1.6520038604
96895348,1.5591397285
96899280,1.4809384346
96903196,1.4467253684
96907112,1.4369501113
96911032,1.3978494453
96914956,1.3440860509
96918860,1.2952101230
96922788,1.3000977039
96926684,1.3343108892
96930604,1.3440860509
96934516,1.3489736318
96938444,1.3294233083
96942364,1.3782991170
96946284,1.4222873687
96950200,1.4516129493
96954120,1.4369501113
96958036,1.4320625305
96961960,1.4565005302
96965872,1.4907135963
96969780,1.5053763389
96973696,1.4613881111
96977628,1.4125122070
96981548,1.4076246261
96985476,1.4467253684
96989408,1.4809384346
96993324,1.4760508537
96997236,1.4711632728
97001160,1.4907135963
97005084,1.5444769859
97008996,1.5982404708
97012908,1.5835777282
97016828,1.5591397285
97020756,1.5786901473
97024676,1.6324535369
97028604,1.6911046504
97032516,1.6959922313
97036444,1.6764417648
97040364,1.6813293457
97044296,1.7155425548
97048216,1.7448680400
97052120,1.7253177165
97056048,1.6911046504
97059968,1.6911046504
97063880,1.7302052974
97067796,1.7741935253
97071724,1.7693059444
97075644,1.7350928783
97079564,1.7595307826
97083480,1.8719452857
97087396,2.0381231307
97091316,2.2482893466
97095244,2.4828934669
97099156,2.7468230724
97103088,2.9960899353
97106996,3.0987291336
97110912,2.9178886413
97114836,2.5171065330
97118756,2.0185728073
97122668,1.5053763389
97126584,1.1094819307
97130492,0.8015640258
97134396,0.5767350673
97138308,0.4545454502
97142212,0.4349951267
97146124,0.4692081928
97150020,0.4887585639
97153924,0.4594330310
97157828,0.4105571746
97161740,0.3861192512
97165660,0.3763440847
97169580,0.3714565038
97173492,0.3225806236
97177404,0.2639296054
97181316,0.2394916772
97185236,0.2297165155
97189148,0.2443792819
97193060,0.2248289346
97196972,0.1857282543
97200900,0.1808406734
97204812,0.2199413537
97208732,0.2492668628
97212652,0.2443792819
97216572,0.2199413537
97220484,0.2248289346
97224404,0.2834799575
97228316,0.3274682044
97232228,0.3665689229
97236132,0.3861192512
97240036,0.4398827075
97243936,0.5083088874
97247836,0.6109481811
97251748,0.7086998939
97255660,0.7771260738
97259568,0.8553275108
97263476,0.9775171279
97267392,1.1094819307
97271308,1.1974585056
97275228,1.2512218952
97279148,1.2952101230
97283056,1.3734115362
97286992,1.4760508537
97290900,1.5493645668
97294820,1.5738025665
97298740,1.5982404708
97302652,1.6471162796
97306584,1.7106549739
97310500,1.7546432018
97314420,1.7546432018
97318340,1.7644183635
97322272,1.8084066390
97326168,1.8621701240
97330072,1.8963831901
97333988,1.8817204475
97337912,1.8572825431
97341840,1.8670577049
97345748,1.8866080284
97349668,1.8768328666
97353580,1.8230694770
97357500,1.7595307826
97361424,1.7302052974
97365332,1.7350928783
97369252,1.6959922313
97373168,1.6226783752
97377092,1.5298142433
97381012,1.4613881111
97384940,1.4320625305
97388860,1.4076246261
97392780,1.3440860509
97396676,1.2658846378
97400604,1.2121212482
97404532,1.1974585056
97408444,1.1779080629
97412356,1.1192570924
97416264,1.0361680984
97420164,0.9628542900
97424068,0.9286412239
97427988,0.9042033195
97431892,0.8406646728
97435804,0.7575757503
97439708,0.6940371513
97443628,0.6793744087
97447540,0.6793744087
97451452,0.6549364089
97455356,0.6060606002
97459240,0.5767350673
97463140,0.6011730194
97467044,0.6451612472
97470964,0.6842619895
97474884,0.6891495704
97478796,0.7184750556
97482700,0.8064516067
97486612,0.8846529960
97490516,0.9335288047
97494428,0.9530791282
97498340,0.9481915473
97502256,0.9726295471
97506156,0.9921798706
97510060,0.9726295471
97513980,0.8846529960
97517884,0.7869012832
97521796,0.7086998939
97525692,0.6549364089
97529604,0.5913978099
97533516,0.4887585639
97537428,0.3567937374
97541348,0.2639296054
97545260,0.2003910064
97549148,0.1417399787
97553060,0.0928641223
97556980,0.0537634420
97560892,0.0342130994
97564804,0.0146627559
97568708,0.0244379281
97572628,0.0048875851
97576500,0.0000000000
97580324,0.0000000000
97584172,0.0097751703
97588060,0.0244379281
97591980,0.0195503416
97595900,0.0146627559
97599812,0.0488758563
97603724,0.1319648027
97607628,0.2248289346
97611548,0.3030303001
97615444,0.3665689229
97619364,0.4496578693
97623276,0.5718474864
97627176,0.7038123130
97631076,0.8064516067
97634988,0.8699902534
97638900,0.9384163856
97642816,1.0361680984
97646720,1.1485825777
97650644,1.2365591526
97654572,1.2658846378
97658492,1.2805473804
97662404,1.3294233083
97666324,1.3782991170
97670244,1.3831867027
97674148,1.3489736318
97678068,1.3049852848
97681988,1.2903225421
97685908,1.3000977039
97689812,1.3098728656
97693728,1.2463343143
97697648,1.1876833438
97701568,1.1681329011
97705488,1.1876833438
97709412,1.1827956390
97713328,1.1339198350
97717244,1.0752688646
97721144,1.0557184219
97725056,1.0703812837
97728972,1.0850440216
97732872,1.0752688646
97736788,1.0459432601
97740696,1.0508308410
97744600,1.0948191833
97748520,1.1290322542
97752444,1.1192570924
97756364,1.0850440216
97760272,1.0801564407
97764168,1.1094819307
97768072,1.1339198350
97771996,1.1143695116
97775920,1.0557184219
97779840,1.0166177749
97783756,0.9970674514
97787668,0.9921798706
97791580,0.9530791282
97795500,0.8846529960
97799412,0.8504399299
97803316,0.8455523490
97807212,0.8699902534
97811124,0.8699902534
97815028,0.8308895111
97818940,0.8064516067
97822844,0.8211143493
97826756,0.8651026725
97830668,0.9042033195
97834572,0.8895405769
97838476,0.8993157386
97842396,0.9530791282
97846304,1.0410556793
97850204,1.0850440216
97854112,1.0899316024
97858020,1.1192570924
97861940,1.2267839908
97865860,1.4320625305
97869772,1.6911046504
97873688,1.9892473220
97877604,2.3020527362
97881524,2.6197457313
97885452,2.8299119949
97889372,2.7761485576
97893292,2.4535679817
97897216,1.9745845794
97901136,1.4956011772
97905052,1.0899316024
97908964,0.8260019302
97912864,0.6695992469
97916772,0.6353860855
97920684,0.7038123130
97924588,0.8553275108
97928496,1.0215053558
97932408,1.1388074159
97936324,1.2023460865
97940228,1.2463343143
97944148,1.3098728656
97948064,1.3734115362
97951988,1.3929618644
97955912,1.3734115362
97959836,1.3636363744
97963764,1.3782991170
97967684,1.4173997879
97971612,1.4173997879
97975540,1.3880742835
97979460,1.3831867027
97983372,1.4027370452
97987292,1.4467253684
97991216,1.4565005302
97995124,1.4320625305
97999036,1.4173997879
98002964,1.4662756919
98006884,1.5249266624
98010812,1.5689149856
98014740,1.5689149856
98018668,1.5689149856
98022592,1.6129032135
98026516,1.6715541839
98030436,1.6911046504
98034348,1.6617790222
98038268,1.6422286987
98042192,1.6715541839
98046124,1.7204301357
98050032,1.7399804592
98053952,1.7155425548
98057880,1.6911046504
98061800,1.7106549739
98065716,1.7595307826
98069636,1.7937438488
98073556,1.7888562679
98077476,1.7741935253
98081408,1.8084066390
98085312,1.8719452857
98089228,1.9305962562
98093144,1.9257086753
98097048,1.9257086753
98100968,1.9599218368
98104884,2.0332355499
98108804,2.0967741012
98112724,2.1016616821
98116652,2.0869989395
98120564,2.0967741012
98124484,2.1456501483
98128404,2.1847507953
98132324,2.1749756336
98136252,2.1212120056
98140156,2.0967741012
98144068,2.1114368438
98147996,2.0967741012
98151908,2.0430107116
98155824,1.9501466751
98159748,1.8817204475
98163664,1.8475073814
98167584,1.8377322196
98171508,1.7937438488
98175440,1.7253177165
98179364,1.7057673931
98183296,1.7106549739
98187200,1.7448680400
98191108,1.7546432018
98195032,1.7302052974
98198952,1.7302052974
98202868,1.7741935253
98206800,1.8426198005
98210712,1.8866080284
98214628,1.8914956092
98218548,1.8914956092
98222472,1.9403715133
98226396,2.0087976455
98230308,2.0527858734
98234212,2.0527858734
98238132,2.0527858734
98242044,2.0869989395
98245964,2.1407625675
98249892,2.1798632144
98253812,2.1749756336
98257740,2.1652004718
98261660,2.1896383762
98265588,2.2385141849
98269516,2.2678396701
98273444,2.2385141849
98277364,2.1896383762
98281292,2.1798632144
98285212,2.1994135379
98289140,2.2091886997
98293052,2.1798632144
98296980,2.1212120056
98300892,2.0918865203
98304804,2.1114368438
98308732,2.1163244247
98312660,2.0674486160
98316572,2.0087976455
98320480,1.9892473220
98324392,1.9892473220
98328308,2.0087976455
98332216,1.9892473220
98336132,1.9501466751
98340048,1.9354838371
98343972,1.9696969985
98347888,1.9843597412
98351812,1.9696969985
98355736,1.9159335136
98359664,1.8866080284
98363576,1.9012707710
98367484,1.9305962562
98371408,1.9208210945
98375324,1.8817204475
98379240,1.8719452857
98383156,1.8817204475
98387072,1.9305962562
98390984,1.9403715133
98394904,1.9159335136
98398832,1.9012707710
98402744,1.9354838371
98406672,1.9794721603
98410584,1.9941349029
98414492,1.9696969985
98418416,1.9550342559
98422336,1.9843597412
98426260,2.0430107116
98430164,2.0723361968
98434076,2.0527858734
98437988,2.0381231307
98441900,2.0625610351
98445820,2.1065492630
98449740,2.1309874057
98453660,2.1065492630
98457572,2.0869989395
98461492,2.0918865203
98465404,2.1456501483
98469324,2.1847507953
98473236,2.1749756336
98477148,2.1505377292
98481052,2.1652004718
98484972,2.1945259571
98488900,2.2287390232
98492820,2.2091886997
98496732,2.1700880527
98500644,2.1652004718
98504556,2.2091886997
98508476,2.2531769275
98512404,2.2336266040
98516324,2.1994135379
98520244,2.2043011188
98524152,2.2531769275
98528068,2.2873899936
98531988,2.2727272510
98535908,2.2238514423
98539836,2.1994135379
98543764,2.2336266040
98547676,2.2580645084
98551588,2.2482893466
98555508,2.1994135379
98559436,2.1652004718
98563356,2.1603128910
98567268,2.1700880527
98571164,2.1309874057
98575068,2.0527858734
98578992,1.9843597412
98582920,1.9648094177
98586840,1.9696969985
98590756,1.9501466751
98594680,1.8963831901
98598596,1.8523949623
98602528,1.8572825431
98606456,1.8621701240
98610376,1.8670577049
98614292,1.8328446388
98618204,1.8132943153
98622132,1.8426198005
98626048,1.8963831901
98629968,1.9257086753
98633892,1.8914956092
98637808,1.8670577049
98641716,1.8914956092
98645640,1.9941349029
98649556,2.1456501483
98653476,2.3313782215
98657404,2.5708699226
98661316,2.8885631561
98665236,3.2306940555
98669148,3.4799609184
98673064,3.4604105949
98676972,3.1769306659
98680884,2.7614858150
98684796,2.2678396701
98688720,1.8230694770
98692628,1.4418377876
98696556,1.2023460865
98700476,1.1241446733
98704400,1.1876833438
98708316,1.3098728656
98712228,1.4125122070
98716148,1.4858260154
98720060,1.5493645668
98723988,1.6275659561
98727908,1.6764417648
98731836,1.6911046504
98735748,1.6617790222
98739676,1.6471162796
98743608,1.6715541839
98747532,1.7057673931
98751452,1.7057673931
98755372,1.6568914413
98759300,1.6275659561
98763220,1.6422286987
98767140,1.6862169265
98771056,1.6911046504
98774964,1.6617790222
98778884,1.6568914413
98782812,1.6911046504
98786728,1.7448680400
98790632,1.7790811061
98794544,1.7693059444
98798452,1.7644183635
98802384,1.7888562679
98806312,1.8279570579
98810224,1.8426198005
98814132,1.8181818962
98818044,1.7986314296
98821972,1.8181818962
98825900,1.8523949623
98829832,1.8719452857
98833760,1.8377322196
98837684,1.8035190582
98841596,1.7986314296
98845528,1.8377322196
98849456,1.8670577049
98853368,1.8523949623
98857292,1.8181818962
98861220,1.8328446388
98865140,1.8866080284
98869048,1.9305962562
98872968,1.9305962562
98876888,1.9012707710
98880800,1.9208210945
98884704,1.9599218368
98888624,1.9892473220
98892544,1.9599218368
98896464,1.8866080284
98900376,1.8426198005
98904296,1.8377322196
98908216,1.8328446388
98912132,1.7839686870
98916040,1.7008798122
98919956,1.6471162796
98923884,1.6373411178
98927812,1.6324535369
98931740,1.5982404708
98935644,1.5151515007
98939564,1.4613881111
98943492,1.4418377876
98947424,1.4271749496
98951348,1.3685239553
98955260,1.2707722187
98959180,1.1925709247
98963092,1.1534701585
98967008,1.1339198350
98970932,1.1045943498
98974840,1.0215053558
98978748,0.9677418708
98982660,0.9579667091
98986572,0.9775171279
98990492,0.9824047088
98994396,0.9237536430
98998308,0.8748778343
99002212,0.8797654151
99006132,0.9188660621
99010036,0.9286412239
99013956,0.9090909004
99017836,0.8895405769
99021740,0.9042033195
99025644,0.9530791282
99029556,0.9921798706
99033468,0.9970674514
99037380,0.9872922897
99041296,1.0166177749
99045196,1.0752688646
99049100,1.1192570924
99053016,1.1290322542
99056940,1.0997067642
99060840,1.1094819307
99064744,1.1485825777
99068668,1.1925709247
99072588,1.2023460865
99076508,1.1925709247
99080428,1.2023460865
99084340,1.2658846378
99088252,1.3343108892
99092168,1.3587487936
99096084,1.3343108892
99100004,1.3294233083
99103924,1.3636363744
99107860,1.4027370452
99111772,1.3831867027
99115700,1.3343108892
99119628,1.3147605657
99123556,1.3343108892
99127480,1.3587487936
99131404,1.3538612127
99135324,1.3049852848
99139236,1.2756597995
99143156,1.2903225421
99147076,1.3196481466
99151012,1.3147605657
99154924,1.2707722187
99158844,1.2072336673
99162764,1.2023460865
99166676,1.2267839908
99170588,1.2365591526
99174508,1.1974585056
99178420,1.1632453203
99182340,1.1534701585
99186248,1.1876833438
99190164,1.1974585056
99194072,1.1583577394
99197996,1.1192570924
99201916,1.1192570924
99205832,1.1730204820
99209748,1.2072336673
99213668,1.2023460865
99217588,1.1779080629
99221488,1.1876833438
99225412,1.2267839908
99229332,1.2707722187
99233244,1.2609970569
99237152,1.2365591526
99241068,1.2463343143
99244988,1.2805473804
99248900,1.2952101230
99252820,1.2805473804
99256732,1.2316715717
99260660,1.2316715717
99264588,1.2854349613
99268512,1.3391984701
99272436,1.3538612127
99276364,1.3343108892
99280292,1.3391984701
99284212,1.3782991170
99288116,1.4271749496
99292040,1.4369501113
99295964,1.4076246261
99299892,1.4076246261
99303816,1.4662756919
99307740,1.5395894050
99311652,1.5640274047
99315564,1.5444769859
99319484,1.5444769859
99323412,1.5786901473
99327332,1.6275659561
99331252,1.6520038604
99335156,1.6422286987
99339076,1.6275659561
99343004,1.6422286987
99346924,1.6666666030
99350844,1.6568914413
99354764,1.6031280517
99358676,1.5542521476
99362604,1.5542521476
99366532,1.5835777282
99370460,1.5982404708
99374372,1.5835777282
99378300,1.5640274047
99382204,1.5835777282
99386132,1.6373411178
99390056,1.6715541839
99393980,1.6520038604
99397892,1.6275659561
99401812,1.6422286987
99405736,1.6862169265
99409664,1.7106549739
99413580,1.6911046504
99417500,1.6568914413
99421432,1.6715541839
99425348,1.7204301357
99429256,1.8084066390
99433164,1.9208210945
99437068,2.0918865203
99440980,2.3655912876
99444912,2.7321603298
99448828,3.0596284866
99452752,3.2453567981
99456680,3.1867058277
99460600,2.9374389648
99464516,2.5610947608
99468428,2.1163244247
99472356,1.6813293457
99476284,1.3343108892
99480200,1.1436949968
99484112,1.1339198350
99488036,1.2365591526
99491956,1.3440860509
99495864,1.4320625305
99499780,1.5298142433
99503708,1.6422286987
99507636,1.7350928783
99511556,1.7644183635
99515480,1.7399804592
99519396,1.7350928783
99523320,1.7448680400
99527220,1.7350928783
99531140,1.6862169265
99535064,1.5933528900
99538980,1.5102639198
99542892,1.4711632728
99546820,1.4467253684
99550748,1.3978494453
99554668,1.3049852848
99558588,1.2072336673
99562504,1.1485825777
99566428,1.1192570924
99570348,1.0752688646
99574256,1.0068426132
99578176,0.9384163856
99582084,0.9188660621
99585988,0.9188660621
99589900,0.9188660621
99593812,0.8895405769
99597716,0.8748778343
99601636,0.8651026725
99605552,0.9090909004
99609436,0.9481915473
99613356,0.9530791282
99617268,0.9237536430
99621180,0.9335288047
99625080,1.0019550323
99628980,1.0752688646
99632888,1.0801564407
99636792,1.0703812837
99640704,1.0899316024
99644616,1.1436949968
99648536,1.2170088291
99652444,1.2170088291
99656356,1.2023460865
99660268,1.2072336673
99664180,1.2561094760
99668084,1.3000977039
99671980,1.3147605657
99675900,1.2952101230
99679820,1.3000977039
99683728,1.3587487936
99687652,1.4027370452
99691568,1.4222873687
99695484,1.3978494453
99699404,1.3880742835
99703328,1.4173997879
99707248,1.4565005302
99711156,1.4760508537
99715064,1.4271749496
99718988,1.3929618644
99722908,1.3929618644
99726828,1.4076246261
99730748,1.3831867027
99734668,1.3147605657
99738580,1.2561094760
99742492,1.2414467334
99746420,1.2658846378
99750340,1.2658846378
99754252,1.2365591526
99758168,1.2121212482
99762084,1.2365591526
99766012,1.3000977039
99769916,1.3538612127
99773856,1.3685239553
99777780,1.3929618644
99781704,1.4662756919
99785620,1.5640274047
99789532,1.6568914413
99793460,1.6959922313
99797392,1.7057673931
99801312,1.7399804592
99805228,1.7937438488
99809148,1.8377322196
99813072,1.8377322196
99816996,1.8230694770
99820920,1.8475073814
99824840,1.9061583518
99828756,1.9501466751
99832680,1.9599218368
99836608,1.9501466751
99840536,1.9599218368
99844452,2.0087976455
99848364,2.0527858734
99852268,2.0527858734
99856184,2.0283479690
99860092,2.0185728073
99864012,2.0576734542
99867932,2.0967741012
99871836,2.0869989395
99875740,2.0478982925
99879652,2.0234603881
99883564,2.0527858734
99887484,2.1065492630
99891404,2.1163244247
99895332,2.0772237777
99899236,2.0527858734
99903156,2.0821113586
99907076,2.1065492630
99910996,2.1016616821
99914916,2.0576734542
99918828,2.0283479690
99922740,2.0430107116
99926652,2.0821113586
99930572,2.1016616821
99934492,2.0576734542
99938404,2.0332355499
99942316,2.0674486160
99946220,2.1309874057
99950124,2.1749756336
99954052,2.1652004718
99957972,2.1260998249
99961892,2.1456501483
99965804,2.1945259571
99969732,2.2336266040
99973644,2.2189638614
99977564,2.1945259571
99981492,2.2043011188
99985404,2.2482893466
99989332,2.2922775745
99993252,2.2580645084
99997164,2.2238514423
100001084,2.2189638614
100005044,2.2678396701
100009004,2.2776148319
100012956,2.2385141849
100016924,2.1798632144
100020892,2.1603128910
100024844,2.1798632144
100028804,2.2140762805
100032756,2.1798632144
100036716,2.1358749866
100040676,2.1163244247
100044644,2.1358749866
100048604,2.1603128910
100052556,2.1407625675
100056516,2.0967741012
100060468,2.0918865203
100064420,2.1163244247
100068384,2.1407625675
100072340,2.1065492630
100076292,2.0478982925
100080244,2.0332355499
100084196,2.0478982925
100088156,2.0674486160
100092100,2.0332355499
100096056,1.9696969985
100100004,1.9110459327
100103968,1.9061583518
100107928,1.9208210945
100111884,1.8768328666
100115844,1.8181818962
100119812,1.7888562679
100123776,1.8084066390
100127712,1.8475073814
100131672,1.8523949623
100135636,1.8181818962
100139604,1.8035190582
100143560,1.8377322196
100147524,1.8768328666
100151488,1.8719452857
100155448,1.8523949623
100159404,1.8132943153
100163376,1.8426198005
100167328,1.8963831901
100171276,1.9110459327
100175232,1.9061583518
100179188,1.9501466751
100183132,2.1016616821
100187084,2.3216030597
100191036,2.5904202461
100194996,2.8787879943
100198956,3.1769306659
100202916,3.4555230140
100206876,3.5826001167
100210836,3.4115347862
100214804,3.0205278396
100218748,2.5317692756
100222708,2.1016616821
100226676,1.7937438488
100230640,1.5933528900
100234592,1.4858260154
100238548,1.5053763389
100242500,1.6422286987
100246456,1.8377322196
100250424,1.9990224838
100254380,2.0967741012
100258340,2.1652004718
100262284,2.2434017658
100266236,2.3411533832
100270196,2.4242424964
100274140,2.4731183052
100278112,2.4975562095
100282068,2.5562071800
100286020,2.6441838741
100289992,2.7028348445
100293948,2.7077224254
100297908,2.7126100063
100301868,2.7468230724
100305820,2.8054740905
100309764,2.8250244140
100313724,2.7908113002
100317676,2.7370479106
100321636,2.7223851680
100325600,2.7419354915
100329556,2.7517106533
100333516,2.7126100063
100337468,2.6735093593
100341428,2.6686217784
100345392,2.7028348445
100349348,2.7272727489
100353316,2.6881721019
100357260,2.6441838741
100361212,2.6588466167
100365180,2.6832845211
100369140,2.7077224254
100373100,2.6783969402
100377052,2.6148581504
100381012,2.6001954078
100384960,2.6246333122
100388916,2.6490714550
100392860,2.6197457313
100396828,2.5659823417
100400788,2.5562071800
100404740,2.5806450843
100408692,2.6099705696
100412644,2.5904202461
100416588,2.5366568565
100420548,2.5268816947
100424508,2.5610947608
100428460,2.5953078269
100432412,2.5757575035
100436372,2.5171065330
100440324,2.4926686286
100444276,2.5219941139
100448228,2.5366568565
100452196,2.5073313713
100456148,2.4389052391
100460108,2.3949170112
100464068,2.3753666877
100468028,2.3655912876
100471988,2.3069403171
The trick seems to be the initial conditions. Load the first 13 values of input and output of low pass filter to zero and the bias goes away.
#low-pass filter
def lpf(x):
y = x.copy()
for n in range(13):
y[n,1] = 0
x[n,1] = 0
for n in range(len(x)):
if(n < 12):
continue
y[n,1] = 2*y[n-1,1] - y[n-2,1] + x[n,1] - 2*x[n-6,1] + x[n-12,1]
return y

PuLP solvers do not respond to options fed to them

So I've got a fairly large optimization problem and I'm trying to solve it within a sensible amount of time.
Ive set it up as:
import pulp as pl
my_problem = LpProblem("My problem",LpMinimize)
# write to problem file
my_problem.writeLP("MyProblem.lp")
And then alternatively
solver = CPLEX_CMD(timeLimit=1, gapRel=0.1)
status = my_problem .solve(solver)
solver = pl.apis.CPLEX_CMD(timeLimit=1, gapRel=0.1)
status = my_problem .solve(solver)
path_to_cplex = r'C:\Program Files\IBM\ILOG\CPLEX_Studio1210\cplex\bin\x64_win64\cplex.exe' # and yes this is the actual path on my machine
solver = pl.apis.cplex_api.CPLEX_CMD(timeLimit=1, gapRel=0.1, path=path_to_cplex)
status = my_problem .solve(solver)
solver = pl.apis.cplex_api.CPLEX_CMD(timeLimit=1, gapRel=0.1, path=path_to_cplex)
status = my_problem .solve(solver)
It runs in each case.
However, the solver does not repond to the timeLimit or gapRel instructions.
If I use timelimit it does warn this is depreciated for timeLimit. Same for fracgap: it tells me I should use relGap. So somehow I am talking to the solver.
However, nor matter what values i pick for timeLimit and relGap, it always returns the exact same answer and takes the exact same amount of time (several minutes).
Also, I have tried alternative solvers, and I cannot get any one of them to accept their variants of time limits or optimization gaps.
In each case, the problem solves and returns an status: optimal message. But it just ignores the time limit and gap instructions.
Any ideas?
out of the zoo example:
import pulp
import cplex
bus_problem = pulp.LpProblem("bus", pulp.LpMinimize)
nbBus40 = pulp.LpVariable('nbBus40', lowBound=0, cat='Integer')
nbBus30 = pulp.LpVariable('nbBus30', lowBound=0, cat='Integer')
# Objective function
bus_problem += 500 * nbBus40 + 400 * nbBus30, "cost"
# Constraints
bus_problem += 40 * nbBus40 + 30 * nbBus30 >= 300
solver = pulp.CPLEX_CMD(options=['set timelimit 40'])
bus_problem.solve(solver)
print(pulp.LpStatus[bus_problem.status])
for variable in bus_problem.variables():
print ("{} = {}".format(variable.name, variable.varValue))
Correct way to pass solver option as dictionary
pulp.CPLEX_CMD(options={'timelimit': 40})
#Alex Fleisher has it correct with pulp.CPLEX_CMD(options=['set timelimit 40']). This also works for CBC using the following syntax:
prob.solve(COIN_CMD(options=['sec 60','Presolve More','Multiple 15', 'Node DownFewest','HEUR on', 'Round On','PreProcess Aggregate','PassP 10','PassF 40','Strong 10','Cuts On', 'Gomory On', 'CutD -1', 'Branch On', 'Idiot -1', 'sprint -1','Reduce On','Two On'],msg=True)).
It is important to understand that the parameters, and associated options, are specific to a solver. PuLP seems to be calling CBC via the command line so an investigation of those things is required. Hope that helps

Parsing heterogenous data from a text file in Python

I am trying to parse raw data results from a text file into an organised tuple but having trouble getting it right.
My raw data from the textfile looks something like this:
Episode Cumulative Results
EpisodeXD0281119
Date collected21/10/2019
Time collected10:00
Real time PCR for M. tuberculosis (Xpert MTB/Rif Ultra):
PCR result Mycobacterium tuberculosis complex NOT detected
Bacterial Culture:
Bottle: Type FAN Aerobic Plus
Result No growth after 5 days
EpisodeST32423457
Date collected23/02/2019
Time collected09:00
Gram Stain:
Neutrophils Occasional
Gram positive bacilli Moderate (2+)
Gram negative bacilli Numerous (3+)
Gram negative cocci Moderate (2+)
EpisodeST23423457
Date collected23/02/2019
Time collected09:00
Bacterial Culture:
A heavy growth of
1) Klebsiella pneumoniae subsp pneumoniae (KLEPP)
ensure that this organism does not spread in the ward/unit.
A heavy growth of
2) Enterococcus species (ENCSP)
Antibiotic/Culture KLEPP ENCSP
Trimethoprim-sulfam R
Ampicillin / Amoxic R S
Amoxicillin-clavula R
Ciprofloxacin R
Cefuroxime (Parente R
Cefuroxime (Oral) R
Cefotaxime / Ceftri R
Ceftazidime R
Cefepime R
Gentamicin S
Piperacillin/tazoba R
Ertapenem R
Imipenem S
Meropenem R
S - Sensitive ; I - Intermediate ; R - Resistant ; SDD - Sensitive Dose Dependant
Comment for organism KLEPP:
** Please note: this is a carbapenem-RESISTANT organism. Although some
carbapenems may appear susceptible in vitro, these agents should NOT be used as
MONOTHERAPY in the treatment of this patient. **
Please isolate this patient and practice strict contact precautions. Please
inform Infection Prevention and Control as contact screening might be
indicated.
For further advice on the treatment of this isolate, please contact.
The currently available laboratory methods for performing colistin
susceptibility results are unreliable and may not predict clinical outcome.
Based on published data and clinical experience, colistin is a suitable
therapeutic alternative for carbapenem resistant Acinetobacter spp, as well as
carbapenem resistant Enterobacteriaceae. If colistin is clinically indicated,
please carefully assess clinical response.
EpisodeST234234057
Date collected23/02/2019
Time collected09:00
Authorised by xxxx on 27/02/2019 at 10:35
MIC by E-test:
Organism Klebsiella pneumoniae (KLEPN)
Antibiotic Meropenem
MIC corrected 4 ug/mL
MIC interpretation Resistant
Antibiotic Imipenem
MIC corrected 1 ug/mL
MIC interpretation Sensitive
Antibiotic Ertapenem
MIC corrected 2 ug/mL
MIC interpretation Resistant
EpisodeST23423493
Date collected18/02/2019
Time collected03:15
Potassium 4.4 mmol/L 3.5 - 5.1
EpisodeST45445293
Date collected18/02/2019
Time collected03:15
Creatinine 32 L umol/L 49 - 90
eGFR (MDRD formula) >60 mL/min/1.73 m2
Creatinine 28 L umol/L 49 - 90
eGFR (MDRD formula) >60 mL/min/1.73 m2
Essentially the pattern is that ALL information starts with a unique EPISODE NUMBER and follows with a DATE and TIME and then the result of whatever test. This is the pattern throughout.
What I am trying to parse into my tuple is the date, time, name of the test and the result - whatever it might be. I have the following code:
with open(filename) as f:
data = f.read()
data = data.splitlines()
DS = namedtuple('DS', 'date time name value')
parsed = list()
idx_date = [i for i, r in enumerate(data) if r.strip().startswith('Date')]
for start, stop in zip(idx_date[:-1], idx_date[1:]):
chunk = data[start:stop]
date = time = name = value = None
for row in chunk:
if not row: continue
row = row.strip()
if row.startswith('Episode'): continue
if row.startswith('Date'):
_, date = row.split()
date = date.replace('collected', '')
elif row.startswith('Time'):
_, time = row.split()
time = time.replace('collected', '')
else:
name, value, *_ = row.split()
print (name)
parsed.append(DS(date, time, name, value))
print(parsed)
My error is that I am unable to find a way to parse the heterogeneity of the test RESULT in a way that I can use later, for example for the tuple DS ('DS', 'date time name value'):
DATE = 21/10/2019
TIME = 10:00
NAME = Real time PCR for M tuberculosis or Potassium
RESULT = Negative or 4.7
Any advice appreciated. I have hit a brick wall.

Calculate the average of Spearman correlation

I have 2 columns A and B which contain the Spearman's correlation values as follows:
0.127272727 -0.260606061
-0.090909091 -0.224242424
0.345454545 0.745454545
0.478787879 0.660606061
-0.345454545 -0.333333333
0.151515152 -0.127272727
0.478787879 0.660606061
-0.321212121 -0.284848485
0.284848485 0.515151515
0.36969697 -0.139393939
-0.284848485 0.272727273
How can I calculate the average of those correlation values in these 2 columns in Excel or Matlab ? I found a close answer in this link : https://stats.stackexchange.com/questions/8019/averaging-correlation-values
The main point is we can not use mean or average in this case, as explained in the link. They proposed a nice way to do that, but I dont know how to implement it in Excel or Matlab.
Following the second answer of the link you provided, which is the most general case, you can calculate the average Spearman's rho in Matlab as follows:
M = [0.127272727 -0.260606061;
-0.090909091 -0.224242424;
0.345454545 0.745454545;
0.478787879 0.660606061;
-0.345454545 -0.333333333;
0.151515152 -0.127272727;
0.478787879 0.660606061;
-0.321212121 -0.284848485;
0.284848485 0.515151515;
0.36969697 -0.139393939;
-0.284848485 0.272727273];
z = atanh(M);
meanRho = tanh(mean(z));
As you can see it gives mean values of
meanRho =
0.1165 0.1796
whereas the simple mean is quite close:
mean(M)
ans =
0.1085 0.1350
Edit: more information on Fisher's transformation here.
In MATLAB, define a matrix with these values and use mean function as follows:
%define a matrix M
M = [0.127272727 -0.260606061;
-0.090909091 -0.224242424;
0.345454545 0.745454545;
0.478787879 0.660606061;
-0.345454545 -0.333333333;
0.151515152 -0.127272727;
0.478787879 0.660606061;
-0.321212121 -0.284848485;
0.284848485 0.515151515;
0.36969697 -0.139393939;
-0.284848485 0.272727273];
%calculates the mean of each column
meanVals = mean(M);
Result
meanVals =
0.1085 0.1350
It is also possible to calculate the total meanm and the mean of each row as follows:
meanVals = mean(M); %total mean
meanVals = mean(M,2); %mean of each row

Resources