This is a relatively common question so I don't want to be voted down for asking something that has been asked before. I will explain as I go along the steps I took to answer this question using StackOver Flow and other sources so that you can see that I have made attempts to solve it without solving a question.
I have a set of values as below:
O P Q "R" Z
6307 586 92.07 1.34
3578 195 94.83 6.00
3147 234 93.08 4.29
3852 227 94.43 15.00
3843 171 95.74 5.10
3511 179 95.15 7.18
6446 648 90.87 1.44
4501 414 91.58 0.38
3435 212 94.19 6.23
I want to take the average of the first six values in row "R" and then put that average in the sixth column in the sixth row of Z as such:
O P Q "R" Z
6307 586 92.07 1.34
3578 195 94.83 6.00
3147 234 93.08 4.29
3852 227 94.43 15.00
3843 171 95.74 5.10
3511 179 95.15 7.18 6.49
6446 648 90.87 1.44
4501 414 91.58 0.38
3435 212 94.19 6.23
414 414 91.58 3.49
212 212 94.19 11.78
231 231 93.44 -1.59 3.6
191 191 94.59 2.68
176 176 91.45 .75
707 707 91.96 2.68
792 420 90.95 0.75
598 598 92.15 7.45
763 763 90.66 -4.02
652 652 91.01 3.75
858 445 58.43 2.30 2.30
I have utilized the following formula I obtained
=AVERAGE(OFFSET(R1510,COUNTA(R:R)-6,0,6,1))
but I received an answer that was different from what I obtained by simply taking the average of the six previous cells as such:
=AVERAGE(R1505:R1510)
I then tried the following code from a Stack OverFlow (excel averaging every 10 rows) conversation that was tangentially similar to what I wanted
=AVERAGE(INDEX(R:R,1+6*(ROW()-ROW($B$1))):INDEX(R:R,10*(ROW()- ROW($B$1)+1)))
but I was unable to get an answer that resembled what I got from taking a rote
==AVERAGE(R1517:R1522)
I also found another approach in the following but was unable to accurately change the coding (F3 to R1510, for example)
=AVERAGE(OFFSET(F3,COUNTA($R$1510:$R$1517)-1,,-6,))
Doing so gave me a negative number for a clearly positive set of data. It was -6.95.
Put this in Z1 and copy down:
=IF(MOD(ROW(),6)=0,AVERAGE(INDEX(R:R,ROW()-5):INDEX(R:R,ROW())),"")
Related
im relatively new to Dataframes in Python and running into an Issue I cant find.
im having a Dataframe with the following column layout
print(list(df.columns.values)) returns:
['iccid', 'system', 'last_updated', '01.01', '02.01', '03.01', '04.01', '05.01', '12.01', '18.01', '19.01', '20.01', '21.01', '22.01', '23.01', '24.01', '25.01', '26.01', '27.01', '28.01', '29.01', '30.01', '31.01']
normally i should have a column for each day in a specific month. in the example above its December 2022. Sometimes Days are missing which isnt an issue.
i tried to first get all given columns that are relevant by filtering them:
# Filter out the columns that are not related to the data
data_columns = [col for col in df.columns if '.' in col]
Now comes the issue:
Sometimes the column "system" could also be empty so i need to put the iccid inside the system value:
df.loc[df['system'] == 'Nicht benannt!', 'system'] = df.loc[df['system'] == 'Nicht benannt!', 'iccid'].iloc[0]
df.loc[df['system'] == '', 'system'] = df.loc[df['system'] == '', 'iccid'].iloc
grouped = df.groupby('system').sum(numeric_only=False)
then i tried to create that needed 'data_usage' column.
grouped['data_usage'] = grouped[data_columns[-1]]
grouped.reset_index(inplace=True)
By that line i should normally only get the result of the last column in the dataframe (which was a workaround that also didnt work as expected)
Now what im trying to get is the sum of all columns which contain a date in their name and add this sum to a new column named data_usage.
the issue im having here is im getting results for systems which dont have an initial system value which have a data_usage of 120000 (which is just value that represents the megabytes used) and if i check the sqlite file the system in total only used 9000 mb in that particular month.
For Example:
im having this column in the sqlite file:
iccid
system
last_updated
06.02
08.02
8931080320014183316
Nicht benannt!
2023-02-06
1196
1391
and in the dataframe i get the following result:
8931080320014183316 48129.0
I cant find the issue and would be very happy if someone can point me into the right direction.
Here are some example data as requested:
iccid
system
last_updated
01.12
02.12
03.12
04.12
05.12
06.12
07.12
08.12
09.12
10.12
11.12
12.12
13.12
14.12
15.12
16.12
17.12
18.12
19.12
20.12
21.12
22.12
23.12
28.12
29.12
30.12
31.12
8945020184547971966
U-O-51
2022-12-01
2
32
179
208
320
509
567
642
675
863
1033
1055
1174
2226
2277
2320
2466
2647
2679
2713
2759
2790
2819
2997
3023
3058
3088
8945020855461807911
L-O-382
2022-12-01
1
26
54
250
385
416
456
481
506
529
679
772
802
832
858
915
940
1019
1117
1141
1169
1193
1217
1419
1439
1461
1483
8945020855461809750
C-O-27
2022-12-01
1
123
158
189
225
251
456
489
768
800
800
800
800
800
800
2362
2386
2847
2925
2960
2997
3089
3116
3448
3469
3543
3586
8931080019070958450
L-O-123
2022-12-02
0
21
76
313
479
594
700
810
874
1181
1955
2447
2527
2640
2897
3008
3215
3412
3554
3639
3698
3782
3850
4741
4825
4925
5087
8931080453114183282
Nicht benannt!
2022-12-02
0
6
45
81
95
98
101
102
102
102
102
102
102
103
121
121
121
121
149
164
193
194
194
194
194
194
194
8931080894314183290
C-O-16 N
2022-12-02
0
43
145
252
386
452
532
862
938
1201
1552
1713
1802
1855
2822
3113
3185
3472
3527
3745
3805
3880
3938
4221
4265
4310
4373
8931080465814183308
L-O-83
2022-12-02
0
61
169
275
333
399
468
858
1094
1239
1605
1700
1928
2029
3031
4186
4333
4365
4628
4782
4842
4975
5265
5954
5954
5954
5954
8931082343214183316
Nicht benannt!
2022-12-02
0
52
182
506
602
719
948
1129
1314
1646
1912
1912
1912
1912
2791
3797
3944
4339
4510
4772
4832
5613
5688
6151
6482
6620
6848
8931087891314183324
L-O-119
2022-12-02
0
19
114
239
453
573
685
800
1247
1341
1341
1341
1341
1341
1341
1341
1341
1341
1341
1341
1341
1341
1423
2722
3563
4132
4385
I am using sns.lineplot to show the confidence intervals in a plot.
sns.lineplot(x = threshold, y = mrl_array, err_style = 'band', ci=95)
plt.show()
I'm getting the following plot, which doesn't show the confidence interval:
What's the problem?
There is probably only a single observation per x value.
If there is only one observation per x value, then there is no confidence interval to plot.
Bootstrapping is performed per x value, but there needs to be more than one obsevation for this to take effect.
ci: Size of the confidence interval to draw when aggregating with an estimator. 'sd' means to draw the standard deviation of the data. Setting to None will skip bootstrapping.
Note the following examples from seaborn.lineplot.
This is also the case for sns.relplot with kind='line'.
The question specifies sns.lineplot, but this answer applies to any seaborn plot that displays a confidence interval, such as seaborn.barplot.
Data
import seaborn as sns
# load data
flights = sns.load_dataset("flights")
year month passengers
0 1949 Jan 112
1 1949 Feb 118
2 1949 Mar 132
3 1949 Apr 129
4 1949 May 121
# only May flights
may_flights = flights.query("month == 'May'")
year month passengers
4 1949 May 121
16 1950 May 125
28 1951 May 172
40 1952 May 183
52 1953 May 229
64 1954 May 234
76 1955 May 270
88 1956 May 318
100 1957 May 355
112 1958 May 363
124 1959 May 420
136 1960 May 472
# standard deviation for each year of May data
may_flights.set_index('year')[['passengers']].std(axis=1)
year
1949 NaN
1950 NaN
1951 NaN
1952 NaN
1953 NaN
1954 NaN
1955 NaN
1956 NaN
1957 NaN
1958 NaN
1959 NaN
1960 NaN
dtype: float64
# flight in wide format
flights_wide = flights.pivot("year", "month", "passengers")
month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
year
1949 112 118 132 129 121 135 148 148 136 119 104 118
1950 115 126 141 135 125 149 170 170 158 133 114 140
1951 145 150 178 163 172 178 199 199 184 162 146 166
1952 171 180 193 181 183 218 230 242 209 191 172 194
1953 196 196 236 235 229 243 264 272 237 211 180 201
1954 204 188 235 227 234 264 302 293 259 229 203 229
1955 242 233 267 269 270 315 364 347 312 274 237 278
1956 284 277 317 313 318 374 413 405 355 306 271 306
1957 315 301 356 348 355 422 465 467 404 347 305 336
1958 340 318 362 348 363 435 491 505 404 359 310 337
1959 360 342 406 396 420 472 548 559 463 407 362 405
1960 417 391 419 461 472 535 622 606 508 461 390 432
# standard deviation for each year
flights_wide.std(axis=1)
year
1949 13.720147
1950 19.070841
1951 18.438267
1952 22.966379
1953 28.466887
1954 34.924486
1955 42.140458
1956 47.861780
1957 57.890898
1958 64.530472
1959 69.830097
1960 77.737125
dtype: float64
Plots
may_flights has one observation per year, so no CI is shown.
sns.lineplot(data=may_flights, x="year", y="passengers")
sns.barplot(data=may_flights, x='year', y='passengers')
flights_wide shows there are twelve observations for each year, so the CI shows when all of flights is plotted.
sns.lineplot(data=flights, x="year", y="passengers")
sns.barplot(data=flights, x='year', y='passengers')
I have a df like this,
ID Machine 17-Dec 18-Jan 18-Feb 18-Mar 18-Apr 18-May
160 Car 348 280 274 265 180 224
163 Var 68248 72013 55441 64505 71097 78006
165 Assus 1337 1279 1536 1461 1555 1700
215 Owen 118 147 104 143 115 153
I calculates the Mean and Std. Dev like this,
df['Avg'] = np.mean(all_np_values, axis=1)
df['Std.Dev'] = np.std(all_np_values, axis=1)
Then I get the following data frame.
ID Machine 17-Dec 18-Jan 18-Feb 18-Mar 18-Apr 18-May Mean Std.Dev
160 Car 348 280 274 265 180 224 261.83 51.70
163 Var 68248 72013 55441 64505 71097 78006 68218.33 7018.24
165 Assus 1337 1279 1536 1461 1555 1700 1478 140.44
215 Owen 118 147 104 143 115 153 130 18.40
Now, I want to have a final dataframe that looks like below, which I would like to look at MAY 18 and say yes or no based on its value Above or Below 2 standard deviation.
ID Machine 17-Dec 18-Jan 18-Feb 18-Mar 18-Apr 18-May Mean Std.Dev Above Below
160 Car 348 280 274 265 180 224 261.83 51.70 No No
163 Var 68248 72013 55441 64505 71097 78006 68218.33 7018.24 No No
165 Assus 1337 1279 1536 1461 1555 1700 1478 140.44 No No
215 Owen 118 147 104 143 115 153 130 18.40 No No
I tried to do the following,
for value in df['18-May']:
if value > (df['Avg'] + 2 * df['Std.Dev']):
df['Above'] = 'Yes'
else:
df['Above'] = 'No'
This gives me an error:
ValueError: The truth value of a Series is ambiguous. Use a.empty,
a.bool(), a.item(), a.any() or a.all().
I understand the error after reading some older posts. My conclusion is, it returns bool values for comparison.
Not sure, how to mask in creating a new df column to create that 'Yes' and 'No' in my 'Above' or 'Below' column. How can I add that into my code above?
Any thoughts would be helpful.
I thought I understood Conditional Formatting, but I'm having trouble applying it.
I have a table like this:
J F M A M J J A S O N D
317 876 781 735 490 790 569 922 976 923 259 270
440 734 788 729 337 876 641 735 892 906 463 872
773 863 537 538 389 988 404 599 459 976 235 891
772 134 845 218 669 414 352 276 965 677 723 342
343 283 251 703 420 430 355 485 537 723 838 199
354 846 503 448 196 249 237 503 309 291 432 807
259 126 213 254 577 688 195 681 658 407 637 827
293 333 562 242 504 335 276 675 173 592 388 418
And I'd just like to highlight, via Green Fill, the Max() in each row.
I highlighted my range, keeping the top left (317) as my active cell. Then I tried this formula:
=max($A2:$L2), but this just highlights my entire table:
If I highlight row 2, use =Max($A2:$L2), then the whole row gets highlighted...I've even tried using the Top 10 default rule, changing to top 1, but no avail.
What I can do, is use the Top 1, on a single row, then paste that formatting down. However, I'd like to get a single formula, if possible, so I can avoid having hundreds of CF formats in the "CF rules box".
Thanks for any ideas. I'm sure it is a simple fix, I'm just overthinking.
I think the formula you're looking for is:
=A2=MAX($A2:$L2)
This compares the value of the cell to the max value of the row. When equal, the formula evaluates to TRUE.
Here's an answer to the question asked in the comments, i.e., what's going on with evaluating MAX to TRUE or FALSE in conditional formatting.
In Excel zero evaluates to FALSE and all other numbers to TRUE (or something like that). So for instance =NOT(NOT(0)) evaluates to FALSE.
When you entered your MAX formula in conditional formatting it will evaluate to TRUE unless the MAX formula result is zero, in which case it results in FALSE:
I'm currently generating an extremely large data set on a remote HPC (high performace computer). We are talking about 3 TB at the moment, and it could reach up to 10 TB once I'm done.
Each of the 450 000 files ranges from a few KB to about 100 MB and contains lines of integers with no repetitive/predictable patterns. Moreover they are split among 150 folders (I use the path to classify them according to the input parameters). Now that could be fine, but my research group is technically limited to 1TB of disk space on the remote server, although the admin are willing to close their eyes until the situation gets sorted out.
What would you recommend to compress such a dataset?
A limitation is that tasks can't run more than 48 hours at a time on this computer. So long but efficient compression methods are possible only if 48 hours is enough... I really have no other options as neither me, neither my group own enough disk space on other machines.
EDIT: Just to clarify, this a remote computer that runs on some variation of linux. All standard compression protocols are available. I don't have super user rights.
EDIT2: As request by Sergio, here is a sample output (first 10 lines of a files)
27 42 46 63 95 110 205 227 230 288 330 345 364 367 373 390 448 471 472 482 509 514 531 533 553 617 636 648 667 682 703 704 735 740 762 775 803 813 882 915 920 936 939 942 943 979 1018 1048 1065 1198 1219 1228 1513 1725 1888 1944 2085 2190 2480 5371 5510 5899 6788 7728 9514 10382 11946 13063 13808 16070 23301 23511 24538
93 94 106 143 157 164 168 181 196 293 299 334 369 372 439 457 508 527 547 557 568 570 573 592 601 668 701 704 799 838 848 870 875 882 890 913 953 959 1022 1024 1037 1046 1169 1201 1288 1615 1684 1771 2043 2204 2348 2387 2735 3149 4319 4890 4989 5321 5588 6453 7475 9277 9649 9654 11433 16966
1463
183 469 514 597 792
25 50 143 152 205 244 253 424 433 446 461 476 486 545 552 570 632 642 647 665 681 682 718 735 746 772 792 811 830 851 891 903 925 1037 1115 1147 1171 1612 1979 2749 3074 3158 6042 12709 20571 20859
24 30 86 312 726 875 1023 1683 1799
33 36 42 65 110 112 122 227 241 262 274 284 305 328 353 366 393 414 419 449 462 488 489 514 635 690 732 744 767 772 812 820 843 844 855 889 893 925 936 939 981 1015 1020 1060 1064 1130 1174 1304 1393 1477 1939 2004 2200 2205 2208 2216 2234 3284 4456 5209 6810 6834 8067 10811 10895 12771 15291
157 761 834 875 1001 2492
21 141 146 169 181 256 266 337 343 367 397 402 405 433 454 466 513 527 656 684 708 709 732 743 811 883 913 938 947 986 987 1013 1053 1190 1215 1288 1289 1333 1513 1524 1683 1758 2033 2684 3714 4129 6015 7395 8273 8348 9483 23630
1253
All integers are separated by one whitespace, and each line corresponds to a given element. I use implicit line numbers to store this information, because my data is assosiative i.e. the 0th element is associated to elements 27 42 46 63 110.. etc. I believe that there is no extra information whatsoever.
A few points that may help:
It looks like your numbers are sorted. If this is always the case, then it will be more efficient to compress the differences between adjacent numbers rather than the numbers themselves (since the differences will be somewhat smaller on average)
There are good ways of encoding small integer values in binary format, that are probably better than encoding them in text format. See the technique used by Google in their protocol buffers: (https://developers.google.com/protocol-buffers/docs/encoding)
Once you have applied the above techniques, then zipping / some standard form of compression should improve everything even further.
There is some research done at this LINK that breaks down the pro/cons of using gzip, bzip2, and lzma. Hopefully this can let you make an informed decision on your best approach.
All your numbers seem to be increasing in size (each line). A rather common approach in database technology would be to only store the size difference, making a line like
24 30 86 312 726 875 1023 1683 1799
to something like
6 56 226 414 149 148 660 116
Other lines of your example would even show more benefit, as the differences are smaller. This also works when the numbers decrease in-between, but you have to be able to deal with negative differences then.
Second thing to do would be changing the encoding. While compression will reduce this overhead, you're currently using 8 bit per digit, whereas you only need 4 bit of those (0-9, space as divisor). Implementing your own "4 bit character set" will already cut your storage requirements to half of the current size! In the end, this would be some kind of binary encoding of numbers of arbitrary length.