I am sorry if this question doesn't make too much sense, but I am trying to make a post hoc analysis with my gam selected models. It's normal that my contrast doesn't have any significant p-value?
I was expecting to find which level of my factor was selected in the interaction.
OBS: I have a data set of 169 samples and I am trying to understand which environmental feature (depth, chlorophyll, bottom type, and SST)is more relevant for the CPUE (catch per unit effort). My only categorical variable is the Bottom Type with 13 levels.
dredge(res)
Global model call: gam(formula = nCPUE ~ s(Depth) + s(Chlorophyll) + BottomType +
s(SST), data = morio)
---
Model selection table
(Int) BtT s(Chl) s(Dpt) s(SST) df logLik AICc delta weight
4 -0.62310 + 1.597 15 -304.318 641.8 0.00 0.821
10 18.10000 + -0.70940 15 -306.983 647.1 5.33 0.057
3 -0.16020 1.939 3 -320.855 647.9 6.08 0.039
8 -0.61760 + 1.479 0.0001209 16 -306.497 648.6 6.80 0.027
2 -0.29430 + 14 -308.959 648.6 6.87 0.026
12 4.10300 + 1.373 -0.18050 16 -306.995 649.6 7.80 0.017
14 17.05000 + 0.0001351 -0.67000 16 -308.616 652.8 11.04 0.003
6 -0.33730 + 0.0002019 15 -310.039 653.2 11.44 0.003
7 -0.08551 1.753 0.0002077 4 -322.626 653.5 11.72 0.002
11 1.19800 1.936 -0.05251 4 -322.824 653.9 12.12 0.002
16 5.40200 + 1.168 0.0001235 -0.22980 17 -308.923 655.9 14.13 0.001
1 0.34150 2 -326.765 657.6 15.83 0.000
9 17.61000 -0.66810 3 -325.910 658.0 16.19 0.000
15 2.63300 1.689 0.0001938 -0.10460 5 -324.395 659.2 17.38 0.000
5 0.37710 0.0002781 3 -326.857 659.9 18.09 0.000
13 15.94000 0.0002114 -0.60260 4 -326.783 661.8 20.04 0.000
Models ranked by AICc(x)
post.hoc = emmeans(res, specs = ~ BottomType|Chlorophyll)
moriotukey<-contrast(post.hoc, method = "pairwise")
summary(moriotukey)
contrast estimate SE df z.ratio p.value
CA 0.258698224852071 - (CA+RB 0.258698224852071) -2.3155 1.101 Inf -2.103 0.6619
CA 0.258698224852071 - (CA+RB+RE 0.258698224852071) -0.9394 1.729 Inf -0.543 1.0000
CA 0.258698224852071 - (CA+RE 0.258698224852071) -0.1050 1.966 Inf -0.053 1.0000
CA 0.258698224852071 - (CA+RE+MI+RB+SS 0.258698224852071) -3.9677 1.742 Inf -2.277 0.5338
CA 0.258698224852071 - (CA+RE+MI+RB+SS+TE 0.258698224852071) -4.2754 1.772 Inf -2.413 0.4346
CA 0.258698224852071 - MI 0.258698224852071 -0.4388 1.000 Inf -0.439 1.0000
CA 0.258698224852071 - (MI+RE 0.258698224852071) 0.0124 1.731 Inf 0.007 1.0000
CA 0.258698224852071 - RB 0.258698224852071 -0.6403 0.798 Inf -0.803 0.9999
CA 0.258698224852071 - (RB+RE 0.258698224852071) -1.9403 1.200 Inf -1.616 0.9253
CA 0.258698224852071 - (RB+SS+RE 0.258698224852071) -4.0034 1.804 Inf -2.220 0.5767
CA 0.258698224852071 - RE 0.258698224852071 -0.2632 0.795 Inf -0.331 1.0000
CA 0.258698224852071 - SS 0.258698224852071 -3.4633 1.341 Inf -2.583 0.3205
(CA+RB 0.258698224852071) - (CA+RB+RE 0.258698224852071) 1.3761 1.734 Inf 0.794 0.9999
(CA+RB 0.258698224852071) - (CA+RE 0.258698224852071) 2.2105 1.908 Inf 1.158 0.9950
(CA+RB 0.258698224852071) - (CA+RE+MI+RB+SS 0.258698224852071) -1.6521 1.731 Inf -0.954 0.9992
(CA+RB 0.258698224852071) - (CA+RE+MI+RB+SS+TE 0.258698224852071) -1.9599 1.748 Inf -1.121 0.9963
(CA+RB 0.258698224852071) - MI 0.258698224852071 1.8767 1.011 Inf 1.856 0.8210
(CA+RB 0.258698224852071) - (MI+RE 0.258698224852071) 2.3279 1.733 Inf 1.344 0.9818
(CA+RB 0.258698224852071) - RB 0.258698224852071 1.6753 0.809 Inf 2.070 0.6857
(CA+RB 0.258698224852071) - (RB+RE 0.258698224852071) 0.3752 1.184 Inf 0.317 1.0000
(CA+RB 0.258698224852071) - (RB+SS+RE 0.258698224852071) -1.6879 1.771 Inf -0.953 0.9992
(CA+RB 0.258698224852071) - RE 0.258698224852071 2.0523 0.797 Inf 2.574 0.3260
(CA+RB 0.258698224852071) - SS 0.258698224852071 -1.1478 1.353 Inf -0.848 0.9998
(CA+RB+RE 0.258698224852071) - (CA+RE 0.258698224852071) 0.8344 2.381 Inf 0.351 1.0000
(CA+RB+RE 0.258698224852071) - (CA+RE+MI+RB+SS 0.258698224852071) -3.0282 2.198 Inf -1.378 0.9776
(CA+RB+RE 0.258698224852071) - (CA+RE+MI+RB+SS+TE 0.258698224852071) -3.3359 2.222 Inf -1.502 0.9562
(CA+RB+RE 0.258698224852071) - MI 0.258698224852071 0.5007 1.671 Inf 0.300 1.0000
(CA+RB+RE 0.258698224852071) - (MI+RE 0.258698224852071) 0.9518 2.189 Inf 0.435 1.0000
(CA+RB+RE 0.258698224852071) - RB 0.258698224852071 0.2992 1.558 Inf 0.192 1.0000
(CA+RB+RE 0.258698224852071) - (RB+RE 0.258698224852071) -1.0009 1.799 Inf -0.556 1.0000
(CA+RB+RE 0.258698224852071) - (RB+SS+RE 0.258698224852071) -3.0639 2.247 Inf -1.363 0.9795
(CA+RB+RE 0.258698224852071) - RE 0.258698224852071 0.6763 1.557 Inf 0.434 1.0000
(CA+RB+RE 0.258698224852071) - SS 0.258698224852071 -2.5239 1.895 Inf -1.332 0.9831
(CA+RE 0.258698224852071) - (CA+RE+MI+RB+SS 0.258698224852071) -3.8626 2.302 Inf -1.678 0.9039
(CA+RE 0.258698224852071) - (CA+RE+MI+RB+SS+TE 0.258698224852071) -4.1704 2.254 Inf -1.850 0.8242
(CA+RE 0.258698224852071) - MI 0.258698224852071 -0.3338 1.929 Inf -0.173 1.0000
(CA+RE 0.258698224852071) - (MI+RE 0.258698224852071) 0.1174 2.363 Inf 0.050 1.0000
(CA+RE 0.258698224852071) - RB 0.258698224852071 -0.5352 1.824 Inf -0.293 1.0000
(CA+RE 0.258698224852071) - (RB+RE 0.258698224852071) -1.8353 1.927 Inf -0.953 0.9992
(CA+RE 0.258698224852071) - (RB+SS+RE 0.258698224852071) -3.8984 2.227 Inf -1.750 0.8737
(CA+RE 0.258698224852071) - RE 0.258698224852071 -0.1582 1.793 Inf -0.088 1.0000
(CA+RE 0.258698224852071) - SS 0.258698224852071 -3.3583 2.143 Inf -1.567 0.9400
(CA+RE+MI+RB+SS 0.258698224852071) - (CA+RE+MI+RB+SS+TE 0.258698224852071) -0.3077 2.193 Inf -0.140 1.0000
(CA+RE+MI+RB+SS 0.258698224852071) - MI 0.258698224852071 3.5289 1.689 Inf 2.089 0.6718
(CA+RE+MI+RB+SS 0.258698224852071) - (MI+RE 0.258698224852071) 3.9801 2.195 Inf 1.813 0.8438
(CA+RE+MI+RB+SS 0.258698224852071) - RB 0.258698224852071 3.3274 1.575 Inf 2.113 0.6551
(CA+RE+MI+RB+SS 0.258698224852071) - (RB+RE 0.258698224852071) 2.0273 1.786 Inf 1.135 0.9959
(CA+RE+MI+RB+SS 0.258698224852071) - (RB+SS+RE 0.258698224852071) -0.0357 2.207 Inf -0.016 1.0000
(CA+RE+MI+RB+SS 0.258698224852071) - RE 0.258698224852071 3.7045 1.566 Inf 2.366 0.4682
(CA+RE+MI+RB+SS 0.258698224852071) - SS 0.258698224852071 0.5043 1.915 Inf 0.263 1.0000
(CA+RE+MI+RB+SS+TE 0.258698224852071) - MI 0.258698224852071 3.8366 1.722 Inf 2.228 0.5706
(CA+RE+MI+RB+SS+TE 0.258698224852071) - (MI+RE 0.258698224852071) 4.2878 2.215 Inf 1.936 0.7743
(CA+RE+MI+RB+SS+TE 0.258698224852071) - RB 0.258698224852071 3.6351 1.609 Inf 2.259 0.5472
(CA+RE+MI+RB+SS+TE 0.258698224852071) - (RB+RE 0.258698224852071) 2.3351 1.794 Inf 1.301 0.9861
(CA+RE+MI+RB+SS+TE 0.258698224852071) - (RB+SS+RE 0.258698224852071) 0.2720 2.190 Inf 0.124 1.0000
(CA+RE+MI+RB+SS+TE 0.258698224852071) - RE 0.258698224852071 4.0122 1.594 Inf 2.517 0.3626
(CA+RE+MI+RB+SS+TE 0.258698224852071) - SS 0.258698224852071 0.8121 1.948 Inf 0.417 1.0000
MI 0.258698224852071 - (MI+RE 0.258698224852071) 0.4512 1.674 Inf 0.269 1.0000
MI 0.258698224852071 - RB 0.258698224852071 -0.2015 0.662 Inf -0.304 1.0000
MI 0.258698224852071 - (RB+RE 0.258698224852071) -1.5015 1.121 Inf -1.339 0.9823
MI 0.258698224852071 - (RB+SS+RE 0.258698224852071) -3.5646 1.757 Inf -2.029 0.7138
MI 0.258698224852071 - RE 0.258698224852071 0.1756 0.660 Inf 0.266 1.0000
MI 0.258698224852071 - SS 0.258698224852071 -3.0245 1.264 Inf -2.392 0.4494
(MI+RE 0.258698224852071) - RB 0.258698224852071 -0.6527 1.563 Inf -0.418 1.0000
(MI+RE 0.258698224852071) - (RB+RE 0.258698224852071) -1.9527 1.794 Inf -1.088 0.9972
(MI+RE 0.258698224852071) - (RB+SS+RE 0.258698224852071) -4.0158 2.238 Inf -1.795 0.8529
(MI+RE 0.258698224852071) - RE 0.258698224852071 -0.2756 1.560 Inf -0.177 1.0000
(MI+RE 0.258698224852071) - SS 0.258698224852071 -3.4757 1.899 Inf -1.830 0.8348
RB 0.258698224852071 - (RB+RE 0.258698224852071) -1.3001 0.942 Inf -1.380 0.9774
RB 0.258698224852071 - (RB+SS+RE 0.258698224852071) -3.3631 1.645 Inf -2.044 0.7032
RB 0.258698224852071 - RE 0.258698224852071 0.3771 0.266 Inf 1.418 0.9719
RB 0.258698224852071 - SS 0.258698224852071 -2.8231 1.112 Inf -2.539 0.3484
(RB+RE 0.258698224852071) - (RB+SS+RE 0.258698224852071) -2.0631 1.811 Inf -1.139 0.9957
(RB+RE 0.258698224852071) - RE 0.258698224852071 1.6771 0.927 Inf 1.809 0.8455
(RB+RE 0.258698224852071) - SS 0.258698224852071 -1.5230 1.440 Inf -1.058 0.9979
(RB+SS+RE 0.258698224852071) - RE 0.258698224852071 3.7402 1.626 Inf 2.301 0.5163
(RB+SS+RE 0.258698224852071) - SS 0.258698224852071 0.5401 1.981 Inf 0.273 1.0000
RE 0.258698224852071 - SS 0.258698224852071 -3.2001 1.113 Inf -2.875 0.1690
P value adjustment: tukey method for comparing a family of 13 estimates
plot(post.hoc)
Related
This question already has answers here:
What function defines accuracy in Keras when the loss is mean squared error (MSE)?
(3 answers)
Closed 7 months ago.
import keras
import pandas as pd
import numpy as np
from google.colab import files
uploaded = files.upload()
import io
dataset = pd.read_csv(io.BytesIO(uploaded['kc_house_data.csv']))
dataset.head()
id date price bedrooms bathrooms sqft_living sqft_lot floors waterfront view ... grade sqft_above sqft_basement yr_built yr_renovated zipcode lat long sqft_living15 sqft_lot15
0 7129300520 20141013T000000 221900.0 3 1.00 1180 5650 1.0 0 0 ... 7 1180 0 1955 0 98178 47.5112 -122.257 1340 5650
1 6414100192 20141209T000000 538000.0 3 2.25 2570 7242 2.0 0 0 ... 7 2170 400 1951 1991 98125 47.7210 -122.319 1690 7639
2 5631500400 20150225T000000 180000.0 2 1.00 770 10000 1.0 0 0 ... 6 770 0 1933 0 98028 47.7379 -122.233 2720 8062
3 2487200875 20141209T000000 604000.0 4 3.00 1960 5000 1.0 0 0 ... 7 1050 910 1965 0 98136 47.5208 -122.393 1360 5000
4 1954400510 20150218T000000 510000.0 3 2.00 1680 8080 1.0 0 0 ... 8 1680 0 1987 0 98074 47.6168 -122.045 1800 7503
5 rows × 21 columns
features = dataset.drop(columns=['price', 'id', 'date'])
labels = dataset[['price']]
model = keras.models.Sequential([
keras.layers.Dense(19, 'relu', input_shape=(18,)),
keras.layers.Dense(19, 'relu'),
keras.layers.Dense(1)
])
from keras import metrics
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])
model.fit(features, labels, epochs=10, batch_size=5)
Epoch 1/10
676/676 [==============================] - 2s 3ms/step - loss: 65482420224.0000 - accuracy: 0.0000e+00
Epoch 2/10
676/676 [==============================] - 2s 3ms/step - loss: 65566482432.0000 - accuracy: 0.0000e+00
Epoch 3/10
676/676 [==============================] - 2s 3ms/step - loss: 65582268416.0000 - accuracy: 0.0000e+00
Epoch 4/10
676/676 [==============================] - 2s 3ms/step - loss: 65601855488.0000 - accuracy: 0.0000e+00
Epoch 5/10
676/676 [==============================] - 2s 3ms/step - loss: 65537380352.0000 - accuracy: 0.0000e+00
Epoch 6/10
676/676 [==============================] - 2s 2ms/step - loss: 65665077248.0000 - accuracy: 0.0000e+00
Epoch 7/10
676/676 [==============================] - 2s 2ms/step - loss: 65604460544.0000 - accuracy: 0.0000e+00
Epoch 8/10
676/676 [==============================] - 2s 2ms/step - loss: 65511895040.0000 - accuracy: 0.0000e+00
Epoch 9/10
676/676 [==============================] - 2s 3ms/step - loss: 65589620736.0000 - accuracy: 0.0000e+00
Epoch 10/10
676/676 [==============================] - 2s 3ms/step - loss: 65584775168.0000 - accuracy: 0.0000e+00
<keras.callbacks.History at 0x7f3dc673af90>
This is a regression problem, not a classification one.
You can't use accuracy as a metric for a regression problem.
Change metrics to 'mean_squared_error' in the model.compile method.
something like this:-
model.compile(loss= "mean_squared_error" , optimizer="adam", metrics=["mean_squared_error"])
I have multiple text files which contain some tables , most of the tables are of these two types. I want a way to convert these tables into python dictionaries.
precision recall f1-score support
BASE_RENT_ANNUAL 0.53 0.57 0.55 1408
BASE_RENT_MONTHLY 0.65 0.54 0.59 3904
BASE_RENT_PSF 0.68 0.59 0.63 1248
RENT_INCREMENT_MONTHLY 0.63 0.44 0.52 7530
SECURITY_DEPOSIT_AMOUNT 0.88 0.89 0.88 3557
micro avg 0.69 0.58 0.63 17647
macro avg 0.67 0.61 0.63 17647
weighted avg 0.68 0.58 0.62 17647
Hard Evaluation Metrics
--------------------------------------------------
Reading predictions from /mnt/c/Users/Aleksandra/mlbuddy/python/bilstm/training/test_predictions.txt...
Nb tokens in test set: 957800
Reading training data from /mnt/c/Users/Aleksandra/mlbuddy/python/bilstm/corpus/train.txt...
Nb tokens in training set: 211153
Strict mode: OFF
---------------------------------------------------------------------
Test tokens Nb tokens Nb words Nb errors Token error rate
---------------------------------------------------------------------
all 957800 5408 39333 0.0411
---------------------------------------------------------------------
unseen-I 704 19 704 1.0000
unseen-O 59870 1724 10208 0.1705
unseen-all 60574 1743 10912 0.1801
---------------------------------------------------------------------
diff-I 13952 70 13952 1.0000
diff-O 5285 121 4645 0.8789
diff-etype 0 0 0 0.0000
diff-all 19237 191 18597 0.9667
---------------------------------------------------------------------
all-unseen+diff 79811 1934 29509 0.3697
---------------------------------------------------------------------
Avg TER on unseen and diff: 0.5734
I have tried this in my code to convert the second table to dictionary but it is not working as expected.:
from itertools import dropwhile, takewhile
with open("idm.txt") as f:
dp = dropwhile(lambda x: not x.startswith("-"), f)
next(dp) # skip ----
names = next(dp).split() # get headers names
next(f) # skip -----
out = []
for line in takewhile(lambda x: not x.startswith("-"), f):
a, b = line.rsplit(None, 1)
out.append(dict(zip(names, a.split(None, 7) + [b])))
Expected output:
{BASE_RENT_ANNUAL: {precision:0.53,recall:0.57,f1-score:0.55,support:1408},
BASE_RENT_MONTHLY: {...}, ..
}
Not the same approach, but the following could be a beginning to your true solution
txt = ''' precision recall f1-score support
BASE_RENT_ANNUAL 0.53 0.57 0.55 1408
BASE_RENT_MONTHLY 0.65 0.54 0.59 3904
BASE_RENT_PSF 0.68 0.59 0.63 1248
RENT_INCREMENT_MONTHLY 0.63 0.44 0.52 7530
SECURITY_DEPOSIT_AMOUNT 0.88 0.89 0.88 3557
micro avg 0.69 0.58 0.63 17647
macro avg 0.67 0.61 0.63 17647
weighted avg 0.68 0.58 0.62 17647
Hard Evaluation Metrics
--------------------------------------------------
Reading predictions from /mnt/c/Users/Aleksandra/mlbuddy/python/bilstm/training/test_predictions.txt...
Nb tokens in test set: 957800
Reading training data from /mnt/c/Users/Aleksandra/mlbuddy/python/bilstm/corpus/train.txt...
Nb tokens in training set: 211153
Strict mode: OFF
---------------------------------------------------------------------
Test tokens Nb tokens Nb words Nb errors Token error rate
---------------------------------------------------------------------
all 957800 5408 39333 0.0411
---------------------------------------------------------------------
unseen-I 704 19 704 1.0000
unseen-O 59870 1724 10208 0.1705
unseen-all 60574 1743 10912 0.1801
---------------------------------------------------------------------
diff-I 13952 70 13952 1.0000
diff-O 5285 121 4645 0.8789
diff-etype 0 0 0 0.0000
diff-all 19237 191 18597 0.9667
---------------------------------------------------------------------
all-unseen+diff 79811 1934 29509 0.3697
---------------------------------------------------------------------
Avg TER on unseen and diff: 0.5734'''
lst1 = [x.split() for x in txt.split('\n') if x]
lst2 = [(x[0],x[1:]) for x in lst1 if (not x[0].startswith('-') and x[0] == x[0].upper())]
dico = dict(lst2)
dico2 = {}
for k in dico:
dico2[k] = {'precision':dico[k][0],'recall':dico[k][1],'f1-score':dico[k][2],'support':dico[k][3]}
print(dico2)
I have a dataframe with daily returns. I want to add another column with price series calculated based on daily returns.
This is the dataframe:
date daily
0 2020-09-01 0.0000
1 2020-09-02 0.0012
2 2020-09-03 -0.0005
3 2020-09-04 -0.0004
4 2020-09-07 0.0032
5 2020-09-08 -0.0015
6 2020-09-09 0.0005
7 2020-09-10 0.0003
8 2020-09-11 0.0001
9 2020-09-14 0.0043
10 2020-09-15 0.0037
11 2020-09-16 -0.0008
and this is the column of prices that i want to add:
prices
0 100.000000
1 100.120000
2 100.069940
3 100.029912
4 100.350008
5 100.199483
6 100.249582
7 100.279657
8 100.289685
9 100.720931
10 101.093598
11 101.012724
I've tried to do a loop on column 'daily' and then calculate the price but i do not preserve the new values into the list prz.
prz= []
for row in df['daily']:
prz.append(100 *(1+row))
First add 1, then use Series.cumprod and last multiply by 100, for invert use Series.pct_change with replace first NaN to 0:
df['prices'] = df['daily'].add(1).cumprod().mul(100)
df['back'] = df['prices'].pct_change().fillna(0)
print (df)
date daily prices back
0 2020-09-01 0.0000 100.000000 0.0000
1 2020-09-02 0.0012 100.120000 0.0012
2 2020-09-03 -0.0005 100.069940 -0.0005
3 2020-09-04 -0.0004 100.029912 -0.0004
4 2020-09-07 0.0032 100.350008 0.0032
5 2020-09-08 -0.0015 100.199483 -0.0015
6 2020-09-09 0.0005 100.249582 0.0005
7 2020-09-10 0.0003 100.279657 0.0003
8 2020-09-11 0.0001 100.289685 0.0001
9 2020-09-14 0.0043 100.720931 0.0043
10 2020-09-15 0.0037 101.093598 0.0037
11 2020-09-16 -0.0008 101.012724 -0.0008
You can also use numpy.cumproduct() here:
In [1349]: import numpy as np
In [1350]: df['prices'] = np.cumproduct(df.daily + 1) * 100
In [1351]: df
Out[1351]:
date daily prices
0 2020-09-01 0.0000 100.000000
1 2020-09-02 0.0012 100.120000
2 2020-09-03 -0.0005 100.069940
3 2020-09-04 -0.0004 100.029912
4 2020-09-07 0.0032 100.350008
5 2020-09-08 -0.0015 100.199483
6 2020-09-09 0.0005 100.249582
7 2020-09-10 0.0003 100.279657
8 2020-09-11 0.0001 100.289685
9 2020-09-14 0.0043 100.720931
10 2020-09-15 0.0037 101.093598
11 2020-09-16 -0.0008 101.012724
I am using the following Gnuplot script with Gnuplot version 4.6 patchlevel 5 :
##### Prologue #####
clear # erases the current screen or output device
reset # all graph-related options take on their default values
###### Plot options #####
set style data lines
set surface
set dgrid3d 64,64 qnorm 2
set hidden3d
set ticslevel 0.8
set isosample 40,40
set view 60, 30, 1, 1
set contour base
###### Plot data #####
stats 'modele.out'
do for [i=0:int(STATS_blocks-1)] {
set key title 'density at t'.i
splot 'modele.out' every :::i::i using 2:3:5 notitle
pause 0.5
}
and the following file :
# Time is 0.000000000000000E+000
0.0000 0.0000 0.0000 0.9787 0.0213
0.0000 0.0000 0.3333 0.9077 0.0923
0.0000 0.0000 0.6667 0.9738 0.0262
0.0000 0.0000 1.0000 0.9673 0.0327
0.0000 0.3333 0.0000 0.9044 0.0956
0.0000 0.3333 0.3333 0.9246 0.0754
0.0000 0.3333 0.6667 0.9181 0.0819
0.0000 0.3333 1.0000 0.9089 0.0911
0.0000 0.6667 0.0000 0.9348 0.0652
0.0000 0.6667 0.3333 0.9372 0.0628
0.0000 0.6667 0.6667 0.9933 0.0067
0.0000 0.6667 1.0000 0.9273 0.0727
0.0000 1.0000 0.0000 0.9909 0.0091
0.0000 1.0000 0.3333 0.9771 0.0229
0.0000 1.0000 0.6667 0.9014 0.0986
0.0000 1.0000 1.0000 0.9312 0.0688
# Time is 50.0000000000000
50.0000 0.0000 0.0000 0.1036 0.8370
50.0000 0.0000 0.3333 0.1036 0.9093
50.0000 0.0000 0.6667 0.1031 0.9368
50.0000 0.0000 1.0000 0.1042 0.8378
50.0000 0.3333 0.0000 0.1034 0.9556
50.0000 0.3333 0.3333 0.1039 0.9127
50.0000 0.3333 0.6667 0.1041 0.9761
50.0000 0.3333 1.0000 0.1041 0.9587
50.0000 0.6667 0.0000 0.1033 0.9432
50.0000 0.6667 0.3333 0.1043 0.9503
50.0000 0.6667 0.6667 0.1087 0.5931
50.0000 0.6667 1.0000 0.1057 0.9579
50.0000 1.0000 0.0000 0.1044 0.8390
50.0000 1.0000 0.3333 0.1046 0.9101
50.0000 1.0000 0.6667 0.1062 0.9597
50.0000 1.0000 1.0000 0.1063 0.8494
# Time is 100.000000000000
100.0000 0.0000 0.0000 0.0997 0.8433
100.0000 0.0000 0.3333 0.0998 0.9123
100.0000 0.0000 0.6667 0.0995 0.9501
100.0000 0.0000 1.0000 0.0999 0.8442
100.0000 0.3333 0.0000 0.0999 0.9593
100.0000 0.3333 0.3333 0.1000 0.9157
100.0000 0.3333 0.6667 0.1000 0.9794
100.0000 0.3333 1.0000 0.1002 0.9612
100.0000 0.6667 0.0000 0.0997 0.9534
100.0000 0.6667 0.3333 0.1000 0.9542
100.0000 0.6667 0.6667 0.1001 0.6028
100.0000 0.6667 1.0000 0.1004 0.9584
100.0000 1.0000 0.0000 0.1000 0.8448
100.0000 1.0000 0.3333 0.1002 0.9143
100.0000 1.0000 0.6667 0.1005 0.9571
100.0000 1.0000 1.0000 0.1006 0.8490
I don't understand why the stats command returns that I have only 1 data block. That should be 3 in my opinion. Is the file badly formatted ?
stats gives you the number of indexable blocks in your data file. These blocks are separated by pairs of blank records (i.e. two blank lines).
If you did plot 'modele.out' index 0 you would find that it plotted all your data points as well, whereas index 1 would give you an error. There is only one (indexable) block in your data.
The solution
separate your blocks by two blank lines
change your splot command to splot 'modele.out' index i using 2:3:5 notitle
When you are using splot, a single blank line separates each row (or datablock, to use the term in the manual). This isn't the same thing as a block! In all other contexts (as far as I'm aware) there are two lines between each block (or indexable block to use the term in the manual).
update
As suggested by Christoph in the comments, if you wanted to keep your file in the same format and were sure that there were no blank lines at the end, you could change your loop to this:
do for [i=0:STATS_blank] {
and use your original splot line (with every, rather than index).
ID RT EZ Z0 Z1 Z2 RHO PHE
1889 UN NA 1.0000 0.0000 0.0000 0.8765 -1
1890 UN NA 1.0000 0.0000 0.0000 0.4567 -1
1891 UN NA 1.0000 0.0000 0.0000 0.0012 -1
1892 UN NA 1.0000 0.0000 0.0000 0.1011 -1
I would like to grep all the IDs that have column 'RHO' with value less than 0.2, and the other columns are included for the selected rows.
Use awk directly by saying awk '$field < value':
$ awk '$7<0.2' file
1891 UN NA 1.0000 0.0000 0.0000 0.0012 -1
1892 UN NA 1.0000 0.0000 0.0000 0.1011 -1
As RHO is the column 7, it checks that field.
In case you just want to print a specific column, say awk '$field < value {print $another_field}'. For the ID:
$ awk '$7<0.2 {print $1}' file
1891
1892