I have a dataset given as such:
#Load the required libraries
import pandas as pd
import matplotlib.pyplot as plt
#Create dataset
data = {'id': [1, 1, 1, 1, 1,1, 1,
2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3,
4, 4, 4, 4,
5, 5, 5, 5, 5,5],
'cycle': [1,2, 3, 4, 5,6,7,
1,2, 3,4,5,6,
1,2, 3, 4, 5,
1,2, 3, 4,
1,2, 3, 4, 5,6,],
'Salary': [7, 7, 7,7,7,7,7,
4, 4, 4,4,4,4,
8,8,8,8,8,
10,10,10,10,
15, 15,15,15,15,15,],
'Jobs': [123, 18, 69, 65, 120, 11, 52,
96, 120,10, 141, 52,6,
101,99, 128, 1, 141,
141,123, 12, 66,
12, 128, 66, 100, 141, 52,],
'Days': [123, 128, 66, 66, 120, 141, 52,
96, 120,120, 141, 52,96,
15,123, 128, 120, 141,
141,123, 128, 66,
123, 128, 66, 120, 141, 52,],
}
#Convert to dataframe
df = pd.DataFrame(data)
print("df = \n", df)
The above dataframe looks as such:
In order to plot the 'cycle' vs 'Salary' for id =1, I have used following codes:
plt.plot(df.groupby(by="id").get_group(1)['cycle'], df.groupby(by="id").get_group(1)['Salary'], label = 'id=1')
plt.xlabel('cycle')
plt.ylabel('Salary')
plt.legend()
plt.xlim(0, 10)
plt.ylim(0, 20)
plt.show()
The plot looks as such:
However, I wish to plot the 'cycle' vs 'Salary' for all id's in one single plot. The graph need to look as such:
Can somebody please let me know how to achieve this task in Python.
Use a pivot:
ax = df.pivot(index='cycle', columns='id', values='Salary').plot()
# display
ax.set_ylim(bottom=0)
ax.set_xlim(left=0)
ax.set_ylabel('Salary')
Output:
swapping axes
using seaborn.lineplot
import seaborn as sns
sns.lineplot(data=df, x='Salary', y='cycle', hue='id',
palette='Set1', estimator=None)
Output:
Related
There has a 3-dimensional array x of shape (2000,60,5). If we think it represents a video, the 2000 can represent 2000 frames. I would like to randomly sample it along with the first dimension, i.e., get a set of frame samples. For instance, how to get an array of (500,60,5) which is randomly sampled from x along with the first dimension?
You can pass x as the first argument of the choice method. If you don't want repeated frames in your sample, use replace=False.
For example,
In [10]: x = np.arange(72).reshape(9, 2, 4) # Small array for the demo.
In [11]: x
Out[11]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23]],
[[24, 25, 26, 27],
[28, 29, 30, 31]],
[[32, 33, 34, 35],
[36, 37, 38, 39]],
[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[48, 49, 50, 51],
[52, 53, 54, 55]],
[[56, 57, 58, 59],
[60, 61, 62, 63]],
[[64, 65, 66, 67],
[68, 69, 70, 71]]])
Sample "frames" from x with the choice method of NumPy random generator instance.
In [12]: rng = np.random.default_rng()
In [13]: rng.choice(x, size=3)
Out[13]:
array([[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[40, 41, 42, 43],
[44, 45, 46, 47]],
[[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [14]: rng.choice(x, size=3, replace=False)
Out[14]:
array([[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[32, 33, 34, 35],
[36, 37, 38, 39]],
[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]]])
Note that the frames will be in random order; if you want to preserve the order, you could use choice to generate an array of indices, then use the sorted indices to pull the frames out of x.
I am currently plotting temporal scatter plot using the following data (you can use these data to reproduce my plot). Data to be plotted in x-axis is time, specifically datetime.datetime object (tp_pass) while data to be plotted in y-axis is angle between -180 and 180 (azip_pass). Also, they are both numpy.array.
tp_pass=np.array([datetime.datetime(2019, 10, 29, 1, 4, 43),
datetime.datetime(2019, 10, 31, 1, 11, 19),
datetime.datetime(2019, 11, 20, 8, 26, 7),
datetime.datetime(2019, 11, 20, 23, 50, 43),
datetime.datetime(2019, 12, 10, 17, 5, 2),
datetime.datetime(2020, 1, 2, 18, 23, 53),
datetime.datetime(2020, 2, 13, 10, 33, 44),
datetime.datetime(2020, 2, 20, 18, 57, 36),
datetime.datetime(2020, 3, 25, 2, 49, 20),
datetime.datetime(2020, 4, 10, 16, 44, 56),
datetime.datetime(2020, 4, 18, 8, 25, 37),
datetime.datetime(2020, 4, 19, 20, 39, 5),
datetime.datetime(2020, 5, 3, 11, 54, 24),
datetime.datetime(2020, 5, 4, 13, 7, 48),
datetime.datetime(2020, 5, 30, 18, 13, 47),
datetime.datetime(2020, 6, 13, 15, 51, 24),
datetime.datetime(2020, 6, 24, 19, 47, 44),
datetime.datetime(2020, 7, 30, 0, 35, 56),
datetime.datetime(2020, 8, 1, 17, 9, 1),
datetime.datetime(2020, 8, 3, 8, 31, 10),
datetime.datetime(2020, 8, 18, 0, 3, 48),
datetime.datetime(2020, 9, 15, 3, 41, 28),
datetime.datetime(2020, 9, 20, 22, 13, 15),
datetime.datetime(2020, 10, 3, 9, 31, 31),
datetime.datetime(2020, 11, 6, 8, 56, 38),
datetime.datetime(2020, 11, 15, 22, 37, 43),
datetime.datetime(2020, 12, 10, 13, 19, 58),
datetime.datetime(2020, 12, 20, 17, 23, 22),
datetime.datetime(2020, 12, 24, 23, 43, 41),
datetime.datetime(2021, 1, 12, 2, 39, 43),
datetime.datetime(2021, 2, 13, 14, 7, 50),
datetime.datetime(2021, 3, 2, 21, 22, 46)], dtype=object)
azip_pass=np.array([168.3472527 , 160.09844756, 175.44976695, 159.46139347,
168.4780719 , 165.17699028, 158.22654417, 151.02735996,
159.39235045, 164.8792118 , 168.84217025, 166.09269395,
-179.97929963, 163.3389004 , 167.24285926, 167.08062597,
163.71540408, 171.13687447, 163.61945117, 172.68473083,
159.89871931, 166.72228462, 162.2774924 , 166.13812415,
14.7128006 , 12.43499853, 11.86328998, 10.56097159,
16.16589956, 12.81530251, 10.0220719 , 4.21173499])
Using the following Python script, I generated the plot.
import matplotlib.pyplot as plt
import numpy as np
import datetime
from matplotlib import dates
from matplotlib import rc
%config InlineBackend.print_figure_kwargs={'facecolor' : "w"}
rc('axes', edgecolor='k', linewidth="5.0")
fig, ax=plt.subplots(1, 1, figsize=(30, 10))
ax.xaxis.set_major_locator(dates.YearLocator())
ax.set_ylim(-185, 185)
ax.scatter(tp_pass, azip_pass, color="b", s=200, alpha=1.0, ec="k")
plt.xticks(fontsize=35)
plt.yticks([-180, -120, -60, 0, 60, 120, 180], ["${}^\circ$".format(x) for x in [-180, -120, -60, 0, 60, 120, 180]], fontsize=35)
plt.tight_layout()
plt.show()
x-axis of the plot automatically marks the year since I used matplotlib.dates.YearLocator(). Actually, I am not really satisfied with it and want to also locate months between years. However, I want months to be shown by their names, not numbers (ex. Jan, Feb, Mar, etc.). The x-axis of figure below shows what I want to implement. Is this possible using matplotlib?
Added (2021-05-18)
Using matplotlib.dates.MonthLocator(), I was able to make months show. However, the year number disappeared. Is there a way to show both year and months together (ex. year beneath month) using matplotlib?
fig, ax=plt.subplots(1, 1, figsize=(30, 10))
ax.xaxis.set_major_locator(dates.YearLocator()) # This line does not work
ax.xaxis.set_major_locator(dates.MonthLocator(bymonthday=15))
ax.xaxis.set_major_formatter(dates.DateFormatter('%b'))
ax.set_ylim(-185, 185)
ax.scatter(tp_pass, azip_pass, color="b", s=200, alpha=1.0, ec="k")
plt.xticks(fontsize=35)
plt.yticks([-180, -120, -60, 0, 60, 120, 180], ["${}^\circ$".format(x) for x in [-180, -120, -60, 0, 60, 120, 180]], fontsize=35)
plt.tight_layout()
plt.show()
Added (2021-05-19)
I found answer by Patrick FitzGerald to this question How to change the datetime tick label frequency for matplotlib plots? very helpful. This answer does not require the usage of secondary x-axis and does what I wanted to do.
You can create a second x-axis, use that to show only the year while using your original x-axis to show the month as a word. Here's this approach using your example. It will look like this.
import matplotlib.pyplot as plt
import numpy as np
import datetime
from matplotlib import dates as mdates
# Using Data from OP: tp_pass and azip_pass
# Creating your plot
fig, ax=plt.subplots(1, 1, figsize=(30, 10))
ax.set_ylim(-185, 185)
ax.scatter(tp_pass, azip_pass, color="b", s=200, alpha=1.0, ec="k")
# Minor ticks every month.
fmt_month = mdates.MonthLocator()
# Minor ticks every year.
fmt_year = mdates.YearLocator()
ax.xaxis.set_minor_locator(fmt_month)
# '%b' to get the names of the month
ax.xaxis.set_minor_formatter(mdates.DateFormatter('%b'))
ax.xaxis.set_major_locator(fmt_year)
ax.xaxis.set_major_formatter(mdates.DateFormatter('%b'))
# fontsize for month labels
ax.tick_params(labelsize=20, which='both')
# create a second x-axis beneath the first x-axis to show the year in YYYY format
sec_xaxis = ax.secondary_xaxis(-0.1)
sec_xaxis.xaxis.set_major_locator(fmt_year)
sec_xaxis.xaxis.set_major_formatter(mdates.DateFormatter('%Y'))
# Hide the second x-axis spines and ticks
sec_xaxis.spines['bottom'].set_visible(False)
sec_xaxis.tick_params(length=0, labelsize=35)
plt.yticks([-180, -120, -60, 0, 60, 120, 180], ["${}^\circ$".format(x) for x in [-180, -120, -60, 0, 60, 120, 180]], fontsize=35)
plt.tight_layout()
plt.show()
I'd suggest using ConciseDateFormatter https://matplotlib.org/stable/gallery/ticks_and_spines/date_concise_formatter.html
and using the auto locator for more ticks if you really want every month located:
fig, ax=plt.subplots(1, 1, figsize=(8, 4), constrained_layout=True)
plt.rcParams['date.converter'] = 'concise'
ax.xaxis.set_major_locator(mdates.AutoDateLocator(minticks=12, maxticks=20))
ax.set_ylim(-185, 185)
ax.scatter(tp_pass, azip_pass, color="b", s=200, alpha=1.0, ec="k")
# plt.xticks(fontsize=35)
plt.yticks([-180, -120, -60, 0, 60, 120, 180], ["${}^\circ$".format(x) for x in [-180, -120, -60, 0, 60, 120, 180]])
plt.show()
Is it possible to get the length of every sentence before padding in torchtext bucketiterator :
train_loader = torchtext.legacy.data.BucketIterator(train_data, batch_size = 64, repeat=True, shuffle=True, sort_key = lambda x: len(x.text), sort=False, sort_within_batch=True, device = device)
bucketiterator dataloader :
inputs: tensor([[ 34, 87, 2, ..., 227, 239, 263],
[ 138, 7, 1006, ..., 840, 142, 665],
[ 549, 4, 1028, ..., 11, 14, 4],
...,
[ 1, 1, 5, ..., 66, 23, 13],
[ 1, 1, 1062, ..., 177, 252, 1587],
[ 1, 1, 66, ..., 553, 52, 73]]), shape: torch.Size([64, 91])
Like when using pytorch dataloader:
train_loader = data.DataLoader(train_data, batch_size = 64, shuffle=True, collate_fn=padding)
def padding(batch):
doc = [doc['input'] for doc in batch]
len_doc = [len(doc['input']) for doc in batch]
doc_pad = pad_sequence(doc, batch_first=True, padding_value=0)
return doc_pad, len_doc
pytorch dataloader :
inputs: tensor([[ 2, 1396, 2686, ..., 0, 0, 0],
[ 2, 1391, 1396, ..., 0, 0, 0],
[ 2, 2018, 2597, ..., 0, 0, 0],
...,
[ 2, 1546, 1623, ..., 0, 0, 0],
[ 2, 1435, 1396, ..., 0, 0, 0],
[ 2, 1391, 1396, ..., 0, 0, 0]]), shape: torch.Size([64, 40])
inputs_len_before_padding: tensor([18, 8, 21, 16, 16, 12, 40, 12, 9, 12, 17, 12, 17, 15, 16, 12, 8, 24,
25, 10, 22, 8, 8, 13, 12, 22, 17, 14, 21, 14, 19, 13, 21, 8, 28, 16,
31, 24, 23, 19, 10, 7, 16, 12, 16, 12, 17, 12, 18, 11, 8, 13, 17, 14,
11, 13, 13, 20, 8, 12, 22, 7, 9, 11]), shape: torch.Size([64])
Here is a minimal example that uses torchtext.data.Field and torchtext.data.BucketIterator:
import torchtext.data as data
# sample data
text = [
'This is sentence 1.',
'This sentence is a bit longer than the previous sentence.'
]
# define field -- notice include_lengths is set to True
text_field = data.Field(include_lengths=True, tokenize=lambda x: x.split())
fields = [('text', text_field)]
# create dataset and build vocabulary
examples = [data.Example.fromlist([t], fields) for t in text]
dataset = data.Dataset(examples, fields)
text_field.build_vocab(dataset)
# create iterator
data_iter = data.BucketIterator(dataset, batch_size=2, shuffle=False)
# the text field will now return both the data tensor and the length of the input text
for x in data_iter:
print('Data:', x.text[0])
print('Lengths:', x.text[1])
This should print (data tensor shortened for brevity):
Data: tensor([[ 2, 2],
...
[ 1, 10]])
Lengths: tensor([ 4, 10])
With a 3D tensor of shape (number of filters, height, width), how can one reduce the number of filters with a reshape which keeps the original filters together as whole blocks?
Assume the new size has dimensions chosen such that a whole number of the original filters can fit side by side in one of the new filters. So an original size of (4, 2, 2) can be reshaped to (2, 2, 4).
A visual explanation of the side by side reshape where you see the standard reshape will alter the individual filter shapes:
I have tried various pytorch functions such as gather and select_index but not found a way to get to the end result in a general manner (i.e. works for different numbers of filters and different filter sizes).
I think it would be easier to rearrange the tensor values after performing the reshape but could not get a tensor of the pytorch reshaped form:
[[[1,2,3,4],
[5,6,7,8]],
[[9,10,11,12],
[13,14,15,16]]]
to:
[[[1,2,5,6],
[3,4,7,8]],
[[9,10,13,14],
[11,12,15,16]]]
for completeness, the original tensor before reshaping:
[[[1,2],
[3,4]],
[[5,6],
[7,8]],
[[9,10],
[11,12]],
[[13,14],
[15,16]]]
Another option is to construct a list of parts and concatenate them
x = torch.arange(4).reshape(4, 1, 1).repeat(1, 2, 2)
y = torch.cat([x[i::2] for i in range(2)], dim=2)
print('Before\n', x)
print('After\n', y)
which gives
Before
tensor([[[0, 0],
[0, 0]],
[[1, 1],
[1, 1]],
[[2, 2],
[2, 2]],
[[3, 3],
[3, 3]]])
After
tensor([[[0, 0, 1, 1],
[0, 0, 1, 1]],
[[2, 2, 3, 3],
[2, 2, 3, 3]]])
Or a little more generally we could write a function that takes groups of neighbors along a source dimension and concatenates them along a destination dimension
def group_neighbors(x, group_size, src_dim, dst_dim):
assert x.shape[src_dim] % group_size == 0
return torch.cat([x[[slice(None)] * (src_dim) + [slice(i, None, group_size)] + [slice(None)] * (len(x.shape) - (src_dim + 2))] for i in range(group_size)], dim=dst_dim)
x = torch.arange(4).reshape(4, 1, 1).repeat(1, 2, 2)
# read as "take neighbors in groups of 2 from dimension 0 and concatenate them in dimension 2"
y = group_neighbors(x, group_size=2, src_dim=0, dst_dim=2)
print('Before\n', x)
print('After\n', y)
You could do it by chunking tensor and then recombining.
def side_by_side_reshape(x):
n_pairs = x.shape[0] // 2
filter_size = x.shape[-1]
x = x.reshape((n_pairs, 2, filter_size, filter_size))
return torch.stack(list(map(lambda x: torch.hstack(x.unbind()), k)))
>> p = torch.arange(1, 91).reshape((10, 3, 3))
>> side_by_side_reshape(p)
tensor([[[ 1, 2, 3, 10, 11, 12],
[ 4, 5, 6, 13, 14, 15],
[ 7, 8, 9, 16, 17, 18]],
[[19, 20, 21, 28, 29, 30],
[22, 23, 24, 31, 32, 33],
[25, 26, 27, 34, 35, 36]],
[[37, 38, 39, 46, 47, 48],
[40, 41, 42, 49, 50, 51],
[43, 44, 45, 52, 53, 54]],
[[55, 56, 57, 64, 65, 66],
[58, 59, 60, 67, 68, 69],
[61, 62, 63, 70, 71, 72]],
[[73, 74, 75, 82, 83, 84],
[76, 77, 78, 85, 86, 87],
[79, 80, 81, 88, 89, 90]]])
but I know it's not ideal since there is map, list and unbind which disrupts memory. This is what I offer till I figure out how to do it via view only (so a real reshape)
How to normalize data loaded from file? Here what I have. Data looks kind of like this:
65535, 3670, 65535, 3885, -0.73, 1
65535, 3962, 65535, 3556, -0.72, 1
Last value in each line is a target. I want to have the same structure of the data but with normalized values.
import numpy as np
dataset = np.loadtxt('infrared_data.txt', delimiter=',')
# select first 5 columns as the data
X = dataset[:, 0:5]
# is that correct? Should I normalize along 0 axis?
normalized_X = preprocessing.normalize(X, axis=0)
y = dataset[:, 5]
Now the question is, how to pack correctly normalized_X and y back, that it has the structure:
dataset = [[normalized_X[0], y[0]],[normalized_X[1], y[1]],...]
It sounds like you're asking for np.column_stack. For example, let's set up some dummy data:
import numpy as np
x = np.arange(25).reshape(5, 5)
y = np.arange(5) + 1000
Which gives us:
X:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
Y:
array([1000, 1001, 1002, 1003, 1004])
And we want:
new = np.column_stack([x, y])
Which gives us:
New:
array([[ 0, 1, 2, 3, 4, 1000],
[ 5, 6, 7, 8, 9, 1001],
[ 10, 11, 12, 13, 14, 1002],
[ 15, 16, 17, 18, 19, 1003],
[ 20, 21, 22, 23, 24, 1004]])
If you'd prefer less typing, you can also use:
In [4]: np.c_[x, y]
Out[4]:
array([[ 0, 1, 2, 3, 4, 1000],
[ 5, 6, 7, 8, 9, 1001],
[ 10, 11, 12, 13, 14, 1002],
[ 15, 16, 17, 18, 19, 1003],
[ 20, 21, 22, 23, 24, 1004]])
However, I'd discourage using np.c_ for anything other than interactive use, simply due to readability concerns.