Related
I used Pytorch DataLoader to create My "batch-data" loder,but I got some problem.
As the definition of the pytorch DataLoader Shuffer.
shuffle (bool, optional) – set to True to have the data reshuffled at every epoch (default: False)
the data will be reshuffled after every epoch.
But,though I set shuffle to False,I will probably also get the completely different batch every iteration in the same epoch which I expect .
testData = torchvision.datasets.FashionMNIST(
root="data",
train=False,
download=True,
transform=ToTensor()
)
CurrentFoldTestDataLoader = data.DataLoader(testData, batch_size=32, shuffle=False)
for i in range(1000):
test_features, test_labels = next(iter(CurrentFoldTestDataLoader))
print(i,test_labels)
Here I got the same batch in every iteration.
0 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
1 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
2 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
3 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
4 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
5 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
6 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
7 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
8 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
9 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
10 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
Why is this? Is my understanding of the definition of shuffle inaccurate?
The problem with your code is that you are re-instantiating the same iterator for each step in the for cycle. With shuffle=False the iterator generates the same first batch of images. Try to instantiate the loader outside the cycle instead:
loader = data.DataLoader(testData, batch_size=32, shuffle=False)
for i, data in enumerate(loader):
test_features, test_labels = data
print(i, test_labels)
Let's say I have defined a function to generate a list of 6 random integers ranging from 0 to 10
import random
def func():
randomlist = random.sample(range(11), 6)
return randomlist
Run:
func()
Output:
[3, 7, 10, 9, 4, 1]
Now I want to define another function which calls func() inside with a parameter n to generate a multidimensional matrix, where each element will be replaced by a newly created list generated by func(), and n is the times of the completed replacement (so that the total number of integers in the matrix would be 6^n) -- for example --
when n=1, expected result:
[3, 7, 10, 9, 4, 1]
when n=2, expected result:
[[6, 2, 9, 1, 4, 0],
[7, 8, 1, 9, 4, 1],
[1, 0, 4, 6, 3, 1],
[9, 4, 3, 8, 6, 7],
[2, 4, 3, 9, 5, 6],
[4, 7, 2, 0, 1, 8]]
when n=3, expected result:
[[[4, 7, 3, 0, 2, 8],[1, 5, 6, 5, 4, 8],[8, 9, 6, 5, 10, 4],[7, 8, 6, 6, 4, 10],[7, 8, 1, 0, 2, 3],[4, 5, 8, 5, 4, 6]],
[[1, 7, 2, 0, 2, 8],[4, 5, 8, 8, 4, 5],[9, 5, 6, 2, 1, 3],[5, 4, 1, 2, 6, 10],[7, 5, 4, 1, 1, 4],[9, 6, 5, 2, 2, 1]],
[[8, 2, 7, 10, 2, 7],[8, 9, 5, 4, 5, 5],[5, 8, 7, 7, 4, 6],[9, 5, 9, 10, 5, 4],[1, 4, 5, 6, 5, 7],[9, 8, 7, 6, 5, 1]],
[[0, 7, 4, 0, 1, 9],[4, 7, 3, 0, 2, 8],[8, 9, 6, 5, 10, 4],[8, 2, 7, 10, 2, 7],[[7, 5, 4, 1, 1, 4],[7, 8, 1, 9, 4, 1]],
[[9, 5, 3, 9, 2, 8],[8, 9, 6, 5, 10, 4],[9, 4, 3, 8, 6, 7],[3, 7, 10, 9, 4, 1],[4, 7, 3, 0, 2, 8],[9, 4, 3, 8, 6, 7]],
[[5, 3, 4, 5, 2, 10],[[7, 5, 4, 1, 1, 4],[4, 7, 3, 0, 2, 8],[4, 5, 8, 8, 4, 5],[7, 8, 1, 9, 4, 1],[8, 2, 7, 10, 2, 7]]]
I think I'd need a recursive function, but I'm totally out of a clue. Any insights? Thanks a lot.
Well base case of 1 is just to call your func, otherwise build a list of the recursion call on n-1 6^(n-1) times:
def recur_6(n):
if n <= 1:
return func()
return [recur_6(n-1) for _ in range(6**(n-1))]
Live example
I'm using the scipy.stats.randint to get random numbers.
Here is my source code and result.
Input:
from scipy.stats import randint
randint.rvs(0.00001, 10, size=100)
Output:
array([6, 4, 6, 7, 9, 7, 3, 0, 2, 5, 1, 1, 0, 3, 6, 7, 3, 6, 4, 8, 6, 5,
0, 0, 5, 1, 3, 2, 3, 1, 0, 6, 5, 2, 0, 0, 9, 1, 5, 2, 3, 6, 1, 4,
3, 1, 4, 4, 9, 5, 6, 3, 4, 3, 7, 7, 2, 4, 0, 2, 0, 6, 8, 1, 5, 6,
4, 6, 5, 0, 8, 8, 5, 9, 3, 2, 8, 7, 1, 4, 6, 0, 7, 3, 9, 1, 2, 7,
7, 6, 4, 3, 3, 3, 4, 7, 7, 4, 1, 1])
My question is, I've set the low to 0.000001, but How the '0's came out from output.
Thanks for your help.
Scipy's randint invokes mtrand.randint, that is a part of Numpy package.
As you can see from its source code, lower bound is truncated using (int)(low).
So, to get random numbers from closed interval [1, 10], do the following:
randint.rvs(1, 11, size=100)
Note, you need to increase high bound by 1, as it seen from the form of probability distribution (pmf) for randint.
l have an RGB image of dimension (224,224,3). l applied superpixel segmentation on it using SLIC algorithm.
As follow :
img= skimageIO.imread("first_image.jpeg")
print('img shape', img.shape) # (224,224,3)
segments_slic = slic(img, n_segments=1000, compactness=0.01, sigma=1) # Up to 1000 segments
segments_slic.shape
(224,224)
Number of returned segments are :
np.max(segments_slic)
Out[49]: 595
From 0 to 595. So, we have 596 superpixels (regions).
Let's take a look at segments_slic[0]
segments_slic[0]
Out[51]:
array([ 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5,
5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7,
8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9,
10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12,
12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14,
14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16,
16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18,
18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20,
20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23,
23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25,
25, 25, 25])
What l would like to get ?
for each superpixel region make two arrays as follow:
1) Array : contain the indexes of the pixels belonging to the same superpixel.
For instance
superpixel_list[0] contains all the indexes of the pixels belonging to superpixel 0 .
superpixel_list[400] contains all the indexes of the pixels belonging to superpixel 400
2)superpixel_pixel_values[0] : contains the pixel values (in RGB) of the pixels belonging to superpixel 0.
For instance, let's say that pixels 0, 24 , 29, 53 belongs to the superpixel 0. Then we get
superpixel[0]= [[223,118,33],[245,222,198],[98,17,255],[255,255,0]]# RGB values of pixels belonging to superpixel 0
What is the efficient/optimized way to do that ? (Because l have l dataset of images to loop over)
EDIT-1
def sp_idx(s, index = True):
u = np.unique(s)
if index:
return [np.where(s == i) for i in u]
else:
return [s[s == i] for i in u]
#return [s[np.where(s == i)] for i in u] gives the same but is slower
superpixel_list = sp_idx(segments_slic)
superpixel = sp_idx(segments_slic, index = False)
In superpixel_list we are supposed to get a list containing the index of pixels belonging to the same superpixel.
For instance
superpixel_list[0] is supposed to get all the pixel indexes of the pixel affected to superpixel 0
however l get the following :
superpixel_list[0]
Out[73]:
(array([ 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5,
5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7,
7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 10,
10, 10, 10, 10, 11, 11, 11, 11, 12, 12, 12, 12, 13, 13, 13]),
array([0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5,
6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6,
7, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 0, 1,
2, 3, 4, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2]))
Why two arrays ?
In superpixel[0] for instance we are supposed to get the RGB pixel values of each pixel affected to supepixel 0 as follow :
for instance pixels 0, 24 , 29, 53 are affected to superpixel 0 then :
superpixel[0]= [[223,118,33],[245,222,198],[98,17,255],[255,255,0]]
However when l use your function l get the following :
superpixel[0]
Out[79]:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
Thank you for your help
Can be done using np.where and the resulting indices.
def sp_idx(s, index = True):
u = np.unique(s)
return [np.where(s == i) for i in u]
superpixel_list = sp_idx(segments_slic)
superpixel = [img[idx] for idx in superpixel_list]
I'm trying to programm the Data Encyption Standard on my own and I'm struggling to Programm the SBoxes. I know there is already a module to encrypt and decrypt with the DES but my teacher asked to programm it myself, so here is what i have:
import random
from re import findall
class DES:
def __init__(self):
self. Eingabe=""
self.Schluessel=""
self.NachrichtBinaer=""
self.Bitslinks=""
self.Bitsrechts=""
self.Teil1=""
self.Teil2=""
self.Subkey=""
self.pcschluessel=""
self.subkeyliste=[]
self.initialpermutation=""
self.liste1=[]
self.Bits48=""
self.ausgabesbox=[]
self.liste2=[]
def EingabeNachricht(self):
self.Eingabe=input("Geben Sie ein Wort ein:")
print ("Eingegebenes Wort: ",self.Eingabe)
def Bitumwandlung(self):
for i in range(0,len(self.Eingabe)):
self.NachrichtBinaer=self.NachrichtBinaer+bin(ord(self.Eingabe[i]))
self.NachrichtBinaer=self.NachrichtBinaer.replace("b","")
if len(self.NachrichtBinaer)<64:
self.NachrichtBinaer=self.NachrichtBinaer.rjust(64,"0")
self.NachrichtBinaer="0000000100100011010001010110011110001001101010111100110111101111"
print ("Nachricht in Binaer: ",self.NachrichtBinaer)
def Teilen(self):
self.Bitslinks=self.initialpermutation[:int(len(self.initialpermutation)/2)]
self.Bitsrechts=self.initialpermutation[int(len(self.initialpermutation)/2):]
print ("Teil links: ",self.Bitslinks)
print ("Teil rechts: ",self.Bitsrechts)
def SchluesselGenerieren(self):
#self.Schluessel= getrandbits(64)
self.Schluessel="0001001100110100010101110111100110011011101111001101111111110001"
def ippermutation(self):
ip = [57, 49, 41, 33, 25, 17, 9,1,
59, 51, 43, 35, 27, 19, 11, 3,
61, 53, 45, 37, 29, 21, 13, 5,
63, 55, 47, 39, 31, 23, 15, 7,
56, 48, 40, 32, 24, 16, 8, 0,
58, 50, 42, 34, 26, 18, 10, 2,
60, 52, 44, 36, 28, 20, 12, 4,
62, 54, 46, 38, 30, 22, 14, 6
]
for i in ip:
self.initialpermutation=self.initialpermutation+self.NachrichtBinaer[i]
print ("Erste Permutation: ",self.initialpermutation)
def Expandieren(self):
ExpandierenTabelle = [
31,0,1,2,3,4,
3,4,5,6,7,8,
7,8,9,10,11,12,
11,12,13,14,15,16,
15,16,17,18,19,20,
9,20,21,22,23,24,
3,24,25,26,27,28,
27,28,29,30,31,0]
for Elemente in ExpandierenTabelle:
self.Bits48 = self.Bits48 + self.Bitsrechts[Elemente]
print ("expandiert: ",self.Bits48)
def XOR(self,wert1,wert2):
antwort=wert1^wert2
return antwort
def SBox(self):
self.sbox=[
[[14, 4, 13, 1, 2, 15, 11, 8, 3, 10, 6, 12, 5, 9, 0, 7],
[0, 15, 7, 4, 14, 2, 13, 1, 10, 6, 12, 11, 9, 5, 3, 8],
[4, 1, 14, 8, 13, 6, 2, 11, 15, 12, 9, 7, 3, 10, 5, 0],
[15, 12, 8, 2, 4, 9, 1, 7, 5, 11, 3, 14, 10, 0, 6, 13]],
# S2
[[15, 1, 8, 14, 6, 11, 3, 4, 9, 7, 2, 13, 12, 0, 5, 10],
[3, 13, 4, 7, 15, 2, 8, 14, 12, 0, 1, 10, 6, 9, 11, 5],
[0, 14, 7, 11, 10, 4, 13, 1, 5, 8, 12, 6, 9, 3, 2, 15],
[13, 8, 10, 1, 3, 15, 4, 2, 11, 6, 7, 12, 0, 5, 14, 9]],
# S3
[[10, 0, 9, 14, 6, 3, 15, 5, 1, 13, 12, 7, 11, 4, 2, 8],
[13, 7, 0, 9, 3, 4, 6, 10, 2, 8, 5, 14, 12, 11, 15, 1],
[13, 6, 4, 9, 8, 15, 3, 0, 11, 1, 2, 12, 5, 10, 14, 7],
[1, 10, 13, 0, 6, 9, 8, 7, 4, 15, 14, 3, 11, 5, 2, 12]],
# S4
[[7, 13, 14, 3, 0, 6, 9, 10, 1, 2, 8, 5, 11, 12, 4, 15],
[13, 8, 11, 5, 6, 15, 0, 3, 4, 7, 2, 12, 1, 10, 14, 9],
[10, 6, 9, 0, 12, 11, 7, 13, 15, 1, 3, 14, 5, 2, 8, 4],
[3, 15, 0, 6, 10, 1, 13, 8, 9, 4, 5, 11, 12, 7, 2, 14]],
# S5
[[2, 12, 4, 1, 7, 10, 11, 6, 8, 5, 3, 15, 13, 0, 14, 9],
[14, 11, 2, 12, 4, 7, 13, 1, 5, 0, 15, 10, 3, 9, 8, 6],
[4, 2, 1, 11, 10, 13, 7, 8, 15, 9, 12, 5, 6, 3, 0, 14],
[11, 8, 12, 7, 1, 14, 2, 13, 6, 15, 0, 9, 10, 4, 5, 3]],
# S6
[[12, 1, 10, 15, 9, 2, 6, 8, 0, 13, 3, 4, 14, 7, 5, 11],
[10, 15, 4, 2, 7, 12, 9, 5, 6, 1, 13, 14, 0, 11, 3, 8],
[9, 14, 15, 5, 2, 8, 12, 3, 7, 0, 4, 10, 1, 13, 11, 6],
[4, 3, 2, 12, 9, 5, 15, 10, 11, 14, 1, 7, 6, 0, 8, 13]],
# S7
[[4, 11, 2, 14, 15, 0, 8, 13, 3, 12, 9, 7, 5, 10, 6, 1],
[13, 0, 11, 7, 4, 9, 1, 10, 14, 3, 5, 12, 2, 15, 8, 6],
[1, 4, 11, 13, 12, 3, 7, 14, 10, 15, 6, 8, 0, 5, 9, 2],
[6, 11, 13, 8, 1, 4, 10, 7, 9, 5, 0, 15, 14, 2, 3, 12]],
# S8
[[13, 2, 8, 4, 6, 15, 11, 1, 10, 9, 3, 14, 5, 0, 12, 7],
[1, 15, 13, 8, 10, 3, 7, 4, 12, 5, 6, 11, 0, 14, 9, 2],
[7, 11, 4, 1, 9, 12, 14, 2, 0, 6, 10, 13, 15, 3, 5, 8],
[2, 1, 14, 7, 4, 10, 8, 13, 15, 12, 9, 0, 3, 5, 6, 11]]
]
for z in range(len(self.sbox)):
for y in range(len(self.sbox[z])):
for x in range(len(self.sbox[z][y])):
ausserebits=self.liste2[x][z][0]+self.liste2[x][z][-1]
innerebits=self.liste2[x][z][1:5]
print ("innere: ",innerebits)
print ("aussere: ",ausserebits)
def pc1undteilen(self):
pc1 = [56,48,40,32,24,16,8,
0,57,49,41,33,25,17,
9,1,58,50,42,34,26,
18,19,2,59,51,43,35,
62,54,46,38,30,22,14,
6,61,53,45,37,29,21,
13,5,60,52,44,36,28,
20,2,4,27,19,11,3]
#Subkey = ""
for j in pc1:
self.Subkey = self.Subkey+self.Schluessel[j]
self.Teil1=self.Subkey[:int(len(self.Subkey)/2)]
self.Teil2=self.Subkey[int(len(self.Subkey)/2):]
print("Schluessel64 :",self.Subkey)
print("SchluesselTeil1 :",self.Teil1)
print("SchluesselTeil2 :",self.Teil2)
def rotationundpc2(self):
pc2 = [
13, 16, 10, 23, 0, 4,
2, 27, 14, 5, 20, 9,
22, 18, 11, 3, 25, 7,
15, 6, 26, 19, 12, 1,
40, 51, 30, 36, 46, 54,
29, 39, 50, 44, 32, 47,
43, 48, 38, 55, 33, 52,
45, 41, 49, 35, 28, 31]
rotation = [
1, 1, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 1]
k=0
#schluesselliste
self.Teil1=list(self.Teil1)
self.Teil2=list(self.Teil2)
while k<16:
#rotation
u=0
while u < rotation[k]:
self.Teil1.append(self.Teil1[0])
del self.Teil1[0]
self.Teil2.append(self.Teil2[0])
del self.Teil2[0]
self.Teil1="".join(self.Teil1)
self.Teil2="".join(self.Teil2)
self.subschluessel=self.Teil1+self.Teil2
print("Teil1: ",self.Teil1)
print("Teil2: ",self.Teil2)
print ("Subschluessel: ",self.subschluessel)
self.Teil1=list(self.Teil1)
self.Teil2=list(self.Teil2)
u+=1
#pc2 und erstellung der 16 subkeys
for index2 in pc2:
self.subkeyliste.append(self.subschluessel[index2])
k+=1
while len(self.subkeyliste)>0:
self.liste1.append("".join(self.subkeyliste[0:48]))
del self.subkeyliste[0:48]
print ("Subkeyliste: ",self.liste1)
def sechsbitunterteilung(self):
for l in range(0,16):
self.liste2.append(findall("......",listenachxor[l]))
print ("liste2: ",self.liste2)
#objekt der klasse DES wird erstellt
listenachxor=[]
Krypto=DES()
#Schluesselgenerieren
Krypto.SchluesselGenerieren()
Krypto.pc1undteilen()
Krypto.rotationundpc2()
Krypto.EingabeNachricht()
Krypto.Bitumwandlung()
Krypto.ippermutation()
Krypto.Teilen()
Krypto.Expandieren()
for p in range(0,16):
listenachxor.append(bin(Krypto.XOR(int(Krypto.liste1[p],2),int(Krypto.Bits48,2))))
listenachxor[p]=listenachxor[p].replace("b","")
print ("listenachxor: ",listenachxor)
Krypto.sechsbitunterteilung()
Krypto.SBox()
By the way the problem is on this part of the programm, the rest just works fine:
for z in range(len(self.sbox)):
for y in range(len(self.sbox[z])):
for x in range(len(self.sbox[z][y])):
ausserebits=self.liste2[x][z][0]+self.liste2[x][z][-1]
innerebits=self.liste2[x][z][1:5]
print ("innere: ",innerebits)
print ("aussere: ",ausserebits)