Is Pytorch DataLoader Iteration order stable? - pytorch

Question

Is the iteration order for a Pytorch Dataloader guaranteed to be the same (under mild conditions)?
For instance:
dataloader = DataLoader(my_dataset, batch_size=4,
shuffle=True, num_workers=4)
print("run 1")
for batch in dataloader:
print(batch["index"])
print("run 2")
for batch in dataloader:
print(batch["index"])
So far, I've tried testing it and it appears to not be fixed, same order for both runs. Is there a way to make the order the same? Thanks
edit: i have also tried doing
unlabeled_sampler = data.sampler.SubsetRandomSampler(unlabeled_indices)
unlabeled_dataloader = data.DataLoader(train_dataset,
sampler=unlabeled_sampler, batch_size=args.batch_size, drop_last=False)
and then iterating through the dataloader twice, but the same non-determinism results.

Answer 1

The short answer is no, when shuffle=True the iteration order of a DataLoader isn't stable between iterations. Each time you iterate on your loader the internal RandomSampler creates a new random order.
One way to get a stable shuffled DataLoader is to create a Subset dataset using a shuffled set of indices.
shuffled_dataset = torch.utils.data.Subset(my_dataset, torch.randperm(len(my_dataset)).tolist())
dataloader = DataLoader(shuffled_dataset, batch_size=4, num_workers=4, shuffled=False)

Answer 2

I actually went with jodag's in-the-comments answer:
torch.manual_seed("0")
for i,elt in enumerate(unlabeled_dataloader):
order.append(elt[2].item())
print(elt)
if i > 10:
break
torch.manual_seed("0")
print("new dataloader")
for i,elt in enumerate( unlabeled_dataloader):
print(elt)
if i > 10:
break
exit(1)
and the output:
[tensor([[-0.3583, -0.6944]]), tensor([3]), tensor([1610])]
[tensor([[-0.6623, -0.3790]]), tensor([3]), tensor([1958])]
[tensor([[-0.5046, -0.6399]]), tensor([3]), tensor([1814])]
[tensor([[-0.5349, 0.2365]]), tensor([2]), tensor([1086])]
[tensor([[-0.1310, 0.1158]]), tensor([0]), tensor([321])]
[tensor([[-0.2085, 0.0727]]), tensor([0]), tensor([422])]
[tensor([[ 0.1263, -0.1597]]), tensor([0]), tensor([142])]
[tensor([[-0.1387, 0.3769]]), tensor([1]), tensor([894])]
[tensor([[-0.0500, 0.8009]]), tensor([3]), tensor([1924])]
[tensor([[-0.6907, 0.6448]]), tensor([4]), tensor([2016])]
[tensor([[-0.2817, 0.5136]]), tensor([2]), tensor([1267])]
[tensor([[-0.4257, 0.8338]]), tensor([4]), tensor([2411])]
new dataloader
[tensor([[-0.3583, -0.6944]]), tensor([3]), tensor([1610])]
[tensor([[-0.6623, -0.3790]]), tensor([3]), tensor([1958])]
[tensor([[-0.5046, -0.6399]]), tensor([3]), tensor([1814])]
[tensor([[-0.5349, 0.2365]]), tensor([2]), tensor([1086])]
[tensor([[-0.1310, 0.1158]]), tensor([0]), tensor([321])]
[tensor([[-0.2085, 0.0727]]), tensor([0]), tensor([422])]
[tensor([[ 0.1263, -0.1597]]), tensor([0]), tensor([142])]
[tensor([[-0.1387, 0.3769]]), tensor([1]), tensor([894])]
[tensor([[-0.0500, 0.8009]]), tensor([3]), tensor([1924])]
[tensor([[-0.6907, 0.6448]]), tensor([4]), tensor([2016])]
[tensor([[-0.2817, 0.5136]]), tensor([2]), tensor([1267])]
[tensor([[-0.4257, 0.8338]]), tensor([4]), tensor([2411])]
which is as desired. However, I think jodag's main answer is still better; this is just a quick hack which works for now ;)