How to pass efficiently a huge sized parameter to a function? - python-3.x

Question

thank you reading. Please forgive my English, I'm french. Is it a justification ?
The aim of my function is to scan many IPv4:PORT via multiprocessing/threading (do not, I forsee you, do not !) in Python3, send a SYN and register any result. Multisomething is called here because of the timeout needed to wait to any potential reply.
So.
This function is called a producer. It stores potential TCP connection into a database. Consumers
then read the database and do some pentesting indepently. So.
I prepared an IPv4 list. It's a random valid IP list of 10's K elements. Remember we have 65K ports to test per IPv4.
My method is then to suffle the list port with a new seed for each producer launched. I have many. Each have a valid IPv4 list, but, if we consider 65K ports with 100K IP's, we have a 6.5G elements to pass.
Why ?
IP[] are random.shuffle()-like, by construction. Ports[] are too.
If I read p in Ports and for each p, join IPv4:Ports and append into params[], I can launch the parallelized job via scan(u) for u in params[]
I launch it via run(rounds) like this :
def run(rounds):
for r in rounds:
scan(r)
But, the problem is size(rounds) = size(params) ~ 6.5G elements
I cannot find a efficient (memory talking) way to pass such a huge parameter (big list) to a parallelized job. I'm running out of memory.
I'm not asking how to manage it on a mind-blowing-capable workstation, while having designed this function on a paper, it doesn't fit into the raspi2 (1GB mem).
I do not need Python3 at all for myself, it a teaching product. Thus, i'm stuck.
My question is : could you help me to find a efficient way to attack a huge list by a parallelized function that pop's the list without sending it via a parameter ?
I googled, followed forums and threads I'm aware of, but, as I refactor the programm, the constant problem stays, laughing at me, at the same place in the main().
What I dont want to reproduce :
MAX_IPV4 = i.IPv4Address._ALL_ONES
MAX_PORT = 65535
def r_ipv4():
while True:
a=i.IPv4Address._string_from_ip_int(r.randint(0, MAX_IPV4))
b=i.IPv4Address(a)
if b.is_multicast or b.is_private or b.is_loopback or b.is_reserved or b.is_link_local :
continue
else :
return a
break
def generate_target():
return r_ipv4()
def generate_r_targets(n):
ip=[]
for i in range(0,n):
ip.append(generate_target())
ports=[]
for p in range(1,65535):
ports.append(p)
r.shuffle(ports)
targets=[]
for p in ports:
for z in ip:
targets.append((str(z)+","+str(p)))
It's not my optimal method, but the way I can explain and show the problem at best.
I dont want to lost the (shuffled ports) iterable, it is my principal vector and it is at the heart.
I'm testing a dynamic configuration of knockd.
Say. Computer_one register a sequence in knockd.
Computer_one is the only one to know the seed.
The seed is the key of the sequence of keys of knockd, that's done.

Answer 1

We passed it via a plain file :)