Loop over Set of Functions for Variable Pipelining and Separate Usage - python-3.x

Question

I have the following 10 functions:
def function1(data1,data2):
...
return value
def function2(data1,data2):
...
return value
...
def function10(data1,data2):
...
return value
I want to use these functions separately when needed but also
in a pipeline for calculating properties and appending to a list.
Like this:
collecting_list = []
for idx in range(10):
collecting_list.append(function1(data1[idx],data2[idx]))
collecting_list.append(function2(data1[idx],data2[idx]))
collecting_list.append(function3(data1[idx],data2[idx]))
collecting_list.append(function4(data1[idx],data2[idx]))
collecting_list.append(function5(data1[idx],data2[idx]))
collecting_list.append(function6(data1[idx],data2[idx]))
collecting_list.append(function7(data1[idx],data2[idx]))
collecting_list.append(function8(data1[idx],data2[idx]))
collecting_list.append(function9(data1[idx],data2[idx]))
collecting_list.append(function10(data1[idx],data2[idx])
Obviously I would need some property to loop over function names, but I never came across this problem before and was just wondering if I can call those functions in a loop without hard coding this and just adjusting the function-number (e.g. function1(), function2(), ... function10()).
Hints and ideas appreciated!

Answer 1

use lambda and exec.
you could have a string array of the function names, and lambda functions that return the data like something below. With lambda functions, you can reuse the same name dataX over and over again and with proper implementation get the right data needed. See below for a very basic, abstract example:
import random
def getData1():
return random.randint(1, 10)
def getData2():
return random.randint(11, 20)
def function1(data1):
print("f1, {}".format(data1))
def function2(data1, data2):
print("f2, {} and {}".format(data1, data2))
data1 = lambda: getData1() # these can be any function that serves as the
data2 = lambda: getData2() # source for your data. using lambda allows for
# anonymization and reuse
functionList = ["function1({})".format(data1()), "function2({},{})".format(data1(), data2())]
for f in functionList:
exec(f)
function1(data1())
You might ask why not just use getData1() in the function list instead of data1, and the answer has to do with parameters. If the getDataX functions required parameters, you wouldn't want to compute the functionList every time a parameter name changed. This is one of the benefits of using lambda and exec.

Answer 2

Um, sure?
import sys
import types
module_name = sys.modules[__name__]
def function1(data1, data2):
return ("func1", data1 + data2)
def function2(data1, data2):
return ("func2", data1 + data2)
def function3(data1, data2):
return ("func3", data1 + data2)
def function4(data1, data2):
return ("func4", data1 + data2)
def function5(data1, data2):
return ("func5", data1 + data2)
def get_functions():
func_list = list()
for k in sorted(module_name.__dict__.keys()):
if k.startswith('function'):
if isinstance(module_name.__dict__[k], types.FunctionType):
func_list.append(module_name.__dict__[k])
return func_list
def get_functions_2():
func_list = list()
for itr in range(1, 100):
try:
func_list.append(getattr(module_name, "function%s" % itr))
except:
break
return func_list
def run_pipeline(function_list):
collecting_list = list()
for idx, func in enumerate(function_list):
collecting_list.append(func(idx, idx))
return collecting_list
if __name__ == "__main__":
funcs = get_functions()
results = run_pipeline(funcs)
print(results)
Outputs:
[('func1', 0), ('func2', 2), ('func3', 4), ('func4', 6), ('func5', 8)]
Note: I probably wouldn't do it this way if I was trying to construct dynamic computational pipelines, but you can use this method. You could in theory create a file per pipeline and name them in order to use this method though?
Edit: Added get_functions_2 per request