How to update Gtk.ListStore data efficiently in python - python-3.x

I'm at the last stages of completing my program's transition from Tkinter to Gtk. The last step is updating the ListStore in an efficient manner. So far, what I'm trying is working...but for only about 15-120 seconds before it crashes (error code is 139; sometimes a coredump with no other message, sometimes a "Warning: corrupted double-linked list detected"). While it's running, top reports less than 100 MB of RAM in use.
The ListStore contains 51 rows with 25 columns, created as follows:
def init_ListStore():
# initialize the main ListStore database
for symbol in cbp_symbols.keys():
name = cbp_symbols[symbol]['name']
row = [symbol, name, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
liststore1.append(row)
treeview1.set_model(liststore1)
cell = Gtk.CellRendererText()
columns = list(my_vars['liststore_main']['column_data'].keys())
for i in range(liststore1.get_n_columns()):
treeviewcolumn = Gtk.TreeViewColumn(columns[i])
treeview1.append_column(treeviewcolumn)
treeviewcolumn.pack_start(cell, True)
treeviewcolumn.add_attribute(cell, "text", i)
After this is done, I do the following (3) steps:
Update my account information (token balances), get a new message from the websocket (stored as dictionary), update the new info, on a cell-to-cell basis, something like:
liststore1[row][col.index('Open')] = float(ticker_data["open_24h"])
liststore1[row][col.index('Last')] = float(ticker_data["price"])
liststore1[row][col.index('Low')] = float(ticker_data["low_24h"])
liststore1[row][col.index('High')] = float(ticker_data["high_24h"])
liststore1[row][col.index('Volume')] = float(ticker_data["volume_24h"])
I would like to be able to edit/replace the entire row in one shot; the above seems slow and burdensome. Once the initial creation/updates are done, everything is being done with the same function (there are about 20 total updates to be made with each message).
UPDATE #1: this time I got a more definitive error. The problem is in the lib/python3.8/site-packages/gi/overrides/Gtk.py module, line 1680:
if GTK2 or GTK3:
_Gtk_main_quit = Gtk.main_quit
#override(Gtk.main_quit)
def main_quit(*args):
_Gtk_main_quit()
_Gtk_main = Gtk.main
#override(Gtk.main)
def main(*args, **kwargs):
with register_sigint_fallback(Gtk.main_quit):
with wakeup_on_signal():
return _Gtk_main(*args, **kwargs) << this line
UPDATE #2: I think this is related to multithread locking, but I don't have a clue how that works (module is imported by the websocket client, not by me). I did try to apply a lock on the main function that parses the incoming socket message, but it didn't help. I do know that if I run the program without the Gtk window active, it never crashes.

Finally got this working. The trick was to use Gtk.TreeIter to reference the current row, then update the entire row in one shot:
titer = liststore1.get_iter(row)
my_columns = [
df.columns.get_loc('Last') + 1,
df.columns.get_loc('Open') + 1,
df.columns.get_loc('Change') + 1,
df.columns.get_loc('Change %') + 1,
df.columns.get_loc('Low') + 1,
df.columns.get_loc('High') + 1,
df.columns.get_loc('Value') + 1,
df.columns.get_loc('Val Open') + 1,
df.columns.get_loc('Port %') + 1,
df.columns.get_loc('Cost Basis') + 1,
df.columns.get_loc('CB Change') + 1,
df.columns.get_loc('Value') + 1,
df.columns.get_loc('Val Open') + 1,
df.columns.get_loc('Profit') + 1,
df.columns.get_loc('Volume') + 1,
df.columns.get_loc('30d Avg') + 1,
df.columns.get_loc('Vol Act %') + 1,
df.columns.get_loc('Trade Val') + 1
]
my_data = [
df.at[row, 'Last'],
df.at[row, 'Open'],
df.at[row, 'Change'],
df.at[row, 'Change %'],
df.at[row, 'Low'],
df.at[row, 'High'],
df.at[row, 'Value'],
df.at[row, 'Val Open'],
df.at[row, 'Port %'],
df.at[row, 'Cost Basis'],
df.at[row, 'CB Change'],
df.at[row, 'Value'],
df.at[row, 'Val Open'],
df.at[row, 'Profit'],
df.at[row, 'Volume'],
df.at[row, '30d Avg'],
df.at[row, 'Vol Act %'],
df.at[row, 'Trade Val']
]
liststore1.set(titer, my_columns, my_data)
I know this code needs to be streamlined, but I tend to get something working first, then simplify/streamline afterwards.

Related

rust vulkan, normals are messed up

I am trying to render a gltf model with ash (vulkan) in rust.
I sent all my data to the gpu and I am seeing this:
Naturally my suspicion is that the normal data is wrong. So I checked with renderdoc:
Those seem ok, maybe the attributes are wrong?
All those normals seem like they add to 1, should be fine. Maybe my pipeline is wrong?
Seems like the correct format and binding (I am sending 3 buffers and binding one to binding 0, one to 1 and the last to 2, binding 1 has the normals).
The only thing I find that is weird is, if I go to the vertex input pipeline stage see the buffers:
This is what the buffer at index 1 shows:
This does not happen for the buffer at index 0 (positions) which also render properly. So Whatever is causing the normals to show up as hex codes here is likely the cause of the bug. But i have no idea why this is happening. As far as I can see the pipeline and buffer were all set properly.
You presumably want to use one separate buffer for each vertex attribute (aka non-interleaved vertex buffer, SoA),
but your VkVertexInputAttributeDescription::offset values [0, 12, 24] is what you would use for one vertex buffer interleaving all attributes (provided that their binding values point to one and the same VkVertexInputBindingDescription).
e.g.
// Interleaved:
// Buffer 0: |Position: R32G32B32_FLOAT, Normal: R32G32B32_FLOAT, Uv: R32G32B32_FLOAT|, * vertex count
VkVertexInputBindingDescription {
.binding = 0,
.stride = 12 * 3, // 3 `R32G32B32_FLOAT`s !
.inputRate = VK_VERTEX_INPUT_RATE_VERTEX
};
// All attributes in the same `binding` == `0`
VkVertexInputAttributeDescription[3] {
{
.location = 0,
.binding = 0,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 0 // [0, 11] portion
},
{
.location = 1,
.binding = 0,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 12 // [12, 23] portion
},
{
.location = 2,
.binding = 0,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 24 // [24, 35] portion
}
};
Your VkVertexInputBindingDescription[1].stride == 12 tells Vulkan that your vertex buffer 1 uses 12 bytes for each vertex, and your VkVertexInputAttributeDescription[1].offset == 12 says the normal value is at offset 12, which is out of bounds.
Same deal with your VkVertexInputAttributeDescription[2].offset == 24 overstepping (by a large amount) VkVertexInputBindingDescription[2].stride == 12.
For using one tightly-packed buffer for each vertex attribute, you need to correctly set your VkVertexInputAttributeDescription[n].offset values to 0, which looks something like:
// Non-interleaved:
// Buffer 0: |Position: R32G32B32_FLOAT|, * vertex count
// Buffer 1: |Normal: R32G32B32_FLOAT|, * vertex count
// Buffer 2: |Uv: R32G32B32_FLOAT|, * vertex count
VkVertexInputBindingDescription[3] {
{
.binding = 0,
.stride = 12,
.inputRate = VK_VERTEX_INPUT_RATE_VERTEX
},
{
.binding = 1,
.stride = 12,
.inputRate = VK_VERTEX_INPUT_RATE_VERTEX
},
{
.binding = 2,
.stride = 12,
.inputRate = VK_VERTEX_INPUT_RATE_VERTEX
}
};
// Each attribute in its own `binding` == `location`
VkVertexInputAttributeDescription[3] {
{
.location = 0,
.binding = 0,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 0 // Whole [0, 11]
},
{
.location = 1,
.binding = 1,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 0 // Whole [0, 11]
},
{
.location = 2,
.binding = 2,
.format = VK_FORMAT_R32G32G32_SFLOAT,
.offset = 0 // Whole [0, 11]
}
};
Worth noting is the comment line // vertex stride 12 less than total data fetched 24 generated by RenderDoc in the Buffer Format section, and how it does so.
It detects when your vertex attribute description oversteps its binding description's stride:
if(i + 1 == attrs.size())
{
// for the last attribute, ensure the total size doesn't overlap stride
if(attrs[i].byteOffset + cursz > stride && stride > 0)
return tr("// vertex stride %1 less than total data fetched %2")
.arg(stride)
.arg(attrs[i].byteOffset + cursz);
}

Aligning QGridLayouts

I'm trying to create a 'table' where the headers and row labels remain visible. Kind of like the Frozen Column Example, but using a QGridLayout.
My approach is to put three QGridLayouts into aligned QScrollAreas, where the main table contains RxC rows and columns and is scrollable in any direction, and the column and row headers are contained in 1xC and Rx1 cells, respectively. Then I can just get the different QScrollAreas to track each other.
I've got this working (though I haven't touched the matched-scrolling bit yet), but I'm having a lot of trouble getting the three different QGridLayouts to have the same size cells (in width or height).
Part of the problem is that the table cells can be variously sized, meaning that each row and column is not the same height/width as all of the others. But I also can't figure out how to get the heights/widths of the rows of the different QGridLayouts to match.
As you can see in the image above, there are three rows and two columns. I want the Row Headers to line up with the three rows, and the Column Headers to line up with the two columns.
Here's a minimal example. There's a lot still for me to figure out and get working, but this first step is an important one.
import PyQt5.QtGui
import PyQt5.QtWidgets
import PyQt5.QtCore
class MainBuyerWindow(PyQt5.QtWidgets.QMainWindow):
def __init__(self, *args, **kwargs):
super(PyQt5.QtWidgets.QMainWindow, self).__init__(*args, **kwargs)
(
columns_widget, rows_widget, table_widget,
columns_layout, rows_layout, table_layout,
) = self.setup_table()
self.columns_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.rows_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.table_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.columns_scroll_widget.setWidget(columns_widget)
self.rows_scroll_widget.setWidget(rows_widget)
self.table_scroll_widget.setWidget(table_widget)
central_layout = PyQt5.QtWidgets.QGridLayout()
central_layout.addWidget(self.columns_scroll_widget, 0, 1, 1, 5)
central_layout.addWidget( self.rows_scroll_widget, 1, 0, 5, 1)
central_layout.addWidget( self.table_scroll_widget, 1, 1, 5, 5)
central_widget = PyQt5.QtWidgets.QWidget()
central_widget.setLayout(central_layout)
self.setCentralWidget(central_widget)
self.show()
def setup_table(self):
columns_layout = PyQt5.QtWidgets.QGridLayout()
rows_layout = PyQt5.QtWidgets.QGridLayout()
table_layout = PyQt5.QtWidgets.QGridLayout()
columns_widget = PyQt5.QtWidgets.QWidget()
columns_widget.setLayout(columns_layout)
rows_widget = PyQt5.QtWidgets.QWidget()
rows_widget.setLayout(rows_layout)
table_widget = PyQt5.QtWidgets.QWidget()
table_widget.setLayout(table_layout)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("This cell\nis a\ntall one"), 0, 0)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("This cell is shorter"), 1, 0)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("This cell is of\nmedium height"), 2, 0)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("Also notice that widths of columns are not all the same"), 0, 1)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("Though that doesn't matter"), 1, 1)
table_layout.addWidget(PyQt5.QtWidgets.QLabel("(In this example anyway)"), 2, 1)
columns_layout.addWidget(PyQt5.QtWidgets.QLabel("Column Header 1"), 0, 0)
columns_layout.addWidget(PyQt5.QtWidgets.QLabel("Column Header 2"), 0, 1)
rows_layout.addWidget(PyQt5.QtWidgets.QLabel("Row Header 1"), 0, 0)
rows_layout.addWidget(PyQt5.QtWidgets.QLabel("Row Header 2"), 1, 0)
rows_layout.addWidget(PyQt5.QtWidgets.QLabel("Row Header 3"), 2, 0)
return (
columns_widget, rows_widget, table_widget,
columns_layout, rows_layout, table_layout,
)
######################################################################
if __name__ == "__main__":
app = PyQt5.QtWidgets.QApplication([])
window = MainBuyerWindow()
app.exec_()
I think my main problem is I don't understand Qt well enough, so I'm not sure what to look at. What I tried was calling the show() (or activate()) method to get the geometries all worked out, and then going through the first row and column of the table_layout to get their dimensions using cellRect, and then calling columnMinimumWidth and rowMinimumHeight on the corresponding row/column of rows_layout and columns_layout.
This is the same idea used in this (almost identical) question, but when I implement it here it also doesn't seem to work. (Essentially just adding the following lines)
table_layout.activate()
for i in range(table_layout.columnCount()):
w = table_layout.columnMinimumWidth(i)
columns_layout.setColumnMinimumWidth(i, w)
for i in range(table_layout.rowCount()):
h = table_layout.rowMinimumHeight(i)
rows_layout.setRowMinimumHeight(i, h)
Can someone help me out here? Thank you so much!
I would set a fixed height to each row header label based on the height of that row in the table (and similarly a fixed width to each column header label based on the width of that column in the table), which fixes the layout alignment but also makes it simple to match up the scrolling.
class MainBuyerWindow(PyQt5.QtWidgets.QMainWindow):
def __init__(self, *args, **kwargs):
super(PyQt5.QtWidgets.QMainWindow, self).__init__(*args, **kwargs)
(
columns_widget, rows_widget, table_widget,
columns_layout, rows_layout, table_layout,
) = self.setup_table()
self.columns_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.rows_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.table_scroll_widget = PyQt5.QtWidgets.QScrollArea()
self.columns_scroll_widget.setWidget(columns_widget)
self.rows_scroll_widget.setWidget(rows_widget)
self.table_scroll_widget.setWidget(table_widget)
for i in range(rows_layout.count()):
label = rows_layout.itemAt(i).widget()
label.setFixedHeight(table_layout.cellRect(i, 0).height())
for i in range(columns_layout.count()):
label = columns_layout.itemAt(i).widget()
label.setFixedWidth(table_layout.cellRect(0, i).width())
...
Use QScrollArea.setWidgetResizable(True) so the scroll area can resize the widget as needed, and allow the header labels to take up the required amount of space to be aligned with the table.
...
self.columns_scroll_widget.setWidgetResizable(True)
self.rows_scroll_widget.setWidgetResizable(True)
self.table_scroll_widget.setWidgetResizable(True)
# Size policies to keep the row and column headers minimal/sufficient
# and allow table_scroll_widget to get as much space as possible when resized
self.columns_scroll_widget.setSizePolicy(PyQt5.QtWidgets.QSizePolicy.Preferred, PyQt5.QtWidgets.QSizePolicy.Minimum)
self.rows_scroll_widget.setSizePolicy(PyQt5.QtWidgets.QSizePolicy.Minimum, PyQt5.QtWidgets.QSizePolicy.Preferred)
...
QScrollBar inherits from QAbstractSlider so you can connect its valueChanged signal to the setValue slot of another scroll bar. These values will be consistent across the layouts now since their row heights and column widths are equal.
...
# User scrolling in table will cause headers to scroll
self.table_scroll_widget.horizontalScrollBar().valueChanged[int].connect(self.columns_scroll_widget.horizontalScrollBar().setValue)
self.table_scroll_widget.verticalScrollBar().valueChanged[int].connect(self.rows_scroll_widget.verticalScrollBar().setValue)
# User scrolling on headers will cause table to scroll
self.columns_scroll_widget.horizontalScrollBar().valueChanged[int].connect(self.table_scroll_widget.horizontalScrollBar().setValue)
self.rows_scroll_widget.verticalScrollBar().valueChanged[int].connect(self.table_scroll_widget.verticalScrollBar().setValue)
central_layout = PyQt5.QtWidgets.QGridLayout()
central_layout.addWidget(self.columns_scroll_widget, 0, 1, 1, 5)
central_layout.addWidget( self.rows_scroll_widget, 1, 0, 5, 1)
central_layout.addWidget( self.table_scroll_widget, 1, 1, 5, 5)
central_widget = PyQt5.QtWidgets.QWidget()
central_widget.setLayout(central_layout)
self.setCentralWidget(central_widget)
self.show()
Outcome:

Error when converting working code to a more object oriented structure Error: string indices must be integers Python 3

I keep running into the error: string indices must be integers when using placeholders for my script.
My program is supposed to track the growth of veggies purely on calculation. The idea is that each plant has it's own characteristics (eg carrotData) but instead of having code for each 'plantData' I replaced the code with (whichPlant and later whichPlantData) as a temporary placeholder (so that I don't need new code for each plant I have in my garden or that I want to add at a later point).
This is when I get the error in the last line (Plant is a class) marked with the ***. When I use (carrotData) instead of (whichPlantData) my script works. But as soon as I put in the temporary placeholder (whichPlantData) is breaks.
What causes this (so that I can avoid doing this in future projects) and how can I fix this?
thanks for the support!!
carrotData = {'plantID': '','plantingTime': dt(year=now.year, month=3, day=1), "dateOfPlanting": 0, "numberOfPlants": 0, "germinationTime": 7, "growthTime": 227, "flowerTime": 247, "harvestTime": 254, "liveCycles": 1, "status": 0}
potatoData = {'plantID': '','plantingTime': dt(year=now.year, month=3, day=1), "dateOfPlanting": 0, "numberOfPlants": 0, "germinationTime": 7, "growthTime": 227, "flowerTime": 247, "harvestTime": 254, "liveCycles": 1, "status": 0}
print ("imported plant datasheets")
#functions:
#if plant is added
def addPlant():
whichPlant = input("Which plant do you want to add? ")
n = int(input("How many plants do you want to add? "))
i = 0
whichPlantData = whichPlant + "Data"
if whichPlant in inventory:
while i < n:
i += 1
if whichPlant in plants:
plants[whichPlant] += 1
else:
plants[whichPlant] = 1
***Error*** whichPlant = Plant("", whichPlantData['plantingTime'], dt.now(), n, dt.now() + timedelta(days=whichPlantData['germinationTime']), dt.now() + timedelta(days=whichPlantData['growthTime']), dt.now() + timedelta(days=whichPlantData['flowerTime']),whichPlantData['harvestTime'], whichPlantData['liveCycles'], whichPlantData['status'])
Your problem seems to be with whichPlantData = whichPlant + "Data". whichPlant is a string returned by the input function. I think what you're trying to do is get a dictionary of plant information based on input from the user. Furthermore; whichPlant + "Data" seems like an attempt at making whichPlant the same as a variable name pointing towards a dictionary of plant information. Just because the string whichPlant may be equal to the variable name carrotData does not make it the same as the variable. I would suggest making a list of dictionaries full of the information about the plant then iterate over the items in that list to see if the dictionaries name key is the same as the user input.
Similar to this:
plants = [{"Name": "Carrot", 'plantID': '','plantingTime':0, "dateOfPlanting": 0, "numberOfPlants": 0, "germinationTime": 7, "growthTime": 227, "flowerTime": 247, "harvestTime": 254, "liveCycles": 1, "status": 0},
{"Name": "Potato", 'plantID': '','plantingTime': 0, "dateOfPlanting": 0, "numberOfPlants": 0, "germinationTime": 7, "growthTime": 227, "flowerTime": 247, "harvestTime": 254, "liveCycles": 1, "status": 0}]
PlantName = input("Enter a Plant: ")
for plant in plants:
if plant['Name'] == PlantName:
print("{}'s germinationTime is {}".format(PlantName, plant["germinationTime"]))
# DO SOMETHING

Creating unique comboboxes in tabs of a notebook using tkinter and python 3

I am trying to create a program which will hold the sweep values of an RF directional coupler.
I have 5 couplers and each coupler has 4 forward ports.
I have managed to get the 5 couplers on seperate tabs of a tk notebook and loading the stored data from excel for a single port. the problem comes when i want to make the UI reload data for a different port using a combobox.
The code i have at the moment is:
def __init__(self, parent, controller):
tk.Frame.__init__(self, parent)
Notebook_Tabbed = ttk.Notebook(self)
VHF_1, VHF_2 = ttk.Frame(Notebook_Tabbed), ttk.Frame(Notebook_Tabbed)
Notebook_Tabbed.grid(column = 2, row = 2, columnspan = 5, sticky = "nwes")
Notebook_Tabbed.add(VHF_1, text = ' VHF 1 '), Notebook_Tabbed.add(VHF_2, text = ' VHF 2 ')
# Dictionaries to hold each couplers variables
vhf_1_dict = {1 : [2, 7, 4, 5], 3 : [2, 7, 9, 10]}
vhf_2_dict = {1 : [9, 14, 4, 5], 3 : [9, 14, 9, 10]}
# Dictionary to hold above dictionaries
var_dict = {VHF_1 : vhf_1_dict, VHF_2 : vhf_2_dict}
def populate_table_tab(self, tab, Row_Start, Row_End, Column_Start,
Column_End, Coupler_Type, port):
print(tab, Row_Start, Row_End, Column_Start,
Column_End, Coupler_Type, port)
# Load data for chosen port
def change_port(self, tab, port):
populate_table_tab(self, tab, var_dict[tab][port][0], var_dict[tab][port][1],
var_dict[tab][port][2], var_dict[tab][port][3], 'VHF', port)
# Generate comboboxes for each tab of the notebook
ports = StringVar()
ports.set(1)
def generate_port_selections(self, tab, row_num):
port_selection = ttk.Combobox(tab, textvariable = ports, state = 'readonly',
justify = "center")
port_selection['values'] = (1, 3)
port_selection.grid(column = 2, row = row_num, sticky = "nwes")
port_selection.bind('<<ComboboxSelected>>',
lambda _: change_port(self, tab, int(ports.get())))
# _ denotes a throwaway argument required to make lambda work with a combobox event
# Generate port selection comboboxes on each tab
for tab, row_num in {VHF_1 : 4, VHF_2 : 11}.items():
generate_port_selections(self, tab, row_num)
This code does work but it changes the combobox on all of the other tabs at the same time, without changing the data loaded from excel.
I have tried: ('<<ComboboxSelected>>', lambda i = i: change_port(self, tab, port_selection_list[i].get())) but TypeError: list indices must be integers or slices, not Event occurs.
Any help would be greatly appreciated, also i posted this in a rush as i'm working and busy so if i have missed out any necessary code / information please ask.
Edit: Formatted code to be readable by making single lines into multiple.
Edit 2: Simplified code.
Edit 3: You should now be able to run the code within a blank container and frame from this code.
You have problem because you create only one StringVar (ports = StringVar()) and you use it with all Combobox( textvariable=ports ) so they share the same information.
You have to use ports = StringVar() inside generate_port_selections() to create new unique ports for new Combobox.

Python3.4 Multiprocessing

I am trying to run a multiprocessing pool to access multiple urls and pull data with a specific function simultaneously. Unfortunately I keep getting duplicate output for eachticker in the ticker_list. I want a single line of output for eachTicker
code
ticker_list = []
with open('/home/a73nk-xce/Documents/Python/SharkFin/SP500_Fin/SP_StockChar/ticker.csv', 'r', newline='') as csvfile:
spamreader = csv.reader(csvfile)
for rows in spamreader:
pass
for eachTicker in rows:
ticker_list.append(eachTicker)
if __name__ == '__main__':
pool = mp.Pool(20)
pool.map(IncData, ticker_list)
pool.terminate()
OUTPUT
[28105000, 16863000, 11242000, 0, 8355000, 87000, 0, 0, 2800000, -15000, 2785000, 395000, 2390000, 864000, 0, 1509000, 0, 0, 0, 0, 1526000, 0, 1526000]
[1262006, 829648, 432358, 0, 252384, 0, 0, 0, 179974, -2082, 177892, 11392, 166500, 45959, 0, 120541, -2171, 0, 0, 0, 118370, 0, 118370]
[6981000, 3388000, 3593000, 719000, 2043000, 0, 0, 0, 831000, -72000, 759000, 113000, 646000, 142000, 0, 504000, 0, 0, 0, 0, 504000, 0, 504000]
[6981000, 3388000, 3593000, 719000, 2043000, 0, 0, 0, 831000, -72000, 759000, 113000, 646000, 142000, 0, 504000, 0, 0, 0, 0, 504000, 0, 504000]
[69269000, 0, 69269000, 0, 55852000, 20058000, 6666000, 0, 13794000, 570000, 28054000, 13690000, 14364000, 686400
As you can see the output ablove as duplicates in it and it does that while running the entire program
Not sure I understand the problem correctly as the provided information are incomplete.
If IncData is writing its results on a single file, then the issue is due to the concurrent access of the worker processes to that file. If a file is written by more than one process at the same time, the writes will overlap resulting in the corruption of the file. This might be the reason for what you call "duplicated lines".
The best approach would be letting the IncData callable return the result output to the parent process instead of writing itself on the file. The parent process would then aggregate the results from the workers and write them sequentially on a file.
Something like:
if __name__ == '__main__':
with mp.Pool(20) as pool:
results_list = pool.map(IncData, ticker_list)
with open('/path/to/your/file.csv', 'w') as csvfile:
csvfile.writelines(results_list)

Resources