I'm running out of memory (I have 16GB) in a script I am running. Here is a little background:
I am generating an array that is ~150k rows by 8 column as the basis for my calculations.
Then, for each "row" of values (nothing is ever written to the worksheet), I am making a LOT of calculations, which I am storing in various data structures (I have 3 main structures at this point). One of these structures is composed of the following:
A Variant Array with 150k Dictionaries. Each Dictionary has ~4 Key-Item pairs. Each item in each dictionary is another dictionary, containing exactly 9 key-item pairs. All keys are Strings and all items are Doubles.
Essentially, this is the same thing as:
Dim Array(1 to 150000, 1 to 4, 1 to 9) as Double
Except I want to be able to reference the values with text strings -- hence the dictionaries.
An example would be
Value = Array(2401)("Key1")("Key2")
I wouldn't think this would be too much for VBA to handle -- we're talking 150,000 * 4 * 9 individual doubles = 5.4M doubles for each of the 3 main data structures. I don't have that much experience with programming and memory management, but sure that wouldn't consume 16GB of memory!
As such, I'm thinking there's a problem in how I'm generating these data structures that is causing a memory leak somewhere.
Essentially the loop looks like this
Dim TempDict1 as Dictionary
Dim TempDict2 as Dictionary
Dim FinalArray() as Variant
ReDim FinalArray(1 to 150000) as Variant
Dim Calculations as Double
For i = 1 to 150000
Set TempDict = NewDictionary
for j = 1 to 5
Set TempDict2 = NewDictionary
for k = 1 to 9
Calculations = 2*2
TempDict2.Add Key:= KeyK, Item:= Calculations
Next k
TempDict.Add Key:= KeyJ, Item:= TempDict2
' TempDict2.RemoveAll (This causes an error)
next j
Set FinalArray(i) = TempDict
' TempDict.RemoveAll (This causes an error)
Next i
What am I doing wrong here? I've tried destroying the temporary dictionaries after adding them to the parent item, but that actually gives me a type error.
UPDATE: I've tried setting the temporary dictionaries to nothing instead of removing all. This doesn't cause any errors, but it still consumes a lot of memory. With 37k iterations, it consumes 8.4GB of memory.
I'm not sure if this is some sort of bug or memory leak, but the fact is that this is an extremely memory-inefficient way to store data.
As such, I've edited the code to use 3 dimensional arrays with separate dictionaries to associate the indices of arrays with text strings, which is working great.
I was able to store 8 different 3D arrays (150k by 8 by 13) and 7 different 2D arrays (150k 13), both of doubles. Overhead doesn't seem to be too high. Given a size of 8 bytes per double, theoretical memory usage is 1.11GB, which isn't too bad.
Related
I'm a total beginner with CPLEX and OPL, so maybe you can help me with the coding of a mixed integer programming model.
In my case: I have an optimization function including a parameter transportation cost which are specific for the starting point (Hubs h), the destination (DCs i), the transported good (Products k) and the mode of transportation (TransportOptions r) used.
I wrote it like this:
float transportC_Hub_DC[Hubs][DCs][Products][TransportOptions] = ...;
//transport cost of one unit of gook k vor starting point h to destination i using transportation option r
I would like to fill this array with its multiple dimensions from an excel spread sheet. At the moment my spreadsheet has the four indexes in separate columns and the specifice transportation cost in another column. It looks like this:
Excel Datasheet
My problem is that I do not know how to make the programme understand how the transportation cost data are ordered. How does the programme know that in the first cell of the column "transportation cost" is the cost for the specific combination of the different indexes? So how do I tell the programme that I used h=1, i=1, k=1, r=1 in the first cell and h=1, i=1, k=1, r=2 in the second cell and not h=1, i=1, k=2, r=1 in the second cell? What do I have to write in the model or the data file in CPLEX to make this clear?
See technote http://www-01.ibm.com/support/docview.wss?rs=0&context=SSCMS55&uid=swg21401340&loc=en_US&cs=utf-8&cc=us&lang=all
The idea is to read a tuple set and then turn your tuple set into a 4D array.
I'm reading an excelfile in matlab with
[NUM,TXT,RAW]=xlsread(DATENEXCEL,sSheet_Data);
In the excelfile are different datamatrices in different sheets in the following form
Date Firm1 Firm2 Firm3 ...
1.1.16 12 12 12
... ... ... ...
Currently I'm handling the pure data with the NUM object and the header row with the TXT object. My first issue is how to combine the header row with the data rows. Looping does not work, since I predefine the data matrix with
daten=zeros([length(sDatesequence) size(RAW,2)]);
because I want to be able to add more data from different sources to that object. Predefining with zeros, however, leads Matlab to expect doubles and not characters. Converting the cell array TXT with cell2mat delivers unsatisfying results:
cell2mat(TXT(1,:))=Firm1Firm2Firm3...
hence only a long string vector.
Question: Is there another way to combine character vectors and double matrices?
Regards,
Richard
You can combine them in a cell array.
c{1,1} = 'Firm1';
c{1,2} = datavector;
c{2,1} = 'Firm2';
c{2,2} = datavector;
But as far as I know it is not possible to add text headers to a numerical matrix, unless you do something with typcasting. But I would not recommend that.
d(1:8)='Firm1 '; %must have exactly eight characters (a double has a length of 8 bytes)
y = typecast(uint8(d),'double') %now you have a number that would fit in a matrix of doubles
x=char(typecast(y,'uint8')) %now it's converted back to text
I'm new here, and I thought I would ask a question that certainly isn't found in the Microsoft Help Center and that I haven't been able to find a solution to either.
I am trying to calculate probability on things, and for the most part, Excel is very helpful in it. I'm running into problems though as I add additional variables.
My sheet currently is comparing dice rolls of 4 8-sided dice. 2 dice have certain symbols and the two other dice have different symbols. Some symbols negate each other, and in the end I come to a damage output number. When comparing 2 or 3 dice, the possible combinations are limited. 3 dice having 512 possibilities. With 4 now, there are 4096 possibilities and it's only going to get higher. This is why I need what I'm asking for.
Is there a way for a cell to understand is current position in reference to the block of cells it's currently in?
For example: I'm calculating a reroll possibility, but it will only happen half the time, meaning there are 12 possibilities of a single die with reroll option. So the current possibility table I'm developing is going to be 96 separate tables of 96 possible outcomes each. Table 1/1 is going to compare the first row of the 2 dice Attack roll table vs. the first row of the 2 dice Defense roll table. Row 1 Column 1 of this table is going to give the outcome of R1C1 of Attack table vs. R1C1 of Defense table. R1C2 of the table is going to give the outcome of R1C1 of Attack table vs. R1C2 of Defense table. R2C1 of this table will give the outcome of R1C2 of Attack table vs. R1C1 of Defense table, etc...
I know how to do the referencing to the tables, so I've made it so once I build one table, I can copy and paste it to build the other 96. But as I compare more dice rolls, this will quickly become too cumbersome to handle. If there is a way for a cell to understand where it's relative position is in a given block of cells (i.e. R2C1 of my example table understands that it is R2C1), it would cut down on my load immensely, and allow for me to continue building these probability tables so o can better understand tradeoffs in certain areas.
Any help is greatly appreciated.
Here's an Excel UDF I wrote for basic dice calculation %. It may not work directly for your example with negative/conditional outcomes, but it does have flexibility for testing more than one dice and the number of sides dice, so it might inspire you with some ideas. As previous comments suggested, if you gave exact parameters, you could probably get a specific example. My example returns a %. It currently only measures the probability of a single outcome, but you could do more than one formula per cell =DiceRollOdds(3,2)+DiceRollOdds(4,2) (to measure the probability of 3 and 4), or you can modify the code get something more specific.
Function DiceRollOdds(OutcomeToCheck As Integer, NumberOfDice As Integer, Optional SidesOnDice As Integer) As Double
Dim SuccessResult As Integer, FailedResult As Integer, SingleDice As Integer, RollResult As Integer
If SidesOnDice = 0 Then
SidesOnDice = 6
End If
Dim Rolls As Integer
For Rolls = 1 To (SidesOnDice ^ NumberOfDice)
RollResult = 0
For SingleDice = 0 To NumberOfDice - 1
RollResult = Int(Rolls / SidesOnDice ^ SingleDice) Mod SidesOnDice + 1 + RollResult
Next SingleDice
If RollResult = OutcomeToCheck Then
SuccessResult = SuccessResult + 1
Else
FailedResult = FailedResult + 1
End If
Next Rolls
DiceRollOdds = SuccessResult / (FailedResult + SuccessResult)
End Function
It is easy creating an array in MS Excel VBA, be it 1-dimensional or multi-dimensional, and set it to type Integer or String. Is it possible to do complex arrays however along these lines (this is not true code but represents what I mean):
1 to 100 rows
1 to 100 columns
f_Depth: Float
1 to 4 points
l_X: Long
l_Y: Long
The Excel integer-only (or float-only or whatever you choose) makes it very difficult to work with the code.
By "complex array" do you mean array of mixed type? If so, you can use Collection which accepts objects of any type:
Dim Data As New Collection
Data.Add (1)
Data.Add ("Hello world")
Data.Add (3.14)
I was just doing some yicky code and I thought, instead of using three dynamic arrays, as such:
dim x() as string, y() as string, z() as string
It will be nicer to have a 3 dimensional dynamic array. But; the help and my fumbling experiments has not revealed the method of defining them.
This does not work:
dim x()() or dim(,2) or dim(,)
Any ideas anyone?
A dynamic array is declared the same way regardless of the number of dimensions (arrays in LotusScript can have up to 8 dimensions). According to your example I think it is a two dimensional array you want, where the first dimension is limited to three entries.
If you first declare the array as:
Dim x() As String
You can then specify bounds according to the following example:
Redim x( 0 To 2, 0 To 9 ) ' A two dimensional array
And if you need to enlarge the array later (and keep all the values) you can do it like this:
Redim Preserve x( 0 To 2, 0 To 99 )
Please keep in mind that only the bounds of last dimension can be changed once the number of dimensions of the array has been set.
You could use lists instead of arrays.
Dim x list as String
That is fully dynamic and takes a string as index. List can't contain lists, but lists can contain objects, so you might want to do
Public Class ListContainer
Public level2 List as String
End Class
This way you never need to REDIM preserve. A forall loops you savely through a list