I am using fio for storage benchmarking and fio2gnuplot for plotting graphs, every time I run a test and look into logfiles of iops, second coloumn is always 1 which is iops value and due to this graphs are just a straight line perpendicular to Y axis .Which makes no sense. I tried various iodepths,ioengines but no use.am I using any parameters(options)wrong?
following is my jobfile.
[global]
enter code here
rw=randwrite
size=128m
thread=1
iodepth=2
ioengine=libaio
per_job_logs=0
directory=/home/fio
[job_512]
write_bw_log=logfiles_libaio/fio-test_512
write_iops_log=logfiles_libaio/fio-test_512
write_lat_log=logfiles_libaio/fio-test_512
bs=512b
and this is the logfile
1, 1, 0, 512
2, 1, 1, 512
18, 1, 1, 512
19, 1, 0, 512
31, 1, 1, 512
53, 1, 1, 512
55, 1, 1, 512
56, 1, 0, 512
59, 1, 1, 512
63, 1, 1, 512
According to fio manual(man fio), under "FIO FILE FORMATS", it says:
Fio supports a variety of log file formats, for logging latencies, bandwidth, and IOPS. The logs
share a common format, which looks like this:
time (msec), value, data direction, offset
Time for the log entry is always in milliseconds. The value logged depends on the type of log, it
will be one of the following:
Latency log
Value is in latency in usecs
Bandwidth log
Value is in KB/sec
IOPS log
Value is in IOPS
Data direction is one of the following:
0 IO is a READ
1 IO is a WRITE
2 IO is a TRIM
However, I think the 'offset' should be 'IO size'.
So, in your bandwidth case, it's:
timestamp(ms), bandwidth(KB/sec), R/W, size
Related
I am writing a LoadTestShape Class. I want to be able to set number of transaction per minute. If I try to set it by specifying user_count it doesn't work because the RPS will vary cpu to cpu .
If say I want to set 1000 transaction per minute, how can we achieve that in locust?
The only way to "achieve" this in Locust is implementing Pacing, the logic is:
You know how much time one iteration takes for one user
If user manages to finish faster - you need to introduce some "think time" so your thread "sleeps" till the desired time
If iteration is slower - no sleep is required.
This answer shows how you can create a custom load shape
Alternatively you can consider switching to a load testing tool which provides such functionality, i.e. Apache JMeter with Constant Throughput Timer
for this timing that's enough to add a LoadTestShape class to locustfile.py like the below code which I use for one of my tests you can change the number of users or other parameters as you want(I write a full docstring that describes each parameter in that):
class StagesShape(LoadTestShape):
"""
A simply load test shape class that has different user and spawn_rate at
different stages.
Keyword arguments:
stages -- A list of dicts, each representing a stage with the following keys:
duration -- When this many seconds pass the test is advanced to the next stage
users -- Total user count
spawn_rate -- Number of users to start/stop per second
stop -- A boolean that can stop that test at a specific stage
stop_at_end -- Can be set to stop once all stages have run.
"""
stages = [
{"duration": 60, "users": 3, "spawn_rate": 0.05},
{"duration": 60, "users": 6, "spawn_rate": 0.05},
{"duration": 60, "users": 9, "spawn_rate": 0.05},
{"duration": 60, "users": 12, "spawn_rate": 0.05},
{"duration": 60, "users": 15, "spawn_rate": 0.05},
{"duration": 60, "users": 18, "spawn_rate": 0.05},
{"duration": 60, "users": 21, "spawn_rate": 0.05},
{"duration": 60, "users": 24, "spawn_rate": 0.05},
{"duration": 60, "users": 27, "spawn_rate": 0.05},
{"duration": 60, "users": 30, "spawn_rate": 0.05},
]
def tick(self):
run_time = self.get_run_time()
for stage in self.stages:
if run_time < stage["duration"]:
tick_data = (stage["users"], stage["spawn_rate"])
return tick_data
return None
I ended up using constant_throughput to control the number of requests per second.
For different time of the day I'd use a different throughput value.
Simply set the wait_time = constant_throughput(0.1). Based on RPS you want you can either set the value low for less number of requests and more for more RPS.
I'm trying to allocate a set of image buffers in shared memory using multiprocessing.RawArray. It works fine for smaller numbers of images. However, when I get to a certain number of buffers, I get a OSError indicating that I've run out of memory.
Obvious question, am I actually out of memory? By my count, the buffers I'm trying to allocate should be about 1 GB of memory, and according to the Windows Task Manager, I have about 20 GB free. I don't see how I could actually be out of memory!
Am I hitting some kind of artificial memory consumption limit that I can increase? If not, why is this happening, and how can I get around this?
I'm using Windows 10, Python 3.7, 64 bit architecture, 32 GB RAM total.
Here's a minimal reproducible example:
import multiprocessing as mp
import ctypes
imageDataType = ctypes.c_uint8
imageDataSize = 1024*1280*3 # 3,932,160 bytes
maxBufferSize = 300
buffers = []
for k in range(maxBufferSize):
print("Creating buffer #", k)
buffers.append(mp.RawArray(imageDataType, imageDataSize))
Output:
Creating buffer # 0
Creating buffer # 1
Creating buffer # 2
Creating buffer # 3
Creating buffer # 4
Creating buffer # 5
...etc...
Creating buffer # 278
Creating buffer # 279
Creating buffer # 280
Traceback (most recent call last):
File ".\Cruft\memoryErrorTest.py", line 10, in <module>
buffers.append(mp.RawArray(imageDataType, imageDataSize))
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 129, in RawArray
return RawArray(typecode_or_type, size_or_initializer)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py", line 61, in RawArray
obj = _new_value(type_)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\sharedctypes.py", line 41, in _new_value
wrapper = heap.BufferWrapper(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", line 263, in __init__
block = BufferWrapper._heap.malloc(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", line 242, in malloc
(arena, start, stop) = self._malloc(size)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", line 134, in _malloc
arena = Arena(length)
File "C:\Users\Brian Kardon\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\heap.py", line 38, in __init__
buf = mmap.mmap(-1, size, tagname=name)
OSError: [WinError 8] Not enough memory resources are available to process this command
Ok, the folks over at Python bug tracker figured this out for me. For posterity:
I was using 32-bit Python, which is limited to a memory address space of 4 GB, much less than my total available system memory. Apparently enough of that space was taken up by other stuff that the interpreter couldn't find a large enough contiguous block for all my RawArrays.
The error does not occur when using 64-bit Python, so that seems to be the easiest solution.
Is there a way to view exported coloured logs as in console (with colour)?
My program uses colour coding for errors, warnings, etc. If I redirect output of my program to file.log, I get records like:
[32m[1m(INF)[0m /environment/converter: State map: [ 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 18 ][0m
[32m[1m(INF)[0m /environment/converter: Action map: [ -1, -1, -1, 0, 1, 2, 3, 4, 5 ][0m
[32m[1m(INF)[0m PyEnv: Observation dims: 15[0m
[32m[1m(INF)[0m PyEnv: Action dims: 6[0m
Random seed None
[32m[1m(INF)[0m GRL seed 1428[0m
Now I want to see file.log, but without colour codes or even better with colours preserved.
I have tried nano, vi and gedit but they all do not do what I want.
Here are some ideas if you're still wondering for a good way to do this.
A text editor won't be able to read the color codes, as those are Bash specific. However, cat works because it can interpret the color codes. However, the way you're storing the colors isn't formatted correctly in the first place.
I'm not sure how you are writing into your log file, but I echoed your example and redirected the output into a file. I didn't see colors either. The reason is that you have to escape every square bracket for the codes to be interpreted by Bash.
So the way that I escaped the brackets is by manually adding \e before every square bracket and using echo -e to evaluate the escape characters:
`echo -e "\e[32m\e[1m(INF)\e[0m /environment/converter: State map: [ 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 18 ]\e[0m`
`\e[32m\e[1m(INF)\e[0m /environment/converter: Action map: [ -1, -1, -1, 0, 1, 2, 3, 4, 5 ]\e[0m`
`\e[32m\e[1m(INF)\e[0m PyEnv: Observation dims: 15\e[0m`
`\e[32m\e[1m(INF)\e[0m PyEnv: Action dims: 6\e[0m`
`Random seed None`
`\e[32m\e[1m(INF)\e[0m GRL seed 1428\e[0m " > example.txt`
Now when I open the file with cat I see correctly colored text:
The color codes are now stored and escaped correctly in the .txt file:
^[[32m[1m(INF)[0m GRL seed 1428^[[0m
vs
[32m[1m(INF)[0m GRL seed 1428[0m
An alternative solution is to use a package like Ansi Html Adapter (Aha).
It will produce HTML instead of plain text. Then you can open the output in your browser, and it would have the right color coding.
However, you would meet with the same problem. If the color codes aren't escaped correctly, the output is not going to be colored.
I ran the following command to convert the correctly formatted text to HTML:
aha -f example.txt > example.html
Here's the result in the browser:
You can find more info on how to use colors in bash in this Bash Tips article.
i am facing high epoll_wait time when i used this command to check the page execution.
strace -o output.txt -f -r -s4096 -p 21605
the outout was big txt file but i was interested in this piece of data and unable to find out the reason of this.
the data is for a dynamic php page and its not too heavy so i dont expect this much long response time
21605 0.000043 semop(38830083, {{0, -1, SEM_UNDO}}, 1) = 0
21605 0.611909 epoll_wait(30, {{EPOLLIN, {u32=25218632, u64=25218632}}}, 4, 10000) = 1
21605 0.103429 accept4(3, {sa_family=AF_INET, sin_port=htons(56826), sin_addr=inet_addr("ip address")}, [16], SOCK_CLOEXEC) = 33
21605 0.000059 semop(38830083, {{0, 1, SEM_UNDO}}, 1) = 0
i am really unhappy with this 0.611909 time in epoll and 0.103429 accept4 . any help to improve these performance will be great
Speed up the other end. This is time spent waiting to accept a connection or receive data from the other side.
I need to import an excel document into mathematica which has 2000 compounds in it, with each compound have 6 numerical constants assigned to it. The end goal is to type a compound name into mathematica and have the 6 numerical constants be outputted. So far my code is:
t = Import["Titles.txt.", {"Text", "Lines"}] (imports compound names)
n = Import["NA.txt.", "List"] (imports the 6 values for each compound)
n[[2]] (outputs the second compounds 6 values)
Instead of n[[#]] i would like to know how to type in a compound from the imported compound names and have the 6 values be outputted .
I'm not sure if I understand your question - you have two text files, rather than an Excel file, for example, and it's not clear what the data looks like. But there are probably plenty of ways to do this. Here's a suggestion (it might not be the best way):
Let's assume that you've got all your data into a table (a list of lists):
pt = {
{"Hydrogen", "H", 1, 1.0079, -259, -253, 0.09, 0.14, 1776, 1, 13.5984},
{"Helium", "He", 2, 4.0026, -272, -269, 0, 0, 1895, 18, 24.5874},
{"Lithium" , "Li", 3, 6.941, 180, 1347, 0.53, 0, 1817, 1, 5.3917}
}
To find the information associated with a particular string:
Cases[pt, {"Helium", rest__} -> rest]
{"He", 2, 4.0026, -272, -269, 0, 0, 1895, 18, 24.5874}
where the pattern rest__ holds everything that was found after "Helium".
To look for the second item:
Cases[pt, {_, "Li", rest__} -> rest]
{2, 4.0026, -272, -269, 0, 0, 1895, 18, 24.5874}
If you add more information to the patterns, you have more flexibility in how you choose elements from the table:
Cases[pt, {name_, symbol_, aNumber_, aWeight_, mp_, bp_, density_,
crust_, discovered_, rest__}
/; discovered > 1850 -> {name, symbol, discovered}]
{{"Helium", "He", 1895}}
For something interactive, you could knock up a Manipulate:
elements = pt[[All, 1]];
headings = {"symbol", "aNumber", "aWeight", "mp", "bp", "density", "crust", "discovered", "group", "ion"};
Manipulate[
Column[{
elements[[x]],
TableForm[{
headings, Cases[pt, {elements[[x]], rest__} -> rest]}]}],
{x, 1, Length[elements], 1}]