Use a gearman server, registered a function 'stop_xxxxx', then do fatigue testing (cycle request 1000000 times).
At first, the response speed is normal, but by about 1000 times Request timed out.
I'm using gearman 1.1.12 and gearman-python
When timeouts are, gearman server logs are:
DEBUG 2014-07-09 02:26:34.899149 [proc] Received submission, function: A_stop_xxxxxxxx unique: 98588d0149f0a2deb32919e36e395a8a with 2 arguments -> libgearman-server/server.cc: 248
DEBUG 2014-07-09 02:26:34.899166 [proc] Comparing queue 1 to limit 0 for priority 1 -> libgearman-server/job.cc: 175
DEBUG 2014-07-09 02:26:34.899281 [proc] JOB H: orainxiong-vm: 1734: 2878034076 -> libgearman-server/job.cc: 240
DEBUG 2014-07-09 02:26:34.899542 [12] Received RUN wakeup event -> libgearman-server/gearmand_thread.cc: 624
NOTICE 2014-07-09 02:26:34.899573 [proc] accepted, A_stop_one_perf_collect, 98588d0149f0a2deb32919e36e395a8a, 0 -> libgearman-server/server.cc: 317
DEBUG 2014-07-09 02:26:34.899631 [12] send () 32 bytes to peer -> libgearman-server/io.cc: 403
DEBUG 2014-07-09 02:26:34.899646 [12] Sent JOB_CREATED -> libgearman-server/thread.cc: 356
**no more output**
Did not seem to send the request to the worker, how to locate the problem?
My boot parameters are:
gearmand - job-retries = 3 - worker-wakeup = 1 - round-robin - verbose DEBUG - queue-type = MySQL - mysql-host = 127.0.0.1 - mysql-port = 3306 - mysql -user = xxx - mysql-password = xxx - mysql-db = xxxx - mysql-table = job_queue-p 4371-t 16-lstderr
Related
For LivyOperator we set the following parameters:
polling_interval=60
retries_num_timeout=100
We set it up according to this documentation: https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/stable/_api/airflow/providers/apache/livy/operators/livy/index.html
But, in this configuration after 100 * 60 seconds = 6000 seconds = 1 hour 40 minutes Livy-session is interrupted, operator becomes failed, loading is interrupted. Is there any way to resove such inconsistency on Airflow/Livy side?
in my local cluster (4 Raspberry PIs) i try to configure a rgw gateway. Unfortunately the services disappears automatically after 2 minutes.
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host OSD1 and default port 7480
cephuser#admin:~/mycluster $ ceph -s
cluster:
id: 745d44c2-86dd-4b2f-9c9c-ab50160ea353
health: HEALTH_WARN
too few PGs per OSD (24 < min 30)
services:
mon: 1 daemons, quorum admin
mgr: admin(active)
osd: 4 osds: 4 up, 4 in
rgw: 1 daemon active
data:
pools: 4 pools, 32 pgs
objects: 80 objects, 1.09KiB
usage: 4.01GiB used, 93.6GiB / 97.6GiB avail
pgs: 32 active+clean
io:
client: 5.83KiB/s rd, 0B/s wr, 7op/s rd, 1op/s wr
After one minute the service(rgw: 1 daemon active) is no longer visible:
cephuser#admin:~/mycluster $ ceph -s
cluster:
id: 745d44c2-86dd-4b2f-9c9c-ab50160ea353
health: HEALTH_WARN
too few PGs per OSD (24 < min 30)
services:
mon: 1 daemons, quorum admin
mgr: admin(active)
osd: 4 osds: 4 up, 4 in
data:
pools: 4 pools, 32 pgs
objects: 80 objects, 1.09KiB
usage: 4.01GiB used, 93.6GiB / 97.6GiB avail
pgs: 32 active+clean
Many thanks for the help
Solution:
On the gateway node, open the Ceph configuration file in the /etc/ceph/ directory.
Find an RGW client section similar to the example:
[client.rgw.gateway-node1]
host = gateway-node1
keyring = /var/lib/ceph/radosgw/ceph-rgw.gateway-node1/keyring
log file = /var/log/ceph/ceph-rgw-gateway-node1.log
rgw frontends = civetweb port=192.168.178.50:8080 num_threads=100
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_red_hat_enterprise_linux/index
While testing a simple node server (written with Hapi.js):
'use strict';
var Hapi = require("hapi");
var count = 0;
const server = Hapi.server({
port: 3000,
host: 'localhost'
});
server.route({
method: 'GET',
path: '/test',
handler: (request, h) => {
count ++;
console.log(count);
return count;
}
});
const init = async () => {
await server.start();
};
process.on('unhandledRejection', (err) => {
process.exit(1);
});
init();
start the server:
node ./server.js
run the Apache ab tool:
/usr/bin/ab -n 200 -c 30 localhost:3000/test
Env details:
OS: CentOS release 6.9
Node: v10.14.1
Hapi.js: 17.8.1
I found unexpected results in case of multiple concurrent requests (-c 30): the request handler function has been called more times than the number of requests to be performed (-n 200).
Ab output example:
Benchmarking localhost (be patient)
Server Software:
Server Hostname: localhost
Server Port: 3000
Document Path: /test
Document Length: 29 bytes
Concurrency Level: 30
Time taken for tests: 0.137 seconds
Complete requests: 200
Failed requests: 0
Write errors: 0
Total transferred: 36081 bytes
HTML transferred: 6119 bytes
Requests per second: 1459.44 [#/sec] (mean)
Time per request: 20.556 [ms] (mean)
Time per request: 0.685 [ms] (mean, across all concurrent requests)
Transfer rate: 257.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 0
Processing: 15 17 1.5 16 20
Waiting: 2 9 3.9 9 18
Total: 15 17 1.5 16 21
Percentage of the requests served within a certain time (ms)
50% 16
66% 16
75% 17
80% 18
90% 20
95% 20
98% 21
99% 21
100% 21 (longest request)
And the node server print out 211 log lines. During various tests the mismatch is variable but always present:
-n 1000 -c 1 -> 1000 log
-n 1000 -c 2 -> ~1000 logs
-n 1000 -c 10 -> ~1001 logs
-n 1000 -c 70 -> ~1008 logs
-n 1000 -c 1000 -> ~1020 logs
It seems that as concurrency increases, the mismatch increases.
I couldn't figure out whether the ab tool performs more http requests or the node server responds more times than necessary.
Could you please help?
Its very strange and I don't get the same results as you on my machine. I would be very surprised if it was ab that was issuing different numbers of actual requests.
Things i would try:
Write a simple server using express rather than hapi. If the issue still occurs you at least know its not a problem with hapi.
Intercept the network calls using fiddler
ab -X localhost:8888 -n 100 -c 30 http://127.0.0.1:3000/test will use the fiddler proxy which will then let you see the actual calls across the network interface. more details
wireshark if you need more power and your feeling brave (I'd only use it if fiddler has let you down)
If after all these you are still finding an issue then it has been narrowed down to an issue with node, I'm not sure what else it could be. Try using node 8 rather than 10.
Using the Fiddler proxy I found that AB tool runs more times than the number of requests to be performed (example: -n 200).
By running a series of consecutive tests:
# 11 consecutive times
/usr/bin/ab -n 200 -c 30 -X localhost:8888 http://localhost:3000/test
Both the proxy and the node server report a total of 2209 requests. It looks like that AB is less imprecise with the proxy in the middle, but still imprecise.
In general, and more important, I never found mismatches between the requests passed through the proxy and the requests received by the node server.
Thanks!
I am trying to run the demo for UITest on iOS platform here.
I can not run the UI test, even though building the project without a problem.
This is the error information:
SetUp : Calabash.XDB.Core.Exceptions.DeviceAgentException : Failed to launch simulator
ExitCode: 134
2016-11-23 12:00:01.767 iOSDeviceManager[36474:1504101] CoreSimulator: Loading from /Applications/Xcode 8.2-beta.app/Contents/Developer/Library/PrivateFrameworks/CoreSimulator.framework
2016-11-23 12:00:01.772 iOSDeviceManager[36474:1504101] CoreSimulator: Successfully loaded
2016-11-23 12:00:01.773 iOSDeviceManager[36474:1504101] CoreSimulator: SimDevice has correct path of /Applications/Xcode 8.2-beta.app
2016-11-23 12:00:01.773 iOSDeviceManager[36474:1504101] SimulatorKit: Loading from /Applications/Xcode 8.2-beta.app/Contents/Developer/Library/PrivateFrameworks/SimulatorKit.framework
2016-11-23 12:00:01.795 iOSDeviceManager[36474:1504101] SimulatorKit: Successfully loaded
2016-11-23 12:00:01.795 iOSDeviceManager[36474:1504101] SimulatorKit: SimDeviceFramebufferService has correct path of /Applications/Xcode 8.2-beta.app
2016-11-23 12:00:01.796 iOSDeviceManager[36474:1504101] Loaded All Private Frameworks [CoreSimulator, SimulatorKit]
2016-11-23 12:00:01.883 iOSDeviceManager[36474:1504101] Completed Pool Preconditons
2016-11-23 12:00:01.939 iOSDeviceManager[36474:1504101] *** Assertion failure in -[FBSimulatorConfiguration initWithNamedDevice:os:auxillaryDirectory:], /Users/chrisf/FBSimulatorControl/FBSimulatorControl/Configuration/FBSimulatorConfiguration.m:32
2016-11-23 12:00:02.056 iOSDeviceManager[36474:1504101] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid parameter not satisfying: os'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff832256fb __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff97a2ca2a objc_exception_throw + 48
2 CoreFoundation 0x00007fff8322a252 +[NSException raise:format:arguments:] + 98
3 Foundation 0x00007fff84c7d390 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195
4 FBSimulatorControl 0x000000010d2cef3f -[FBSimulatorConfiguration initWithNamedDevice:os:auxillaryDirectory:] + 444
5 FBSimulatorControl 0x000000010d2cf0a3 __48+[FBSimulatorConfiguration defaultConfiguration]_block_invoke + 108
6 libdispatch.dylib 0x00007fff982d3128 _dispatch_client_callout + 8
7 libdispatch.dylib 0x00007fff982d30e5 dispatch_once_f + 38
8 FBSimulatorControl 0x000000010d2cf034 +[FBSimulatorConfiguration defaultConfiguration] + 45
9 FBSimulatorControl 0x000000010d2fc253 +[FBSimulatorConfiguration(CoreSimulator) inferSimulatorConfigurationFromDevice:error:] + 347
10 FBSimulatorControl 0x000000010d2f657f +[FBSimulator fromSimDevice:configuration:launchdSimProcess:containerApplicationProcess:set:] + 178
11 FBSimulatorControl 0x000000010d2f8157 -[FBSimulatorInflationStrategy inflateSimulators:availableDevices:previouslyIdentifiedContainerApplications:] + 810
12 FBSimulatorControl 0x000000010d2f7db8 -[FBSimulatorInflationStrategy inflateFromDevices:exitingSimulators:] + 891
13 FBSimulatorControl 0x000000010d2d615d -[FBSimulatorSet allSimulators] + 129
14 FBSimulatorControl 0x000000010d2d45d5 -[FBSimulatorSet query:] + 108
15 iOSDeviceManager 0x000000010d1609c1 +[Simulator simulatorWithDeviceID:] + 238
16 iOSDeviceManager 0x000000010d1614f5 +[Simulator launchSimulator:] + 88
17 iOSDeviceManager 0x000000010d161ec7 +[LaunchSimulatorCommand execute:] + 74
18 iOSDeviceManager 0x000000010d15cd4a +[CLI process:] + 1171
19 iOSDeviceManager 0x000000010d121504 main + 104
20 libdyld.dylib 0x00007fff98309255 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
I am using Xamarin test cloud to run UITest of Xamarin android and xamarin ios project. I have tried both. For Xamarin.ios project, my setup method looks like below,
app = ConfigureApp
.iOS
.PreferIdeSettings()
.StartApp();
As instructed in default UITest, you need to perform below steps.
Edit the AppDelegate class and add the following snippet to the FinishedLaunching method...
#if ENABLE_TEST_CLOUD
Xamarin.Calabash.Start();
#endif
Make sure you are adding "ENABLE_TEST_CLOUD" debug variable in debug configuration of project or remove if, endif (always allow Xamarin.calabash.start().
and post the UiTest to xamarin test cloud from release configuration build.
Just make sure in this case, remove above statement when you are submiiting your app to app store.
In unit test window (of Xamarin studio), add ios app and then run the test with test cloud. Make sure you have xamarin account to run test on test cloud.
Then it asks devices on which test will be running. You can select 1 or many.
I started to build a Spring Integration app, in which the input gateway generates a fixed number (50) of records and then stops generating new records. There are basic filters/routers/transformers in the middle, and the ending service activator and task executor config are as following:
<int:service-activator input-channel="inChannel" output-channel="outChannel" ref="svcProcessor">
<int:poller fixed-rate="100" task-executor="myTaskExecutor"/>
</int:service-activator>
<task:executor id = "myTaskExecutor" pool-size="5" queue-capacity="100"/>
I tried to put some debug info at the begging of the svcProcessor method:
#Qualifier(value="myTaskExecutor")
#Autowired
ThreadPoolTaskExecutor executor;
#ServiceActivator
public Order processOrder(Order order) {
log.debug("---- " + "executor size: " + executor.getActiveCount() +
" q: " + executor.getThreadPoolExecutor().getQueue().size() +
" r: " + executor.getThreadPoolExecutor().getQueue().remainingCapacity()+
" done: " + executor.getThreadPoolExecutor().getCompletedTaskCount() +
" task: " + executor.getThreadPoolExecutor().getTaskCount()
);
//
//process order takes up to 5 seconds.
//
return order;
}
After sometimes the program runs, the log shows the queue has reached over 50, then eventually gets reject exception:
23:38:31.096 DEBUG [myTaskExecutor-2] ---- executor size: 5 q: 44 r: 56 done: 11 task: 60
23:38:31.870 DEBUG [myTaskExecutor-5] ---- executor size: 5 q: 51 r: 49 done: 11 task: 67
23:38:33.600 DEBUG [myTaskExecutor-4] ---- executor size: 5 q: 69 r: 31 done: 11 task: 85
23:32:46.792 DEBUG [myTaskExecutor-1] ---- executor size: 5 q: 72 r: 28 done: 11 task: 88
It looks like the active count and sum of queue size/remaining looks right with the config of 5 and 100, but I am not clear why there are more than 50 records in the queue, and the taskCount is also larger than the limit 50.
Am I looking at the wrong info from the executor and the queue?
Thanks
UPDATE:
(not sure if I should open another question)
I tried the xml version of the cafeDemo from spring-integration (branch SI3.0.x), and used pool provided in the document, but used 100 milliseconds rate and added capacity:
<int:service-activator input-channel="hotDrinks" ref="barista" method="prepareHotDrink" output-channel="preparedDrinks">
<int:poller task-executor="pool" fixed-rate="100"/>
</int:service-activator>
<task:executor id="pool" pool-size="5" queue-capacity="200"/>
After I ran it, it also got rejection exception after around the 20th delivery:
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#6c31732b[Running, pool size = 5, active threads = 5, queued tasks = 200, completed tasks = 0]]
There are only about 32 orders placed until the exception, so I am not sure why queued tasks = 200 and completed task = 0?
THANKS
getTaskCount() This method gives the number of total task assigned to executor since the start. So, it will increase with time.
And other variables are approximate number not exact as per documentation of java.
getCompletedTaskCount()
Returns the approximate total number of tasks that have completed execution.
public int getActiveCount()
Returns the approximate number of threads that are actively executing tasks.
Ideally getTaskCount() and getCompletedTaskCount() will increase linearly with time, as it includes all the previous tasks assigned since start of execution of your code. However, activeCount should be less than 50, but being approximate number it will go beyond 50 sometimes with little margin.
Refer :-
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html