Here's a dump of the stats provided my mod_pagespeed from one of my sites.
resource_url_domain_rejections: 6105
rewrite_cached_output_missed_deadline: 4801
rewrite_cached_output_hits: 116004
rewrite_cached_output_misses: 934
resource_404_count: 0
slurp_404_count: 0
total_page_load_ms: 0
page_load_count: 0
resource_fetches_cached: 0
resource_fetch_construct_successes: 45
resource_fetch_construct_failures: 0
num_flushes: 947
total_fetch_count: 0
total_rewrite_count: 0
cache_time_us: 572878
cache_hits: 872
cache_misses: 1345
cache_expirations: 242
cache_inserts: 1795
cache_extensions: 50799
not_cacheable: 0
css_file_count_reduction: 0
css_elements: 0
domain_rewrites: 0
google_analytics_page_load_count: 0
google_analytics_rewritten_count: 0
image_inline: 7567
image_rewrite_saved_bytes: 208854
image_rewrites: 34128
image_ongoing_rewrites: 0
image_webp_rewrites: 0
image_rewrites_dropped_due_to_load: 0
image_file_count_reduction: 0
javascript_blocks_minified: 12438
javascript_bytes_saved: 1173778
javascript_minification_failures: 0
javascript_total_blocks: 12439
js_file_count_reduction: 0
converted_meta_tags: 902
url_trims: 54765
url_trim_saved_bytes: 1651244
css_filter_files_minified: 0
css_filter_minified_bytes_saved: 0
css_filter_parse_failures: 2
css_image_rewrites: 0
css_image_cache_extends: 0
css_image_no_rewrite: 0
css_imports_to_links: 0
serf_fetch_request_count: 1412
serf_fetch_bytes_count: 12809245
serf_fetch_time_duration_ms: 28706
serf_fetch_cancel_count: 0
serf_fetch_active_count: 0
serf_fetch_timeout_count: 0
serf_fetch_failure_count: 0
Can someone please explain what all of the stats mean?
There's a lot of stats here. I'm going to just describe a few of them, because this will get long. We should probably add detailed doc. I can follow-up with more answers later if these are useful.
resource_url_domain_rejections: 6105: this means that since your server restarted, mod_pagespeed has found 6105 resources that it's not going rewrite a resource because their domains are not authorized for rewriting with the ModPagespeedDomain directive. This is common & occurs anytime time someone refreshes a page with a twitter, facebook, or google+ widget.
rewrite_cached_output_missed_deadline: 4801: when a resources (e.g. a jpeg image) is optimized, it happens in a background thread, and the result is cached so that subsequent page views referencing the same refresh are fast. To avoid slowing down the first view, however, we use a 10 millisecond timer to avoid slowing down the time-to-first byte. This stat counts how many times that deadline was exceeded, in which case the resource is left unchanged for that view, but the optimization continues in the background & so the cache is written.
rewrite_cached_output_hits: 116004: counts the number of times we served an optimized resource from the cache, thus avoiding the need to re-optimize it.
rewrite_cached_output_misses: 934: counts the number of times we looked up a resource in our cache and it wasn't there, forcing us to rewrite it. Note that we would also rewrite a resource that was in the cache, but whose origin cache expiration-time had expired. E.g. if your images had cache-control:max-age=600 then we would re-fetch them every 10 minutes to see if they've changed. If they have changed we must re-optimize them.
num_flushes: 947: this is the number of times the Apache resource-generator for the HTML (e.g. mod_php or Wordpress) called the Apache function ap_flush(), which causes partial HTML to be flushed all the way through to the user's browser. This is interesting for mod_pagespeed because it can limit the amount of optimization we can do (e.g. we can't combine CSS files whose elements are separated by a Flush).
cache_time_us: 572878 - the total amount of time, in microseconds, spent waiting for mod_pagespeed's HTTP Cache (file + memory) to respond to a lookup request, since the server was started.
I think that's enough for now. Are there specific other statistics you'd like to learn more about?
Most of these were created for us to monitor the health of mod_pagespeed as it's running, and to help diagnose users' issues. I have to admit we haven't used it much for that purpose, but we use them during development.
Related
I have a ZFS pool with the following layout and errors:
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000039ff3d3b114-part2 ONLINE 0 0 0
wwn-0x5000039ff4d3b513-part1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
wwn-0x5000c500a42783bc-part1 ONLINE 0 0 2
wwn-0x5000c500a426d50b-part1 ONLINE 0 0 2
errors: Permanent errors have been detected in the following files:
tank/foo/bar#veryOldSnapshot;corruptFile.qcow2
So it looks as if on two different devices the same record has been corrupted at the same time. The data in question is on this disks since 2019 and the pool is scrubbed every week. How are the chances? IMHO this cannot be a real "bits are flipped due to cosmical radiation or hdd failure" case, because the probability that exactly the same blocks are corrupted on both disks and no other block is are really low.
What else can have caused this? I ran memtestx86 without problems and scrubbing again does not find any other errors. However since the block is used in a long chain of snapshots, even removing the snapshot in question does just make the problem move to the next snapshot.
I run a website with 5 httpd servers(Centos 7) in EC2, type is m3.2xlarge.
The servers are configured with load balancer.
Gradually the server memory keeps going higher in all the instances.
For example:
Memory usage in few seconds after restarting the httpd service:
[centos#ip-10-0-1-77 ~]$ while sleep 1; do free -m; done
total used free shared buff/cache available
Mem: 29741 2700 26732 36 307 26728
Swap: 0 0 0
total used free shared buff/cache available
Mem: 29741 2781 26651 36 307 26647
Swap: 0 0 0
total used free shared buff/cache available
Mem: 29741 2820 26613 36 307 26609
Swap: 0 0 0
[centos#ip-10-0-1-77 ~]$
.
.
.
This is what i see after an hour:
[centos#ip-10-0-1-77 ~]$ free -m
total used free shared buff/cache available
Mem: 29741 29092 363 41 284 346
Swap: 0 0 0
Like above it goes and consumes all the memory(30GB) within an hour.
To avoid this I started using worker mpm configuration.
The following configuration is what I have added at the bottom of /etc/httpd/httpd.conf.
<IfModule mpm_worker_module>
MaxRequestWorkers 2500
MaxSpareThreads 250
MinSpareThreads 75
ServerLimit 100
StartServers 3
ThreadsPerChild 25
</IfModule>
Can Someone help and suggest me the right configuration to utilize the RAM memory properly in all the instances?
A standard Apache process takes up about 12 MB of RAM. If you have 30 GB reserved for Apache you will never reach that with a serverlimit of 100 (=100*12MB=1200MB=1,2GB). So I assume that Apache isn't taking up all that memory.
Is there an application that is involved or a DB? Those can take up larger amounts of RAM.
For your servertuning.conf (or httpd.conf since you put it there):
<IfModule mpm_worker_module>
#max amount of requests one worker handles before it's forced to close, 2,5k seems almost a little low
MaxRequestsperChild 2500
#maximum number of worker threads which are kept spare
#250 seems quite high, but really depends on the traffic you are experiencing, we normally don't use more than 50-75
MaxSpareThreads 250
#minimum number of worker threads which are kept spare
#really high, only useful if you often experience higher bursts of traffic
#otherwise you should be fine with 15-30, maybe 50 if you experience higher fluctuation -> bigger bursts of requests
MinSpareThreads 75
#upper limit on the configurable number of threads per child process
#you have to increase this if you want more than 64 ThreadsPerChild
ThreadLimit 64
#maximum number of simultaneous client connections
#that is really low! --> increase that, definitely! We run on 1000, so about 12GB max
MaxClients 100
#initial number of server processes to start
#3 is really low, if you expected your server to be flodded with requests the second you start it
#maybe turn it up a little to around 20 or even 50 if you receive lots of traffic right after a restart
StartServers 3
#number of worker threads created by each child proces
#25 threads per worker is not tooo much, but at some point the administration of xx threads gets more expensive than creating new ones
#would suggest to leave it at 25 or turn it up to around 40
ThreadsPerChild 25
</IfModule>
Notice that I changed ServerLimit to MaxClients and MaxRequestWorkers to MaxRequestsPerChild, because as far as I know those are the terms used in mpm-worker.
Additionally you can change following variables:
#KeepAlive: Whether or not to allow persistent connections (more than
#one request per connection). Set to "Off" to deactivate.
#if it's on, leave it there
KeepAlive On
#MaxKeepAliveRequests: The maximum number of requests to allow
#during a persistent connection. Set to 0 to allow an unlimited amount.
#We recommend you leave this number high, for maximum performance.
#default=100, but you can turn that up if your sites contain a lot of item (img, css, ...)
#we are using about 20*<average object-count per site> = 600
MaxKeepAliveRequests 600
#KeepAliveTimeout: Number of seconds to wait for the next request from the
#same client on the same connection.
#would recommend to decrease that, otherwise you could become a victim of slow-dos attacks
#default is 15, we are running just fine on 5
KeepAliveTimeout 5
To further prevent slow-dos or piling up of open sessions, you can use mod_reqtimeout:
<IfModule mod_reqtimeout.c>
# allow 10s timeout for the headers and allow 1s more until 20s upon receipt of 1000 bytes.
# almost the same with the body, except that it is tricky to
# limit the request timeout within the body at all - it may take time to generate the body.
# below are the default values
#RequestReadTimeout header=10-20,MinRate=1000 body=20,MinRate=1000
# and this is how I'd set them with today's internet speed
# deduct the according numbers from explanation above...
RequestReadTimeout header=2-5,MinRate=100000 body=5-10,MinRate=1000000
</IfModule>
If that's not enough to help your RAM-issues (if they are really caused by Apache), use the tools of your server's OS accordingly to find out what is taking up all the RAM --> TOOLS
I'd like to lookup a counter of the TCP payload activity (total bytes received) either for a given file descriptor or a given interface. Preferably the given file descriptor, but for the interface would be sufficient. Ideally I'd really like to know about any bytes that have been ack-ed, even ones which I have not read into userspace (yet?).
I've seen the TCP_INFO feature of getsockopt() but none of the fields appear to store "Total bytes received" or "total bytes transmitted (acked, e.g.)" so far as I can tell.
I've also seen the netlink IFLA_STATS+RTNL_TC_BYTES and the SIOCETHTOOL+ETHTOOL_GSTATS ioctl() (rx_bytes field) for the interfaces, and those are great, but I don't think they'll be able to discriminate between the overhead/headers of the other layers and the actual payload bytes.
procfs has /proc/net/tcp but this doesn't seem to contain what I'm looking for either.
Is there any way to get this particular data?
EDIT: promiscuous mode has an unbearable impact on throughput, so I can't leverage anything that uses it. Not to mention that implementing large parts of the IP stack to determine which packets are appropriate is beyond my intended scope for this solution.
The goal is to have an overarching/no-trust/second-guess of what values I store from recvmsg().
The Right Thing™ to do is to keep track of those values correctly, but it would be valuable to have a simple "Hey OS? How many bytes have I really received on this socket?"
One could also use ioctl call with SIOCINQ to get the amount of queued unread data in the receive buffer. Here is usage from the man page: http://man7.org/linux/man-pages/man7/tcp.7.html
int value;
error = ioctl(tcp_socket_fd, SIOCINQ, &value);
For interface TCP stats, we can use " netstat -i -p tcp" to find stats on a per-interface basis.
Do you want this for diagnosis, or for development?
If diagnosis, tcpdump can tell you exactly what's happening on the network, filtered by the port and host details.
If for development, perhaps a bit more information about what you're trying to achieve would help...
ifconfig gives RX and TX totals.
ifconfig gets these details from /proc/net/dev (as you can see via strace ifconfig).
There are also the Send/Receive-Q values given by netstat -t, if that's closer to what you want.
Perhaps the statistics in /proc/net/dev can help. I am not familiar with counting payload versus full packets including headers, so that makes the question harder to answer.
As for statistics on individual file descriptors, I am not aware of any standard means to get that information.
If it's possible to control startup of the programs for which the statistics are needed, it is possible to use an "interceptor" library which implements its own read(), write(), sendto(), and recvfrom() calls, passthrough the calls to the standard C library (or directly to system call), keep counters of the activity, and find a way to publish those values.
In case you don't want to just count total RX/TX per interface (which is already available in ifconfig/iproute2 tools)...
If you look into /proc a bit more, you can get somewhat more information. More specifically /proc/<pid>/net/dev.
Sample output:
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
eth0: 12106810846 8527175 0 15842 0 0 0 682866 198923814 1503063 0 0 0 0 0 0
lo: 270255057 3992930 0 0 0 0 0 0 270255057 3992930 0 0 0 0 0 0
sit0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
If you start looking, the information is coming from net/core/net-procfs.c from Linux kernel (procfs just uses this info). All of this of course means you need specific process to track.
You can either peruse information available in /proc or if you need something more stable, then duplicating net-procfs functionality specifically for your application might make sense.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have searched for possible solutions for days, but have had no luck at getting my SharePoint 2010 to return search results.
The search was working, but was only returning results from a subsite. I have gone through many blog posts and sites on setting up the search and still nothing. My last resort was to delete the search and reimplement it.
The search crawls just fine (no errors). Here are a couple of the blogs and sites I have tried (out of the many), but nothing seems to help.
http://social.msdn.microsoft.com/Forums/en-US/sharepoint2010setup/thread/688b5c52-f478-463b-bc00-debfd0c3be2b
http://sharepointgeorge.com/2010/configuring-enterprise-search-sharepoint-2010/
My setup is an intranet on a VM with SQL 2008 R2 (nothing out of the ordinary for the server, single farm). The search account has the rights to Full Read and is also included on all page permissions.
Here is a look at the log descriptions when a search is performed (no access denied errors). These results are all from the "Query Processor" category.
(w3wp.exe) PluggableSecurityTrimmerManager:SetSearchApplicationToUse: Set SearchApplication to 'Search Service Application'
(w3wp.exe) Resetting cookie: Old value = '', new value = 'nautilusRankDescending'
(mssearch.exe) 63239349-6356-4a02-96db-c40ffb223572-query-0: Query completed 109 ms, detailed time: Query stage execution ms times: 62 47 0 0 47 0 0 0 Query stage cpu ms times: 31 15 0 0 15 0 0 0 Query stage hit counts: 1 1 1 7 2 0 0 0 Cursor count: 13 Mapped page count: 16 Total index count: 1 [srequest.cxx:5526] d:\office\source\search\native\ytrip\tripoli\cifrmwrk\srequest.cxx
(w3wp.exe) Completed query execution with timings: total:140 dup:0 sec:0 join:0 ft:109 sql:31. Join Retry: 0. Security Trimming Retry: 0. Duplicate removal Retry: 0.
I am thoroughly baffled. Hopefully someone has had the same problem and can share how they fixed it.
One of the mistakes that we made is to use default Network Service account for the Application Pool. Make sure that you set up a separate domain\user account e.g. sp_search for search purpose.
Well, It has been quite the headache getting this to work. Below are some suggestions or check points to keep in mind. I hope they might help someone else out there.
if you have tried everything and you still don't have your search working delete the SSA and start over. I did this and then went through step by step
Set up a proper service/system account to handle the search.
Make sure that your application pool is started (at one time for some reason mine had stopped itself
I utilized this well documented article about setting up a search to rewalk myself through the process (after I deleted my initial search).
http://blog.concurrency.com/sharepoint/search-configuration-in-sharepoint-2010/
It has screen shots and useful tips for setting up scopes and crawl rules, etc.
I need to test a web form that takes a file upload.
The filesize in each upload will be about 10 MB.
I want to test if the server can handle over 100 simultaneous uploads, and still remain
responsive for the rest of the site.
Repeated form submissions from our office will be limited by our local DSL line.
The server is offsite with higher bandwidth.
Answers based on experience would be great, but any suggestions are welcome.
Use the ab (ApacheBench) command-line tool that is bundled with Apache
(I have just discovered this great little tool). Unlike cURL or wget,
ApacheBench was designed for performing stress tests on web servers (any type of web server!).
It generates plenty statistics too. The following command will send a
HTTP POST request including the file test.jpg to http://localhost/
100 times, with up to 4 concurrent requests.
ab -n 100 -c 4 -p test.jpg http://localhost/
It produces output like this:
Server Software:
Server Hostname: localhost
Server Port: 80
Document Path: /
Document Length: 0 bytes
Concurrency Level: 4
Time taken for tests: 0.78125 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Non-2xx responses: 100
Total transferred: 2600 bytes
HTML transferred: 0 bytes
Requests per second: 1280.00 [#/sec] (mean)
Time per request: 3.125 [ms] (mean)
Time per request: 0.781 [ms] (mean, across all concurrent requests)
Transfer rate: 25.60 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.6 0 15
Processing: 0 2 5.5 0 15
Waiting: 0 1 4.8 0 15
Total: 0 2 6.0 0 15
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 15
95% 15
98% 15
99% 15
100% 15 (longest request)
Automate Selenium RC using your favorite language. Start 100 Threads of Selenium,each typing a path of the file in the input and clicking submit.
You could generate 100 sequentially named files to make looping over them easyily, or just use the same file over and over again
I would perhaps guide you towards using cURL and submitting just random stuff (like, read 10MB out of /dev/urandom and encode it into base32), through a POST-request and manually fabricate the body to be a file upload (it's not rocket science).
Fork that script 100 times, perhaps over a few servers. Just make sure that sysadmins don't think you are doing a DDoS, or something :)
Unfortunately, this answer remains a bit vague, but hopefully it helps you by nudging you in the right track.
Continued as per Liam's comment:
If the server receiving the uploads is not in the same LAN as the clients connecting to it, it would be better to get as remote nodes as possible for stress testing, if only to simulate behavior as authentic as possible. But if you don't have access to computers outside the local LAN, the local LAN is always better than nothing.
Stress testing from inside the same hardware would be not a good idea, as you would do double load on the server: Figuring out the random data, packing it, sending it through the TCP/IP stack (although probably not over Ethernet), and only then can the server do its magic. If the sending part is outsourced, you get double (taken with an arbitrary sized grain of salt) performance by the receiving end.