I need to find any requests where 5 of them in a row take 30 seconds or more.
Is there a way to do this using a kusto query?
I guess it could be rephrased as 'if there is a request that takes 30 seconds or more to get a response, check if the next 4 requests also take 30 seconds'.
Check this thread -
Check for the request where response time is higher
Hope it helps.
Related
pysolr is taking time first time after the solr restart. Scenario is like this.
1)Restart Solr server
2)Execute the query directly in solr is taking 4 sec
3)when we are trying to execute the same query using pysolr, it is taking 300 seconds or more for the first time. After refreshing it is taking only less than 2 sec.
When I check the pysolr code the time is taking in the below code.
resp = requests_method(url, data=bytes_body, headers=headers, files=files,
timeout=self.timeout, auth=self.auth)
4)Can anybody help me in getting the out faster first time also. I think this abnormal because executing the query directly in Solr is taking very few seconds.
I have an application that it's gathering data from another application, both in NodeJS.
I was wondering, how can I trigger sending the data to a third application on certain conditions? For example, every 10 mins if there's data in a bucket or when I have 20 elements to send?
And if the call on the third parties fails, how can I repeat it after 10-15 mins?
EDIT:
The behaviour should be something like:
if you have 1 data posted (axios.post) AND [10 mins passed OR other 10 data posted] SUBMIT to App n.3
What can help me doing so? Can I keep the value saved until those requirements are satisfied?
Thank you <3
You can use packages like node-schedule which is popular to schedule tasks. When callback runs check if there is enough data(posts) to send.
I have to limit my API request to 10 calls per minute, how can I modify the for loops to accomplish this?
I am trying to add in time.sleep(8) in the for observation loop without any luck... Any ideas?
import arrow # learn more: https://python.org/pypi/arrow
from WunderWeather import weather # learn more: https://python.org/pypi/WunderWeather
import time
api_key = ''
extractor = weather.Extract(api_key)
zip = '53711'
# get 20170101 00:00
begin_date = arrow.get("2017","YYYY")
# get 20171231 23:00
end_date = arrow.get("2018","YYYY").shift(hours=-1)
for date in arrow.Arrow.range('hour',begin_date,end_date):
# get date object for feature
# http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.date
date_weather = extractor.date(zip,date.format('YYYYMMDD'))
# use shortcut to get observations and data
# http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.date.Observation
for observation in date_weather.observations:
time.sleep(8)
print("Date:",observation.date_pretty)
print("Temp:",observation.temp_f)
A possible explanation of why you are still exceeding the API limit might have to do with the line on which you are adding the time wait. If the API response you are getting contains no observations, the inner loop won't execute. So first I would try to move the time wait in the outer loop right after the API call.
You might also consider using something like loopingCall from twisted to schedule your task to run every X seconds
http://twistedmatrix.com/documents/9.0.0/core/howto/time.html
Depending on how realtime you want your data or you can afford to be a day behind, you could get all observations for a date in the past which would be one API call to retrieve data for a day(or it could be an end of day summary for the current day's observations).
Alternatively, if you're trying to get the current weather every x minutes or so (under the limit)
I'd use some sort of loop with a timer (or possibly twisted which seems to abstract the "loop") but make a call to one of the following (depending on what you're looking for). Your current code is looking for dates in the past but these other endpoints are for the current day.
You don't want the timer in the observations loop since, as mentioned above, there might be none.
http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.hourly_daycast
http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.today_now
which can be called similar to the following examples
http://wunderweather.readthedocs.io/en/latest/index.html#additional-examples
I am using Jmeter (started using it a few days ago) as a tool to simulate a load of 30 threads using a csv data file that contains login credentials for 3 system users.
The objective I set out to achieve was to measure 30 users (threads) logging in and navigating to a page via the menu over a time span of 30 seconds.
I have set my thread group as:
Number of threads: 30
Ramp-up Perod: 30
Loop Count: 10
I ran the test successfully. Now I'd like to understand what the results mean and what is classed as good/bad measurements, and what can be suggested to improve the results. Below is a table of the results collated in the Summary report of Jmeter.
I have conducted research only to find blogs/sites telling me the same info as what is defined on the jmeter.apache.org site. One blog (Nicolas Vahlas) that I came across gave me some very useful information,but still hasn't help me understand what to do next with my results.
Can anyone help me understand these results and what I could do next following the execution of this test plan? Or point me in the right direction of an informative blog/site that will help me understand what to do next.
Many thanks.
According to me, Deviation is high.
You know your application better than all of us.
you should focus on, avg response time you got and max response frequency and value are acceptable to you and your users? This applies to throughput also.
It shows average response time is below 0.5 seconds and maximum response time is also below 1 second which are generally acceptable but that should be defined by you (Is it acceptable by your users). If answer is yes, try with more load to check scaling.
In you requirement it is mentioned that you need have 30 concurrent users performing different actions. The response time of your requests is less and you have ramp-up of 30 seconds. Can you please check total active threads during the test. I believe the time for which there will be 30 concurrent users in system is pretty short so the average response time that you are seeing seems to be misleading. I would suggest you run a test for some more time so that there will be 30 concurrent users in the system and that would be correct reading as per your requirements.
You can use Aggregate report instead of summary report. In performance testing
Throughput - Requests/Second
Response Time - 90th Percentile and
Target application resource utilization (CPU, Processor Queue Length and Memory)
can be used for analysis. Normally SLA for websites is 3 seconds but this requirement changes from application to application.
Your test results are good, considering if the users are actually logging into system/portal.
Samples: This means the no. of requests sent on a particular module.
Average: Average Response Time, for 300 samples.
Min: Min Response Time, among 300 samples (fastest among 300 samples).
Max: Max Response Time, among 300 samples (slowest among 300 samples).
Standard Deviation: A measure of the variation (for 300 samples).
Error: failure %age
Throughput: No. of request processed per second.
Hope this will help.
We have a metric that we increment every time a user performs a certain action on our website, but the graphs don't seem to be accurate.
So going off this hunch, we invested the updates.log of carbon and discovered that the action had happened over 4 thousand times today(using grep and wc), but according the Integral result of the graph it returned only 220ish.
What could be the cause of this? Data is being reported to statsd using the statsd php library, and calling statsd::increment('metric'); and as stated above, the log confirms that 4,000+ updates to this key happened today.
We are using:
graphite 0.9.6 with statsD (etsy)
After some research through the documentation, and some conversations with others, I've found the problem - and the solution.
The way the whisper file format is designed, it expect you (or your application) to publish updates no faster than the minimum interval in your storage-schemas.conf file. This file is used to configure how much data retention you have at different time interval resolutions.
My storage-schemas.conf file was set with a minimum retention time of 1 minute. The default StatsD daemon (from etsy) is designed to update to carbon (the graphite daemon) every 10 seconds. The reason this is a problem is: over a 60 second period StatsD reports 6 times, each write overwrites the last one (in that 60 second interval, because you're updating faster than once per minute). This produces really weird results on your graph because the last 10 seconds in a minute could be completely dead and report a 0 for the activity during that period, which results in completely nuking all of the data you had written for that minute.
To fix this, I had to re-configure my storage-schemas.conf file to store data at a maximum resolution of 10 seconds, so every update from StatsD would be saved in the whisper database without being overwritten.
Etsy published the storage-schemas.conf configuration that they were using for their installation of carbon, which looks like this:
[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974
This has a 10 second minimum retention time, and stores 6 hours worth of them. However, due to my next problem, I extended the retention periods significantly.
As I let this data collect for a few days, I noticed that it still looked off (and was under reporting). This was due to 2 problems.
StatsD (older versions) only reported an average number of events per second for each 10 second reporting period. This means, if you incremented a key 100 times in 1 second and 0 times for the next 9 seconds, at the end of the 10th second statsD would report 10 to graphite, instead of 100. (100/10 = 10). This failed to report the total number of events for a 10 second period (obviously).Newer versions of statsD fix this problem, as they introduced the stats_counts bucket, which logs the total # of events per metric for each 10 second period (so instead of reporting 10 in the previous example, it reports 100).After I upgraded StatsD, I noticed that the last 6 hours of data looked great, but as I looked beyond the last 6 hours - things looked weird, and the next reason is why:
As graphite stores data, it moves data from high precision retention to lower precision retention. This means, using the etsy storage-schemas.conf example, after 6 hours of 10 second precision, data was moved to 60 second (1 minute) precision. In order to move 6 data points from 10s to 60s precision, graphite does an average of the 6 data points. So it'd take the total value of the oldest 6 data points, and divide it by 6. This gives an average # of events per 10 seconds for that 60 second period (and not the total # of events, which is what we care about specifically).This is just how graphite is designed, and for some cases it might be useful, but in our case, it's not what we wanted. To "fix" this problem, I increased our 10 second precision retention time to 60 days. Beyond 60 days, I store the minutely and 10-minutely precisions, but they're essentially there for no reason, as that data isn't as useful to us.
I hope this helps someone, I know it annoyed me for a few days - and I know there isn't a huge community of people that are using this stack of software for this purpose, so it took a bit of research to really figure out what was going on and how to get a result that I wanted.
After posting my comment above I found Graphite 0.9.9 has a (new?) configuration file, storage-aggregation.conf, in which one can control the aggregation method per pattern. The available options are average, sum, min, max, and last.
http://readthedocs.org/docs/graphite/en/latest/config-carbon.html#storage-aggregation-conf