I have to limit my API request to 10 calls per minute, how can I modify the for loops to accomplish this?
I am trying to add in time.sleep(8) in the for observation loop without any luck... Any ideas?
import arrow # learn more: https://python.org/pypi/arrow
from WunderWeather import weather # learn more: https://python.org/pypi/WunderWeather
import time
api_key = ''
extractor = weather.Extract(api_key)
zip = '53711'
# get 20170101 00:00
begin_date = arrow.get("2017","YYYY")
# get 20171231 23:00
end_date = arrow.get("2018","YYYY").shift(hours=-1)
for date in arrow.Arrow.range('hour',begin_date,end_date):
# get date object for feature
# http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.date
date_weather = extractor.date(zip,date.format('YYYYMMDD'))
# use shortcut to get observations and data
# http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.date.Observation
for observation in date_weather.observations:
time.sleep(8)
print("Date:",observation.date_pretty)
print("Temp:",observation.temp_f)
A possible explanation of why you are still exceeding the API limit might have to do with the line on which you are adding the time wait. If the API response you are getting contains no observations, the inner loop won't execute. So first I would try to move the time wait in the outer loop right after the API call.
You might also consider using something like loopingCall from twisted to schedule your task to run every X seconds
http://twistedmatrix.com/documents/9.0.0/core/howto/time.html
Depending on how realtime you want your data or you can afford to be a day behind, you could get all observations for a date in the past which would be one API call to retrieve data for a day(or it could be an end of day summary for the current day's observations).
Alternatively, if you're trying to get the current weather every x minutes or so (under the limit)
I'd use some sort of loop with a timer (or possibly twisted which seems to abstract the "loop") but make a call to one of the following (depending on what you're looking for). Your current code is looking for dates in the past but these other endpoints are for the current day.
You don't want the timer in the observations loop since, as mentioned above, there might be none.
http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.hourly_daycast
http://wunderweather.readthedocs.io/en/latest/WunderWeather.html#WunderWeather.weather.Extract.today_now
which can be called similar to the following examples
http://wunderweather.readthedocs.io/en/latest/index.html#additional-examples
Related
I am trying to retrieve a specific step function execution input in the past using the list_executions and describe_execution functions in boto3, first to retrieve all the calls and then to get the execution input (I can't use describe_execution directly as I do not know the full state machine ARN). However, list_executions does not accept a filter argument (such as "name"), so there is no way to return partial matches, but rather it returns all (successful) executions.
The solution for now has been to list all the executions and then loop over the list and select the right one. The issue is that this function can return a max 1000 newest records (as per the documentation), which will soon be an issue as there will be more than 1000 executions and I will need to get old executions.
Is there a way to specify a filter in the list_executions/describe_execution function to retrieve execution partially filtered, for ex. using prefix?
import boto3
sf=boto3.client("stepfunctions").list_executions(
stateMachineArn="arn:aws:states:something-something",
statusFilter="SUCCEEDED",
maxResults=1000
)
You are right that the SFN APIs like ListExecutions do not expose other filtering options. Nonetheless, here are two ideas to make your task of searching execution inputs easier:
Use the ListExecutions Paginator to help with looping through the response items.
If you know in advance which inputs are of interest, add a Task to the State Machine to persist execution inputs and ARNs to, say, a DynamoDB table, in a manner that makes subsequent searches easier.
I know there are many questions relative to this, but I can't find exactly what I am looking for..
I am creating an iOS Rideshare app and am utilizing the Google Distance Matrix. What I am looking to do is taking the current date and set it to midnight. For example: currentDate = 12/6/2018 12:00:00.
I want to take this currentDate value and convert is to Epoch and set it up as a baseEpoch value. This way, I can take the user's time input, and add the difference to get the date/time they entered in Epoch form.
I've tried solutions such as:
function convertToEpoch(time)
{
var sep = time.split(':');
var seconds = (+sep[0]) * 60 * 60 + (+sep[1]) * 60 + (+sep[2]);
return seconds;
}
function currentDateAsEpoch(time) {
var time = new Date();
time.setHours(0,0,0,0);
convertToEpoch(time);
}
const baseEpoch = currentDateAsEpoch();
But am getting the error: TypeError: time.split is not a function
I want the baseEpoch to be set as the current date so Google Distance Matrix doesn't return the departure_time error saying time can only be equal to or in the future.
Thank you in advance for your help!
You can use momentjs package for nodejs. This package helps you in dealing date and time effortlessly. You may need to dig more into the docs for better understanding of the moment module(Documentation is simple to understand).
Here is some snippet from momentjs docs.
moment().unix();
//moment#unix outputs a Unix timestamp (the number of seconds since the Unix Epoch).
moment(1318874398806).unix(); // 1318874398
Also using a library like moment which is well tested is better than writing your own functions to handle date and time for following reasons.
The code would be very well tested due to large number of people using them in a day to day basis.
No need to waste your time in reinventing the wheel.
Additional functionalities for future development work.(Like formatting and other date operations)
Easy to understand and implement even for new team members (due very well written documents and good support due to large number of developers using the library)
I have a question regarding the Python API of Interactive Brokers.
Can multiple asset and stock contracts be passed into reqMktData() function and obtain the last prices? (I can set the snapshots = TRUE in reqMktData to get the last price. You can assume that I have subscribed to the appropriate data services.)
To put things in perspective, this is what I am trying to do:
1) Call reqMktData, get last prices for multiple assets.
2) Feed the data into my prediction engine, and do something
3) Go to step 1.
When I contacted Interactive Brokers, they said:
"Only one contract can be passed to reqMktData() at one time, so there is no bulk request feature in requesting real time data."
Obviously one way to get around this is to do a loop but this is too slow. Another way to do this is through multithreading but this is a lot of work plus I can't afford the extra expense of a new computer. I am not interested in either one.
Any suggestions?
You can only specify 1 contract in each reqMktData call. There is no choice but to use a loop of some type. The speed shouldn't be an issue as you can make up to 50 requests per second, maybe even more for snapshots.
The speed issue could be that you want too much data (> 50/s) or you're using an old version of the IB python api, check in connection.py for lock.acquire, I've deleted all of them. Also, if there has been no trade for >10 seconds, IB will wait for a trade before sending a snapshot. Test with active symbols.
However, what you should do is request live streaming data by setting snapshot to false and just keep track of the last price in the stream. You can stream up to 100 tickers with the default minimums. You keep them separate by using unique ticker ids.
I'm creating an end point that gives the current traffic conditions and the temperature. It combines data from two endpoints:
GET current traffic statistics (at every request)
GET current temperature (every 3 hours)
A simple solution would be to chain two promises together, but I don't need to call 2. at every request. How could I structure my code to hold the data for 2. and refresh it periodically?
create temperature module with current temperature value, setInterval to update this value every 3 hours.
On your endpoint make request for traffic data, and read cached value from temperature module.
I'm building an app where I need to have three scoreboards which I'm implementing with sorted sets and lists. The app is running on node.js using the node_redis (https://github.com/mranney/node_redis) module for the redis client.
The first scoreboard is a 'latest scores' which I'm using a list and LPUSH for. The second is an all time high score which I'm using a sorted list for with the ZADD command.
I'm having trouble implementing a 'high scores this week'. I was thinking that I should use another sorted list using ZADD with an EXPIRE set for one week. That all works fine, but after the list has expired for the first time, it'll continue to add to a new list forever.
Is there a redis command to have an expire to auto renew? (I've been searching for an answer for a couple of hours now but the answer appears to be no). I'm coming to the conclusion that I'll need to do this programmatically. During a function call that uses the set, I could check to see if the TTL is -1 and reset it there and then. Is this best practice? Am I missing a clever trick somewhere? Do I need to be concerned about the extra database requests?
--EDIT--
I've had a reply to this question on twitter https://twitter.com/redsmin/status/302177241167691777
The suggested solution (if I understand correctly) is to use the EXPIREAT command along with each ZADD
expireat myscoreboard {{timestamp of the end of the week}}
zadd myscoreboard 1 "one"
This "feels" right to me but I'm new to redis so would appreciate some discussion on this technique or any other ways of solving the problem.
It depends on how you define "one week". There are several ways to use it, for example:
"The last 7 days"
"Week of the year"
"This week starting on sunday and ending on Saturday"
The simplest to implement are 2 & 3.
You specify a set which includes in it's keyname the date/time to start on, using an expire of one week. You then simply determine on the client side which day you want and grab the data.
For example
zadd scoreboard:weekly:03:March:2013 1 "bob"
Then the following week your keyname would be
zadd scoreboard:weekly:10:March:2013 1 "bob"
When you first create the key you set the expires, and that is all. No need to re-set it every time. Pseudocode follows:
if (ttl scoreboard:weekly:03:March:2013) == 0:
expire scoreboard:weekly:03:March:2013 604800
This way you only set the expiration once, get auto-expiration, and can easily pull the weekly scoreboard.
You could implement a rolling week using the same method but you would need to go to a daily key name and calculate what keys to get, then merge them. You could do this using zunionstore.