Hi I am controlling nest thermostat by Firebase SDK. We have big problem with Too many requests exception. I saw a few threads about that but without solution for that problem. So I would like to ask if someone managed to solve that problem?
From https://developer.nest.com/documentation/data-rate-limits:
To avoid errors, we recommend you limit requests to one call per minute, maximum.
From this, and from other posts on SO, I believe you can make 60 requests in any 60 minute window. But I think this is a Per Access Token limit. Nest say separately that they rate-limit writes to devices and structures - I haven't hit the write limit, but I don't do a lot of writing to the device.
In order to work around this limitation, I did the following:
When the user issues a command, don't send the command to Firebase immediately; just queue it and execute it after a constant delay, D
If the user issues another command within delay D, then de-queue the previous command and queue the most recent one.
With this strategy, the number of issues commands is at most one per D seconds.
Here is the code:
-(void) delayedSetValue:(NSArray *)valuesAndURL {
NSDictionary *values = valuesAndURL[0];
NSString *URL = valuesAndURL[1];
if ([self.subscribedURLs objectForKey:URL]) {
[[self.fireBi objectForKey:URL] runTransactionBlock:^FTransactionResult *(FMutableData *currentData) {
[currentData setValue:values];
return [FTransactionResult successWithValue:currentData];
} andCompletionBlock:^(NSError *error, BOOL committed, FDataSnapshot *snapshot) {
if (error) {
NSLog(#"Error: %#", error);
}
} withLocalEvents:NO];
}
}
/*
* Sets the values for the given firebase URL.
*/
- (void)setValues:(NSDictionary *)values forURL:(NSString *)URL
{
[NSObject cancelPreviousPerformRequestsWithTarget:self];
NSArray *valuesAndURL = #[values, URL];
[self performSelector:#selector(delayedSetValue:) withObject:valuesAndURL afterDelay:5.0f];
}
I have had a similare issue. but it was only when I authenticated too much. I had to wait for a while to get the change to authenticate again. Now i got a websocket running to get the values and that gives no problems. Setting values is the next step i am going to do. So for that i dont know how much request you can send
Related
I'm new in Node JS and i wonder if under mentioned snippets of code has multisession problem.
Consider I have Node JS server (express) and I listen on some POST request:
app.post('/sync/:method', onPostRequest);
var onPostRequest = function(req,res){
// parse request and fetch email list
var emails = [....]; // pseudocode
doJob(emails);
res.status(200).end('OK');
}
function doJob(_emails){
try {
emailsFromFile = fs.readFileSync(FILE_PATH, "utf8") || {};
if(_.isString(oldEmails)){
emailsFromFile = JSON.parse(emailsFromFile);
}
_emails.forEach(function(_email){
if( !emailsFromFile[_email] ){
emailsFromFile[_email] = 0;
}
else{
emailsFromFile[_email] += 1;
}
});
// write object back
fs.writeFileSync(FILE_PATH, JSON.stringify(emailsFromFile));
} catch (e) {
console.error(e);
};
}
So doJob method receives _emails list and I update (counter +1) these emails from object emailsFromFile loaded from file.
Consider I got 2 requests at the same time and it triggers doJob twice. I afraid that when one request loaded emailsFromFile from file, the second request might change file content.
Can anybody spread the light on this issue?
Because the code in the doJob() function is all synchronous, there is no risk of multiple requests causing a concurrency problem.
If you were using async IO in that function, then there would be possible concurrency issues.
To explain, Javascript in node.js is single threaded. So, there is only one thread of Javascript execution running at a time and that thread of execution runs until it returns back to the event loop. So, any sequence of entirely synchronous code like you have in doJob() will run to completion without interruption.
If, on the other hand, you use any asynchronous operations such as fs.readFile() instead of fs.readFileSync(), then that thread of execution will return back to the event loop at the point you call fs.readFileSync() and another request can be run while it is reading the file. If that were the case, then you could end up with two requests conflicting over the same file. In that case, you would have to implement some form of concurrency protection (some sort of flag or queue). This is the type of thing that databases offer lots of features for.
I have a node.js app running on a Raspberry Pi that uses lots of async file I/O and I can have conflicts with that code from multiple requests. I solved it by setting a flag anytime I'm writing to a specific file and any other requests that want to write to that file first check that flag and if it is set, those requests going into my own queue are then served when the prior request finishes its write operation. There are many other ways to solve that too. If this happens in a lot of places, then it's probably worth just getting a database that offers features for this type of write contention.
I have created a simple web interface for vertica.
I expose simple operation above a vertica cluster.
one of the functionality I expose is querying vertica.
when my user enters a multi-query the node modul throws an exception and my process exits with exit 1.
Is there any way to catch this exception?
Is there any way overcome the problem in a different way?
Right now there's no way to overcome this when using a callback for the query result.
Preventing this from happening would involve making sure there's only one query in the user's input. This is hard because it involves parsing SQL.
The callback API isn't built to deal with multi-queries. I simply haven't bothered implementing proper handling of this case, because this has never been an issue for me.
Instead of a callback, you could use the event listener API, which will send you lower level messages, and handle this yourself.
q = conn.query("SELECT...; SELECT...");
q.on("fields", function(fields) { ... }); // 1 time per query
q.on("row", function(row) { ... }); // 0...* time per query
q.on("end", function(status) { ... }); // 1 time per query
I am referring WWDC 2014 sample app NewBox for document provider extension.
I am using following code from NeBox app, to import a document from Document Provider to my app.
- (void)documentPicker:(UIDocumentPickerViewController *)controller didPickDocumentAtURL:(NSURL *)url {
BOOL startAccessingWorked = [url startAccessingSecurityScopedResource];
NSURL *ubiquityURL = [[NSFileManager defaultManager] URLForUbiquityContainerIdentifier:nil];
NSLog(#"ubiquityURL %#",ubiquityURL);
NSLog(#"start %d",startAccessingWorked);
NSFileCoordinator *fileCoordinator = [[NSFileCoordinator alloc] init];
NSError *error;
[fileCoordinator coordinateReadingItemAtURL:url options:0 error:&error byAccessor:^(NSURL *newURL) {
NSData *data = [NSData dataWithContentsOfURL:newURL];
NSLog(#"error %#",error);
NSLog(#"data %#",data);
}];
[url stopAccessingSecurityScopedResource];
}
App totally hangs for coordinateReadingItemAtURL method.
Any inputs will be helpful.
I noticed this problem in NewBox app as well, and decided to trace it. So, there are two extensions in this app: Document Picker and File Provider. To make long story short, there is a race condition between the two when they try to access files within app's document storage folder.
In my opinion, the easiest method to trace down a problem is to put NSLog() in a bunch of locations. The problem is, however, that the debugging output generated by extension won't be visible in Xcode console. The good news is that you can open console in iOS Simulator app by clicking to Debug -> Open System Log menu. This will show all kinds of debugging messages, including those generated by extensions. You can find more about extension debugging here.
By using this method one can easily realize that execution gets stuck in File Provider's startProvidingItemAtURL method. More specifically, the following line causes a deadlock:
[self.fileCoordinator coordinateWritingItemAtURL:url options:0 error:&error byAccessor:^(NSURL *newURL) {
Why is that? Take a look at documentation for coordinateWritingItemAtURL:
If the url parameter specifies a file:
This method waits for other readers and writers of the exact same file to finish in-progress actions.
Function documentPicker that you mentioned calls a read operation, which in its turn triggers a write operation. This is a deadlock. I guess the easiest way to fix it would be to avoid using coordinateWritingItemAtURL in File Provider.
As per documentation:
Each of these methods wait synchronously on the same thread they were invoked on before invoking the passed-in accessor block on the same thread, instead of waiting asynchronously and scheduling invocation of the block on a specific queue.
Apple recommends that you not use file coordination inside this method. The system already guarantees that no other process can access the file while this method is executing. That's the sole reason for this deadlock.
Please refer to this documentation for more details.
You can use block also. Block works too fast, hang problem will get resolve.
Step 1: Take global variable of
UIDocumentPickerViewController *documentPicker;
also decalre
typedef void(^myCompletion)(BOOL);
Step 2: Write a method where allocation takes place and can send callback on completion
-(void) allocateDocumentPicker:(myCompletion) compblock{
//do stuff
documentPicker = [[UIDocumentPickerViewController alloc] initWithDocumentTypes:#[#"public.content"]
inMode:UIDocumentPickerModeImport];
documentPicker.delegate = self;
documentPicker.modalPresentationStyle = UIModalPresentationFormSheet;
compblock(YES);
}
Step 3: Call the method where allocation is taking place every time you want to open the composer but present it on receiving completion as YES.
-(IBAction)attachmentButtonClicked:(id)sender{
[self allocateDocumentPicker:^(BOOL finished) {
if(finished){
[self.parentScreen presentViewController:documentPicker animated:YES completion:nil];
}
}];
}
Simple Syntax to create own block, take reference from this link
Custom completion block for my own method
The Azure Service Bus supports a built-in retry mechanism which makes an abandoned message immediately visible for another read attempt. I'm trying to use this mechanism to handle some transient errors, but the message is made available immediately after being abandoned.
What I would like to do is make the message invisible for a period of time after it is abandoned, preferably based on an exponentially incrementing policy.
I've tried to set the ScheduledEnqueueTimeUtc property when abandoning the message, but it doesn't seem to have an effect:
var messagingFactory = MessagingFactory.CreateFromConnectionString(...);
var receiver = messagingFactory.CreateMessageReceiver("test-queue");
receiver.OnMessageAsync(async brokeredMessage =>
{
await brokeredMessage.AbandonAsync(
new Dictionary<string, object>
{
{ "ScheduledEnqueueTimeUtc", DateTime.UtcNow.AddSeconds(30) }
});
}
});
I've considered not abandoning the message at all and just letting the lock expire, but this would require having some way to influence how the MessageReceiver specifies the lock duration on a message, and I can't find anything in the API to let me change this value. In addition, it wouldn't be possible to read the delivery count of the message (and therefore make a decision for how long to wait for the next retry) until after the lock is already required.
Can the retry policy in the Message Bus be influenced in some way, or can a delay be artificially introduced in some other way?
Careful here because I think you are confusing the retry feature with the automatic Complete/Abandon mechanism for the OnMessage event-driven message handling. The built in retry mechanism comes into play when a call to the Service Bus fails. For example, if you call to set a message as complete and that fails, then the retry mechanism would kick in. If you are processing a message an exception occurs in your own code that will NOT trigger a retry through the retry feature. Your question doesn't get explicit on if the error is from your code or when attempting to contact the service bus.
If you are indeed after modifying the retry policy that occurs when an error occurs attempting to communicate with the service bus you can modify the RetryPolicy that is set on the MessageReciver itself. There is an RetryExponitial which is used by default, as well as an abstract RetryPolicy you can create your own from.
What I think you are after is more control over what happens when you get an exception doing your processing, and you want to push off working on that message. There are a few options:
When you create your message handler you can set up OnMessageOptions. One of the properties is "AutoComplete". By default this is set to true, which means as soon as processing for the message is completed the Complete method is called automatically. If an exception occurs then abandon is automatically called, which is what you are seeing. By setting the AutoComplete to false you required to call Complete on your own from within the message handler. Failing to do so will cause the message lock to eventually run out, which is one of the behaviors you are looking for.
So, you could write your handler so that if an exception occurs during your processing you simply do not call Complete. The message would then remain on the queue until it's lock runs out and then would become available again. The standard dead lettering mechanism applies and after x number of tries it will be put into the deadletter queue automatically.
A caution of handling this way is that any type of exception will be treated this way. You really need to think about what types of exceptions are doing this and if you really want to push off processing or not. For example, if you are calling a third party system during your processing and it gives you an exception you know is transient, great. If, however, it gives you an error that you know will be a big problem then you may decide to do something else in the system besides just bailing on the message.
You could also look at the "Defer" method. This method actually will then not allow that message to be processed off the queue unless it is specifically pulled by its sequence number. You're code would have to remember the sequence number value and pull it. This isn't quite what you described though.
Another option is you can move away from the OnMessage, Event-driven style of processing messages. While this is very helpful you don't get a lot of control over things. Instead hook up your own processing loop and handle the abandon/complete on your own. You'll also need to deal some of the threading/concurrent call management that the OnMessage pattern gives you. This can be more work but you have the ultimate in flexibility.
Finally, I believe the reason the call you made to AbandonAsync passing the properties you wanted to modify didn't work is that those properties are referring to Metadata properties on the method, not standard properties on BrokeredMessage.
I actually asked this same question last year (implementation aside) with the three approaches I could think of looking at the API. #ClemensVasters, who works on the SB team, responded that using Defer with some kind of re-receive is really the only way to control this precisely.
You can read my comment to his answer for a specific approach to doing it where I suggest using a secondary queue to store messages that indicate which primary messages have been deferred and need to be re-received from the main queue. Then you can control how long you wait by setting the ScheduledEnqueueTimeUtc on those secondary messages to control exactly how long you wait before you retry.
I ran into a similar issue where our order picking system is legacy and goes into maintenance mode each night.
Using the ideas in this article(https://markheath.net/post/defer-processing-azure-service-bus-message) I created a custom property to track how many times a message has been resubmitted and manually dead lettering the message after 10 tries. If the message is under 10 retries it clones the message increments the custom property and sets the en queue of the new message.
using Microsoft.Azure.ServiceBus;
public PickQueue()
{
queueClient = new QueueClient(QUEUE_CONN_STRING, QUEUE_NAME);
}
public async Task QueueMessageAsync(int OrderId)
{
string body = JsonConvert.SerializeObject(OrderId);
var message = new Message(Encoding.UTF8.GetBytes(body));
await queueClient.SendAsync(message);
}
public async Task ReQueueMessageAsync(Message message, DateTime utcEnqueueTime)
{
int resubmitCount = (int)(message.UserProperties["ResubmitCount"] ?? 0) + 1;
if (resubmitCount > 10)
{
await queueClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
else
{
Message clone = message.Clone();
clone.UserProperties["ResubmitCount"] = ++resubmitCount;
await queueClient.ScheduleMessageAsync(message, utcEnqueueTime);
}
}
This question asks how to implement exponential backoff in Azure Functions. If you do not want to use the built-in RetryPolicy (only available when autoComplete = false), here's the solution I've been using:
public static async Task ExceptionHandler(IMessageSession MessageSession, string LockToken, int DeliveryCount)
{
if (DeliveryCount < Globals.MaxDeliveryCount)
{
var DelaySeconds = Math.Pow(Globals.ExponentialBackoff, DeliveryCount);
await Task.Delay(TimeSpan.FromSeconds(DelaySeconds));
await MessageSession.AbandonAsync(LockToken);
}
else
{
await MessageSession.DeadLetterAsync(LockToken);
}
}
I have the following code in my application to load some data from my API. It works fine, great in fact in iOS 5 but on iOS 4 I am getting so many responses with status 204.
This only happens on iOS 4, this could have been treated as an API error, but it works great in the browser, on Rested.app, on iOS 5 etc... only iOS 4 fails, it fails in the simulator and on the device (iPhone 4).
I am calling this code each time I load a cell into a table view. I have a core data object with a load state, set to no initially, if it's not loaded I perform this code, if it's loaded, I skip this code. In the mean time I display a spinner inside the cell on the table view.
I am sure it's a problem with multiple requests in GCD on iOS 4.
Can anyone spot anything wrong with my code snippet?
-(void)myFunction{
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// query users participations (Network)
NSError * _urlError = nil;
NSString * url = [NSString stringWithFormat:#"my api url"];
NSMutableURLRequest * loginHTTPRequest = [NSMutableURLRequest requestWithURL:[NSURL URLWithString:url]];
[loginHTTPRequest setValue:#"application/json" forHTTPHeaderField:#"Accept"];
NSLog(#"Description: %#", [loginHTTPRequest description]);
NSHTTPURLResponse * _responseHeaders = nil;
NSData * responseData = [NSURLConnection sendSynchronousRequest:loginHTTPRequest
returningResponse:&_responseHeaders
error:&_urlError];
if(_urlError != nil){
dispatch_async( dispatch_get_main_queue(), ^{
// alert network connection error
});
return;
}
NSString *json_string = [[NSString alloc] initWithData:responseData encoding:NSUTF8StringEncoding];
SBJsonParser *parser = [[SBJsonParser alloc] init];
NSDictionary * jsonData = [NSDictionary dictionaryWithDictionary:[parser objectWithString:json_string]];
[json_string release];
[parser release];
dispatch_async( dispatch_get_main_queue(), ^{
// here [_responseHeaders statusCode] keeps returning 204 and there is nothing in responseData
// do some Core Data stuff
});
});
}
UPDATE
Note this code is working fine if called even with a for loop repeatedly, the issue is when I invoke this method from tableView:cellForRowAtIndexPath:
I have Core Data objects with a property "isLoaded" set to NO and changed to YES upon remote load. When my tableview's datasource loads the cells for each object, the tableView:cellForRowAtIndexPath: method calls this function if the object's "isLoaded" property is NO.
I suspected the problem may be because there 2 or more simultaneous calls to the API happening when the table is loaded and reloaded. Each successful load from the api invokes reloadData for that tableview.
This lets me have a pre filled tableview with spinners and asynchronously load in my data as I need it on screen which is nice because I can efficiently use NSFetchedResultsController with lazy loading my objects core data.
(I have an endpoint for all my objects returning an array of object id's - I create Core Data objects with only the ID's, all rest of data, name, date etc etc... is not loaded until it's needed).
When I start scrolling around the new cells which are created/reused call this method and they always get a 200 response with the data. it's only the first loading which causes this "block".
I think I found the problem, I was performing synchronous requests on an asynchronous GCD thread, and for some reason timeouts were occurring, but only on requests in iOS 4, maybe the headers are sent slightly differently from iOS 4 which is causing the API to take longer to respond ? Or maybe multiple (as in simultaneous to the millisecond) requests sent from different threads synchronously on an asynchronous thread were clashing in the system before being sent ?
Anyhow... this didn't seem to be the case calling google.com or even my own private server, so it must be something to do with the headers and multiple requests...
I am using asi http from github and it's working a lot more efficiently, now I am not using GCD for these requests, just an ASI queue.
Any final thoughts on iOS 4 synchronous requests performing on an asynchronous GCD thread with possible timeouts not being respected and returning early with a status 204 ?
Ok - if it is https, then you probably can't get away with using sendSynchronousRequest. The documentation states that things like [NSURLConnection connection:didReceiveAuthenticationChallenge:] won't call some key things, i.e.:
If authentication is required in order to download the request, the required credentials must be specified as part of the URL. If authentication fails, or credentials are missing, the connection will attempt to continue without credentials.
I'm still surprised it's working on iOS5, to be honest. I think you'll have to use asynchronous methods to at least debug it to find out what is going on.