An user submits a onetime key to my backend, which, if unused, will return content from my server.
If a request comes in, the server will search if it finds the submitted key in the "used keys" database. If it doesn't find the key, it will immediately proceed with adding the key to "used keys" and only then return the content to the user.
How do i prevent someone from getting the content twice by submitting the same key in two different requests at exactly the same time, which could result in both requests not finding the key in "used keys" and therefore returning the content twice.
My solution so far is to wait a random time amount every time a request comes in. This way the key will likely be already marked as used for the request with the longer waiting time. But this would fail if the random wait times for both request are to close together, within 1 second or less.
Unfortunately i didn't know what to google exactly, so link-only responses would be helpful as well.
Related
I'm creating a web application where users earn points for using it (from time to time).
What is the best way to change the amount of points the user has in a safe way?
My first solution was to use a POST Request with the data in the body, but it would be easily circumvented since the user could open the console and send infinite copied requests and earn infinite points. And if I created a token, the user would copy that same token and reuse it until it is invalidated.
My second solution was to create a websocket that while the user maintains connection, he earns X points in X time but it would also be circumvented due to a false connection by the console
What to do in this situation?
Use a POST request, and validate that the request is authentic on the back-end (based upon the criteria in which it is deemed 'reasonable' to award the points).
For example, if a user could only earn 1 point every 30 minutes, store when the user was last awarded points in a database, and then ensure that 30 minutes had passed since that point in time (again, server side).
You could also ensure that the user isn't getting more points than the should by checking the value of the points that they are being awarded, in a similar manner. Want one point at a time? Check that their existing score equals their target score plus one.
I am making a bot on Dialogflow with a Fulfillment. Considering the given strict 5-second window in DialogFlow, I am getting [empty response] as a response.
I want to overcome this issue, but my web service requires more than 9 seconds for the execution.
I am considering to redesigning the conversation flow in such a way that we will start streaming audio till the Response is processed.
Example:
User Question: xx xxx xxx xxxx xxxxx?
Response: a). We'll play fixed audio to keep the user engaged for few seconds till it finds a response text in the back end; b).
Receive answers from the web service and save them in the session to
display further.
How can I achieve this and how can I handle the Timeout issue?
You're on the right track, but there are a number of other things to consider.
First, however, keep in mind that anything that is trying to "avoid" the 5 second timeout already indicates some issues with the design. Waiting 10 seconds for a reply is a pretty long time with something as interactive as voice! Even 5 seconds, which is the timeout, is a long time. (And there is no way to change this timeout.)
So the first thing you may want to do is consider if there is a better/faster way to do what you want.
If not, the rough approach would be something like this:
Get the request from the user.
Track a unique identifier, either tied to the user or tied to the session. You'll be using this as a key into some kind of database or data store.
Start the API call as part of an asynchronous request or in another thread.
Reply immediately that you're working on it in a way that the user will send another request. (See below for this issue.) You'll want to make sure that the ID is maintained as part of this session - so you'll need to save it as part of the Session data.
At this point - you're basically doing two things in parallel.
When the API call completes, it needs to save the result in the datastore against the identifier. (It can't save it in the session itself - that response was already sent back to the Assistant.)
You're also waiting for a reply from the user. When it comes in:
Check to see if you have a response saved for this session yet.
If not, then go back to step 4. (You may want to track how many times you get here and give up at some point.)
If you do have the result, reply to the user with the information.
There is an issue with how you reply in step 4, since you want to do something that will guarantee you another request from the person expecting an answer. There are a few possible approaches:
The most straightforward way would be to send back a Media response to play a few seconds of "hold music". This has the advantage that, when the music stops, it will send an event to Dialogflow which you can capture as an Intent and then continue with step 5.
But there are some problems:
Not all versions of the Assistant support the Media response. You will need to check to confirm the feature is supported before you use it and, if not, use another approach (see below).
The media player that is presented on some Assistants allow the user to stop playback, or will not correctly send an event when the audio stops in some situations. So you may never get another request in this session.
Another approach involves some more advanced conversation design tricks, so may not always be suitable for your conversation. Your response can say that you're looking up the results but then ask the user a question - possibly one that is related to other information that you will need. With their reply, you can collect this information (if you need it) and then see if you have a result yet.
In some conversations - this works really well. For example, if you're looking up flights to somewhere, while you're looking that up you might ask them if they will need a hotel or rental car, which you might ask about anyway.
Other conversations, however, don't easily have such questions. In these cases, you may need to ask something that isn't relevant while you stall for time.
Our application creates/updates database entries based on an external service's webhooks. The webhook sends the external id of the object so that we can fetch more data for processing. The processing of a webhook with roundtrips to get more data is 400-1200ms.
Sometimes, multiple hooks for the same object ID are sent within microseconds of each other. Here are timestamps of the most recent occurrence:
2020-11-21 12:42:45.812317+00:00
2020-11-21 20:03:36.881120+00:00 <-
2020-11-21 20:03:36.881119+00:00 <-
There can also be other objects sent for processing around this time as well. The issue is that concurrent processing of the two hooks highlighted above will create two new database entries for the same single object.
Q: What would be the best way to prevent concurrent processing of the two highlighted entries?
What I've Tried:
Currently, at the start of an incoming hook, I create a database entry in a Changes table which stores the object ID. Right before processing, the Changes table is checked for entries that were created for this ID within the last 10 seconds; if one is found, it quits to let the other process do the work.
In the case above, there were two database entries created, and because they were SO close in time, they both hit the detection spot at the same time, found each other, and quit, resulting in nothing being done.
I've thought of adding some jitter'd timeout before the check (increases processing time), or locking the table (again, increases processing time), but it all feels like I'm fighting the wrong battle.
Any suggestions?
Our API is Django 3.1 with a Postgres db
Okay, this might not be a very satisfactory answer, but it sounds to me like the root of your problem isn't necessarily with your own app, but the webhooks service you are receiving from.
Due to inherent possibility for error in network communication, webhooks which guarantee delivery always use at-least-once semantics. A sender that encounters a failure that leaves receipt uncertain needs to try sending the webhook again, even if the webhook may have been received the first time, thus opening the possibility for a duplicate event.
By extension, all webhook sending services should offer some way of deduplicating an individual event. I help run our webhooks at Stripe, and if you're using those, every webhook sent will come with an event ID like evt_1CiPtv2eZvKYlo2CcUZsDcO6, which a receiver can use for deduplication.
So the right answer for your problem is to ask your sender for some kind of deduplication/idempotency key, because without one, their API is incomplete.
Once you have that, everything gets really easy: you'd create a unique index on that key in the database, and then use upsert to guarantee only a single entry. That would look something like:
CREATE UNIQUE INDEX index_my_table_idempotency_key ON my_table (idempotency_key);
INSERT INTO object_changes (idempotency_key, ...) VALUES ('received-key', ...)
ON CONFLICT (idempotency_key) DO NOTHING;
Second best
Absent an idempotency ID for deduping, all your solutions are going to be hacky, but you could still get something workable together. What you've already suggested of trying to round off the receipt time should mostly work, although it'll still have the possibility of losing two events that were different, but generated close together in time.
Alternatively, you could also try using the entire payload of a received webhook, or better yet, a hash of it, as an idempotency ID:
CREATE UNIQUE INDEX index_my_table_payload_hash ON my_table (payload_hash);
INSERT INTO object_changes (payload_hash, ...) VALUES ('<hash_of_webhook_payload>', ...)
ON CONFLICT (payload_hash) DO NOTHING;
This should keep the field relatively small in the database, while still maintaining accurate deduplication, even for unique events sent close together.
You could also do a combination of the two: a rounded timestamp plus a hashed payload, just in case you were to receive a webhook with an identical payload somewhere down the line. The only thing this wouldn't protect against is two different events sending identical payloads close together in time, which should be a very unlikely case.
If you look at the acquity webhook docs, they supply a field called action, which key to making your webhook idempotent. Here are the quotes I could salvage:
action either scheduled rescheduled canceled changed or order.completed depending on the action that initiated the webhook call
The different actions:
scheduled is called once when an appointment is initially booked
rescheduled is called when the appointment is rescheduled to a new time
canceled is called whenever an appointment is canceled
changed is called when the appointment is changed in any way. This includes when it is initially scheduled, rescheduled, or canceled, as well as when appointment details such as e-mail address or intake forms are updated.
order.completed is called when an order is completed
Based on the wording, I assume that scheduled, canceled, and order.completed are all unique per object_id, which means you can use a unique together constraint for those messages:
class AcquityAction(models.Model):
id = models.CharField(max_length=17, primary_key=True)
class AcquityTransaction(models.Model):
action = models.ForeignKey(AcquityAction, on_delete=models.PROTECT)
object_id = models.IntegerField()
class Meta:
unique_together = [['object_id', 'action_id']]
You can substitute the AcquityAction model for an Enumeration Field if you'd like, but I prefer having them in the DB.
I would ignore the change event entirely, since it appears to trigger on every event, according to their vague definition. For the rescheduled event, I would create a model that allows you to use a unique constraint on the new date, so something like this:
class Reschedule(models.Model):
schedule = models.ForeignKey(MyScheduleModel, on_delete=models.CASCADE)
schedule_date = models.DateTimeField()
class Meta:
unique_together = [['schedule', 'schedule_date']]
Alternatively, you could have a task specifically for updating your schedule model with a rescheduled date, that way it remains idempotent.
Now in your view, you will do something like this:
from django.db import IntegrityError
ACQUITY_ACTIONS = {'scheduled', 'canceled', 'order.completed'}
def webhook_view(request):
validate(request)
action = get_action(request)
if action in ACQUITY_ACTIONS:
try:
insert_transaction()
except IntegrityError:
return HttpResponse(200)
webhook_task.delay()
elif action == 'rescheduled':
other_webhook_task.delay()
...
I've come up with a fancy issue of synchronization in node.js, which I've not able to find an elegant solution:
I setup a express/node.js web app for retrieving statistics data from a one row database table.
If the table is empty, populate it by a long calculation task
If the record in table is older than 15 minutes from now, update it by a long calculation task
Otherwise, respond with a web page showing the record in DB.
The problem is,
when multiple users issue requests simultaneously, in case the record is old, the long calculation task would be executed once per request, instead of just once.
Is there any elegant way that only one request triggers the calculation task, and all others wait for the updated DB record?
Yes, it is called locks.
Put an additional column in your table say lock which will be of timestamp type. Once a process starts working with that record put a now+timeout time into it (by the rule of thumb I choose timeout to be 2x the average time of processing). When the process stops processing update that column with NULL value.
At the begining of processing check that column. If the value > now condition is satisfied then return some status code to client (don't force client to wait, it's a bad user experience, he doesn't know what's going on unless processing time is really short) like 409 Conflict. Otherwise start processing (also ideally processing takes place in a separate thread/process so that user won't have to wait: respond with an appropriate status code like 202 Accepted).
This now+timeout value is needed in case your processing process crashes (so we avoid deadlocks). Also remember that you have to "check and set" this lock column in transaction because of race conditions (might be quite difficult if you are working with MongoDB-like databases).
https: xyz dot com/authenticate/cc.php
This is the page 3 of my registration....where my members enter their credit card. This is tied into my authorize.net account.
One thing i noticed recently is this page can be accessed on its own just by typing in the URL.. there is no required pre-url that leads up to it. This seems unsafe, but regardless if someone wanted to abuse it they could just go through the registration process and keep submitting incorrect CC numbers.. costing me money right?
I dont remember if we put an IP limit on it, or again if that is even 100% safeguard.
I am pretty sure we did something where if they enter mastercard with their number(temporarily stored) and it gets sent back as invalid it will match that and not allow them to keep entering the wrong 16 digit number.
Should i just leave the page accessible without specific pages allowing access and worrying about IP limits instead? Couldn't someone just keep switching their ip and submitting this page with incorrect CC's or fake ones at that?
What is the proper way to secure this page considering i/losing my merchant account could be the one at risk?
Thank you in advance
It is strange that you would allow direct acces to the 3rd step of the process, where is all the other data like the user name, addres, ...?
This are some ideas of what I would do, a completely secure system (which might not exists) would be much more complex than my simple steps.
note that you probably would like to first allow the users to register with some information where you can know who they are (verified email, verified phone number, etc) then and only then, you do the credit card thing, and if they continuosly input wrong or invalid numbers, you can do something else, like black-listing them, call them, insult them, etc..
note 2 I spent long time writting this, the more I read it and think about it, the worse it seems to be, but as it is already written I'll post it anyway.
Some notes before begin:
There is only one address, for example /authenticate/auth.php
The process has a "state" and depending on this, it will show/do different things.
For different states has other extra files which are included depending on the state.
After the process starts, a session is created and linked with the user IP, the process state and any other identifiable information about the user, for example 'User-Agent', this data is saved in the server.
Seems you would like to show a different state using different pages, so it will be like that. But actually I would do in a single page using ajax calls.
There is NO black-listing of suspicious IP addresses(too many normal or buggy or completely wrong requests), it could be added if desired, but the complexity increases. You might or might not want to do this, maybe a capcha would be enough, but..
There is NO capcha which might help in some cases, but the session handling I describe here might need to change.
There is NO email verification which you probably want to do.
Let's say that the process states are ask_name, ask_address, ask_cc, etc...
So, when there is any request to the auth page (/authenticate/auth.php), this is what we could do:
1 If 'Referer' doesn't come from one of the possible process starters (main page, etc,) or this page (/authenticate/auth.php), we redirect to the main page. end.
This first step avoids people writting the address directly or coming from untrusted pages.
2 If there is no session information for this request:
2.1 If there is a 'user_name' parameter AND 'Referer' is this page (/authenticate/auth.php)
2.1.1 If that user name is already registered, show(include, not redirect) 'ask_name.php' with the extra notice "User already registered". end.
2.1.2 Create a session for this user, link it with it's IP, User Agent, etc, other data.
2.1.3 Set the state to ask_address (the second) and show 'ask_address.php'. end.
2.2 Else (no parameter or 'Referer' was wrong)
2.2.1 Show 'ask_name.php'. end.
This second step either shows the first screen (ask_user) or the second (ask_name), it delays the creation of the session until we are sure the user wants to do something real.
It has a couple of problems:
Some user (or program) continuosly sends requests without session but with 'user_name', so forcing you to always check if the user is valid or not, and may slow things down. This could be avoided using several different techniquest, for example using a capcha or by black listing some IPs for some time.
It could be possible that one user start the process with a 'user_name' which doesn't exists, but he is slow and takes some time to finish the process, while this is happening, a second user begins and finish the process with the same 'user_name', so when the first user is going to finish, it will fail at the last step. This could be avoided with several different techniques which are left as an exercice.
3 If there is session information for this requests (this is the else to the previous step)
3.1 If referer is not this page OR the IP stored in the server is not the same as the current request OR some other data like User Agent is different OR the state is invalid (not in the list of states), remove the session id from the request (so the browser deletes it) and show 'ask_name.php' with the extra notice "Looks like your device changed!!!". end.
3.2 'include' the page for the state:
3.2.1 If the parameters are passed and are correct, set the state to the-next-state and show the page for it. If is the last state do something appropiate for the last state. end.
3.2.2 Show the same page for this state with an error message for the user to retry. end.
This last step tries to ensure that the request is not coming from a different computer and/or with stolen session keys.