Okay, first of all I want to tell you that I am new to all this techniques mentioned in the title.
I want to make an new app. Think of it as a real time trading engine (like for stocks for example).
So, there are two things that really matter:
Speed / Performance: Everyone has to see trades in realtime
Security: Same trades can be made simultaneously but only one can be successful
I thought about an approach like this:
If a user wants to buy 10 peaces of stock X for $100 each he places an order which I store with Redis (speed) and push it to all clients with socket.io. Well, as soon as another user wants to sell 15 peaces for $100 the script should check if there is an open buy order. If so, it saves it as a successfull transaction in MongoDB (persistance) and closes the buy order of 10 peaces.
In this example 5 peaces are left. The script would display that with a calculation like this: 15 (sell at $100) minus 10 (buy at $100) equals 5 left. Every time someone want's to trade something this calculation would be made because I don't know how many stocks are left for trading else.
Edit: Or I could subtract 10 peaces of the 15 peaces in Redis so that I don't need to calculate every time. But if something would go wrong, I wouldn't know what the original data was. That's a problem.
Now the questions are:
Would you make it like this? Better ideas maybe?
What would happen if two users make the exact same order in the same time? Could it happen, that it gets stored two times in MongoDB as different successfull transactions? Of course you could run an audit over Redis and MongoDB and compare it. But that would be a horrible solution.
Hope you understand what I'm trying to ask. Thanks in advance!
First of all if you do not know anything from the stack you are using, it is not a good idea to tell I need high performance (high availability, good security and so on). Being absolutely new to all the tools you are using you should be happy if it will just work.
As for your question: first of all take a look how other people have done similar things. Here is an open source bitcoin trading engine which uses node.js which makes it an excellent example to study (it is complex, so take a deep breath). If you want to use mongo you need to know that it does not support transactions, so you need to take a look how to implement them by yourself there. These two examples are really good in explaining it.
Related
I plan on creating an ticket "pass" platform. Basically, imagine you come to a specific city, you buy a "pass" for several days (for which you get things like free entrance to museums and other attractions).
Now, the main question that bothered me for several days is: How will museum staff VALIDATE if the pass is valid? I see platforms like EventBrite etc. using barcodes/QR codes, but that is not quite a viable solution because we'll need to get a good camera phone for every museum to scan the code and that's over-budget. So I was thinking of something like a simple 6-letter code, for eg: GHY-AGF. There are 26^6 = 308 million combinations, which is a tough nut to crack.
I've asked a question on the StackExchange security site about this, and the main concern was the brute forcing. However, I imagine someone doing this kind of attack if: they had access of doing pass lookup. The only people that will be able to do this are:
1) The museum staff (for which there will be a secure user/pass app, and rate limits of no more than 1000 look-ups per day)
b) Actual customers to check the validity of their pass, and this will be protected with Google ReCaptcha v3, which doesn't sacrifice user experience like with v1. Also rate limits and IP bans will be applied
Is a brute force STILL a viable attack if I implement these 2 measures in place? Also, is there something else I'm missing in terms of security, when using this approach?
By the way, Using a max. 6-character-long string as a unique "pass" has many advantages portable-wise, for eg. you could print "blank" passes, where the user will be give instructions on how to obtain it. After they pay, they'll be given a code like: GAS-GFS, which they can easily write with a pen on the pass. This is not possible with a QR/barcode. Also, the staff can check the validity in less than 10 seconds, by typing it in a web-app, or sending an SMS to check if it's valid. If you're aware of any other portable system like this, that may be more secure, let me know.
Brute forcing is a function of sparseness. How many codes at any given time are valid out of how large a space? For example, if out of your 308M possibilities, 10M are valid (for a given museum), then I only need ~30 guesses to hit a collision. If only 1000 are valid, then I need more like 300k guesses. If those values are valid indefinitely, I should expect to hit one in less than a year at 1000/day. It depends on how much they're worth to figure out if that's something anyone would do.
This whole question is around orders of magnitude. You want as many as you can get away with. 7 characters would be better than 6 (exactly 26x better). 8 would be better than that. It depends on how devoted your attackers are and how big the window is.
But that's how you think about the problem to choose your space.
What's much more important is making sure that codes can't be reused, and are limited to a single venue. In all problems like this, reconciliation (i.e. keeping track of what's been issued and what's been used) is more important than brute-force protection. Posting a number online and having everyone use it is dramatically simpler than making millions of guesses.
So, I have a service, which as one of its features allows my clients to optimize their driving routes for the day. Normally, they only have a dozen or so stops before returning to base, so I just use the MapQuest API (paid) which has an optimize route function. However, I just got a new client who has 40+ stops per day. However, the MapQuest API only allows 25 stops (start, 23 waypoints, end) with route optimization. So, does anyone have any ideas how I can best attack the problem of trying to optimize a route for 40+ stops?
So, yes, I know that the traveling salesman problem is a computationally difficult problem. The MapQuest API is super fast with the limited number of stops that they allow, and I have a paid subscription, so I can make multiple calls in a row without getting in trouble with them. So, some ideas that I've played around with is simply dividing the route in half, optimizing each half, and combining, but it seems to be lacking in efficacy. So, if anyone has tackled this, I'd love to hear your solution.
I know this team has worked on it and they can probably help you out.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
This thread is a follow up to my previous one. It's in fact 2 questions, so I hope no one minds, as they are dependent on each other.
We are starting a new project at work and we consider it as a great opportunity to try Agile techniques in action. We had a brainstorming about ideas we read in several books and articles, and came up with concept that would suit us the best: 2 weeks iteration, followed by call with clients who would choose what stuff they want to have in next iteration. I just have few more questions, which we couldn't figure out ourselves.
What to do in the first iteration?
What to, generally, do in the first few iterations if we start from the scratch? Just give it a month of development to code core of the application or start with simple wire-frames with limited pre-coded functionality? What usually clients want to see? Shiny stuff that doesn't work or ugly stuff that does work?
How to communicate with clients?
Our initial thought it to set the process to something like this:
alt text http://img690.imageshack.us/img690/2553/communication.png
Is it a good idea to have a Focal Point on client side or is it better to communicate straight with all the clients to prevent miscommunication?
Any thoughts are welcome! Thanks in advance.
In my opinion, a key success factor for agile development is to focus on delivering value for the customer in each iteration. I would definitely pick "ugly stuff that does work" over "shiny stuff that doesn't work". Doing shiny UIs and trying to get the client to understand hat business logic takes a lot of time to implement is always risky which Joel Spolsky has written a good article about.
If the client wants enhancements to the UI, they can always put that as a requirement for the next iteration.
Regarding communication with clients I think that your scetch should be slightly adjusted. Talking in scrum terms your "focal point" is called "product owner". Having one person coordinating with the clients is good, as it can take quite a lot of time to get the different stakeholders agree on the needs. However the product owner (or focal point) should be in direct contact with the developer, without going through the project manager. In fact, the product owner and the project manager has quite distinct roles that gain a lot by being split on two people.
The product owner is the stakeholders' voice to the development team. The project manager on the other hand is responsible for the wellbeing of the project team and often keeps track of budget etc. These roles sometimes has opposing agendas, and having them split on two people gives a healthy opportunity for negotiation between conflicting interests. If one person has both roles, that person often tend to favour one of them, automatically reducing the other one. You don't want to work on a team where the project manager always puts the client before the team's needs. On the other hand no customer wants a product owner that always puts the team's needs first, neglegting the customer. Splitting the responsibilities on two people helps to remedy that situation.
I'd agree with Anders answer. My one extra observation is that many clients find it impossible to ignoire the Ugly. They get concerned about presentation rather than function. Hence you may need to bite the bullet and do at least one "Nice" screen to show that you will pay attention to presentation details.
What to, generally, do in the first few iterations if we start from the scratch?
Many teams use an Iteration Zero to:
setup the development infrastructure (source control, development machines, the automated build, a continuous integration process, a testing environment, etc),
educated the customer and agree with him on the methodology,
create an initial list of features, identify the most important and do an initial estimation,
define time of meetings (planning meeting, demo, retrospective), choose the the iteration length.
Iteration Zero is very special because it doesn't deliver any functionality to the customer but focus on what is necessary to run the next iterations in an agile way. But subsequent iterations should start to deliver value to the customer.
Just give it a month of development to code core of the application or start with simple wire-frames with limited pre-coded functionality?
No, don't develop the core of your application during one month. Instead, start delivering vertical slice of the application (from the UI to the database) immediately, not horizontal slices. This doesn't mean that a screen has to be complete (e.g. implement only one search field in a search screen) but it should ideally be representative of the final look & feel (unless you agreed with the customer on an intermediate step). The important part is to build things that provide immediate value to the customer incrementally.
What usually clients want to see? Shiny stuff that doesn't work or ugly stuff that does work?
To my experience, they want to see demonstrable progresses and you want to get feedback as soon as possible.
Is it a good idea to have a Focal Point on client side or is it better to communicate straight with all the clients to prevent miscommunication?
You need one person to represent the clients (who is called the Product Owner in Scrum):
he provides a single authoritative voice
he has a perfect knowledge of the business (i.e. he can answer questions)
he knows how to maximize the ROI (i.e. how to prioritize functionalities)
Agile generally wants to provide the client something valuable, quickly.
So I certainly would not spend "month of development to code core of the application". To me, that smells of the "big up front design" anti-pattern. Also, see YAGNI.
Get as much information from the clients about what they need soonest, and implement that in your first iteration. "Valuable" is in the eye of the client. Thet will know if they want to see slick UI (maybe they want to give a slide show about the product at a trade show, so functionality can be fake) or simple working features (maybe you're developing something that they need to start using ASAP). Business Value is what they say will help them do their job.
I'd make my iterations as short as I can (your 2 weeks could work, I suggest considering 1 week) If you absolutely can't have your dev team and your clients co-located, instead of having a call with the clients, I suggest a meeting. Demo what you've done over the previous iteration and solicit feedback about what should stay, what should change, and what should be added.
As others have said, your "Focal point" sounds like a Product Owner. What worries me about your drawing is if it is meant to imply that devs don't interact with the PO or the clients. One thing that makes Agile work is when there is lots of communication. Having communication to/from the dev team always filtered through the Project Manager is almost certainly bound to result in miscommunication, unnecessary work, and missed details.
I agree with the two answers given but I would just add one thing from personal experience. Are your customers bought in to the change towards quick iterations? As well as providing feedback after each iteration which is going to require the customer performing usability tests on each feature.
Now I don't know what your groups relationship is with your customer but its not unusual for customers to take a "Put request in - get working system out" attitude in that they are enthusiastic when giving requirements but not so forthoming with time when it comes to testing the feature.
Now this may be totally inappropriate to your situation but its always worth considering how your customer workflow will have to change as well as your groups.
Cheers
I'm writing a phonebook search, that will query multiple remote sources but I'm wondering how it's best to approach this task.
The easiest way to do this is to take the query, start a thread per remote source query (limiting max results to say 10), waiting for the results from all threads and aggregating the list into a total of 10 entries and returning them.
BUT...which of the remote source is more important if all sources return at least 10 results, so then I would have to do a search on the search results. While this would yield accurate information it seems inefficient and unlikely to scale up well.
Is there a solution commercial or open source that I could use and extend, or is there a clever algorithm I can use that I've missed?
Thanks
John, I believe what you want is federated search. I suggest you check out Solr as a framework for this. I agree with Nick that you will have to evaluate the relative quality of the different sources yourself, and build a merge function. Solr has some infrastructure for this, as this email thread shows.
To be honest I haven't seen a ready solution, but this is why we programmers exist: to create a solution if one is not readily availble :-)
The way I would do it is similar to what you describe: using threads - if this is a web application then ajax is your friend for speed and usability, for a desktop app gui representation is not even an issue.
It sounds like you can't determine or guess upfront which source is the best in terms of reliability, speed & number of results. So you need to setup you program so that it determines best results on the fly. Let's say you have 10 data sources, and therfore 10 threads. When you fire up your threads - wait for the first one to return with results > 0. This is going to be you "master" result. As other threads return you can compare them to your "master" result and add new results. There is really no way to avoid this if you want to provide unique results. You can start displaying results as soon as you have your first thread. You don't have to update your screen right away with all the new results as they come in but if takes some time user may become agitated. You can just have some sort of indicator that shows that more results are available, if you have more than 10 for instance.
If you only have a few sources, like 10, and you limit the number of results per source you are waiting for, to like 10, it really shouldn't take that much time to sort through them in any programming language. Also make sure you can recover if your remote sources are not available. If let's say, you are waiting for all 10 sources to come back to display data - you may be in for a long wait, if one of the sources is down.
The other approach is to f00l user. Sort of like airfare search sites do - where they make you want a few seconds while they collect and sort results. I really like Kayak.com's implementation - as it make me feel like it's doing something unlike some other sites.
Hope that helps.
I wish to create a college project on a simple online multiplayer management game which will involve players setting orders for the day/week and then obtaining profits. Being a relative beginner I am unable to figure out the architecture required for this task.
As far as I am concerned I would be needing the following things:
A text interface to display the status of ongoing events and to set orders in a web browser.
A certain application that would calculate the results every minute and update the database.
A database
Sorry for being so newbish, but any advice or links or books on how to proceed will do.
Please comment if any more information is required.
Any programming language would be fine. Pick a lang / arch you or someone in your group are familiar with. I'm mostly a PHP/ZF, Linux, Postgres guy. So I would...
Write a little ZendFramework app to collect your user's data and save to postgres database. I'd host it on a little Linux server. I like slicehost.com $20/mon, but there are cheaper. Or make friends with someone with a server.
Then for the update of the orders, use a cron job to run every minute. If the update process is complex, use another PHP script, else just straight SQL.
Why do you need to run updates every minute? Are people going to be updating it that often, if they are making orders for a day or week?
I would start with deciding on the equations that will be used in your model.
Then, that will help decide what you need in the database, to give the parameters to the model.
Then, once you have the database, you need to get information from the user, so decide what you need from the user.
For example you should have some random event that will make certain items go up or down in demand, or have resources become more common.
So, you may want to have information in the database that lists what each product is composed of.
If the model will have external information, or, if it is based on what others make, so, for example, last week shoes were not produced, so those that made shoes made a profit. This week everyone is making shoes, so there is too many, so the price went down.
This is why I think starting with your model, and testing your assumptions is the first step.
Any language, system, database will work well, just do what you feel comfy with. When you design the UI, do you want it to look fine on iphones and the Blackberry Razor? Then that will have a big impact on how you design the UI.