The API provides the emit/send callback mechanism to acknowledge received messages. However, this callback doesn't get fired in case of disconnect or error. It appears to me that upon a disconnection one would need to go through some rather messy procedures to clean up outstanding sent messages (e.g. - assume a use case where you may want to store messages for forwarding later, etc.). Any simple ideas out here on how to accomplish this? Wondering if I'm missing something.... Thanks.
The Real Underlying Issue
This issue isn't just limited to socket.io. It is a well known problem called the Two Generals' Problem.
Two armies, each led by a general, are preparing to attack a fortified city. The armies are encamped near the city, each on its own hill. A valley separates the two hills, and the only way for the two generals to communicate is by sending messengers through the valley. Unfortunately, the valley is occupied by the city's defenders and there's a chance that any given messenger sent through the valley will be captured (this scenario assumes that while the two generals have agreed that they will attack, they haven't agreed upon a time for attack before taking up their positions on their respective hills).
You are trying to reach Common Knowledge over an unreliable link.
At any stage of the communication over socket.io the link can be broken, and a callback can be sent but the other side could not be sure that it arrived.
What Can Be Done
You need to embrace the fact this is always a possibility. There is no trivial solution for this. This problem and its generalization are still actively studied in fields like Multi-Agent Systems research.
What can still be done in your specific case
There are some common approaches to mitigate this issue.
What I did when designing an application using socket.io is attach IDs to messages, if a disconnect happens and one side tried to send an already-sent message, the receiving side will be aware that the message was already received.
Note that in practice you don't need to do this everywhere.
More Reading on the Issue
Related question here in SO
The Byzantine Generals Problem by Microsoft Research (LESLIE LAMPORT, ROBERT SHOSTAK, and MARSHALL PEASE) which discusses the problem and suggests some solutions
Acclaimed Book by Yoav Shoham that talks about Multiagent systems and how they address this issue.
Blog post about this problem in TCP.
Related
It's well known that botting is one of the most great thread for MMORPG games. Since it's releatively easy to detect clint injection, I wonder how can MMORPG detect botting from server side. Thanks for any help.
By reading some papers, I figure this question by myself.
Here are two kind of major bot detection methods: detected by Sufficient Condition and detected by Necessary Condition.
For Sufficient Condition, it's always useful to detect behavioral action or social action.
For Necessary Condition, it's usually useful to detect Transaction Network Analysis.
I'm writing notification system and faced a challenge of how to name an API for sending notifications over network, that will also save ones into a database. Imagine that we don't need to distinguish those two responsibilities in own methods.
I've came up with the idea of "publish", however I think this decision is biased.
Which one would you recommend?
We have been looking at automatically logging all unexpected client errors to our bug tracker. For reference our application is written in Java/GWT/Guice/Hibernate/Jetty and our bug tracker is the hosted version of FogBugz which can create bugs programatically or via an email.
The biggest problem I see with doing this is stack traces that happen in a loop overload the bug tracker by creating thousands of cases. Does anybody have a suggested way to handle automatic bug creation like this?
If you're using FogBugz bugscout (also see up-to-date docs here) then it has the ability to just increase number of occurences of same problem, instead of creating new case for same exception again and again.
Are you sure that you want to do that?
It obviously depends on your application but even by carefully taking care of the cases that could generate lots of bug reports (because of the loops) this approach could still end up filling the bug tracker.
How about this?
Code your app so that every time an exception is thrown, you gather info about the client (IP, login, app version, etc) and send that + the stack trace (or the whole exception object .ToString()) by email to yourself (or the dev team).
Then on you email client, have a filter that sorts that incoming mail and throws it in a nice folder for you to look at later.
Thus you can have tons of emails about maybe one of more issues but then you don't really care because you input the issues yourself in the bugtracker, and easily delete that ton of mail.
That's what I did for my app (which is a client-server desktop app). It plays out well in this case.
Hope that helped!
JIRA supports automated issues creation using so called services: documentation.
Does anybody have a suggested way to handle automatic bug creation...?
Well, I have. Don't do that.
What are you going to gain from that? Tester's effort? in my experience, whatever effort one can save from that was lost multiple times with overhead transferred to developers who had to analyze and maintain the automatically created tickets anyway. Not to mention overall frustration caused by that.
The least counterproductive way I can imagine would be something like establishing a dedicated bugs category or issue tracker instance, such that only testers can see and use it.
In that "sandbox", auto-created bugs could be assigned to testers who would later pass analyzed and aggregated bug reports to developers.
And even in that case, I'd recommend to pay close attention to what users (testers) say about the system. If they, say, start complaining about the system, consider trying a manual way of doing things instead.
I'm mostly looking for setup advise and pointers on how to go about going about this. I'll explain in as much detail as I can think and also note possible approaches that may be plausible.
The aim of this is to create a real time browser game, the best method that I have found for my needs would to use "long polling" with ajax, which will basically setup a request with the server that will "hang there" til the server has something to send it, then re-establish the connection upon receipt for more data. For my purposes this will handle a chat system aswell as character movement, IE: if a player enters the same area the clients there will recieve a response to inform them and thus update the browser client to show this.
The above is relatively easy to implement and I have already made a test-case for it, however I want to improve on it, on the server side it runs a loop for X amount of time before it'll auto timeout and send back and empty string, so another connection can be made, this is to prevent infinite loops and use up resources in cases where it shouldn't. Instead of looking up the database on each loop cycle (would be expensive I believe) for messages that need sending to the client, I use flatfiles, if a file has a modified timestamp greater than the last message sent to the client, then there is something new to send. However I believe this would also be expensive (not as much as using a mysql database though?) when done a couple of times per second.
My thought process on this was to have a C++ program (for speed) constantly running, and use that for very fast lookups in memory for new messages and so fourth, this would also give me the added bonus of being able to have bots within the game that the server can control for a more real-time feel/approach, however I have no clue if this is even possible and my searches on google have been fruitless.
The approach I would most love to be able to do, is to continue to use PHP to do the rendering and control of the page etc, and have the ajax requests go to the C++ application (that will always be running) that can handle all the real-time aspects.
CGI defeats the purpose of the above approach, as it creates a new instance of the application on each request, which is both slow and exactly what I do not want, I have php for that and don't want to switch one perfectally running language for another that would be better suited, PHP however (to my knowledge) can't store things in memory (ram) and so fourth.
Another approach that I have thought about was to use php sockets to connect into the C++ application, though I have no idea how feasible this may be. The C++ application only basically will need to control bots (AI) and the chat system messages.. I have absolutely no idea how to go about handling bots via PHP.
I hope this fully explains what my intentions and goals are, so if anyone has any pointers or advise then please reply and help me out, it would be very much appreciated. If you need any extra information (for if I didn't cover something or something very well) then I'll be happy to attempt to better explain.
How fast do the reactions need to be? For anything approaching real-time action games, AJAX/Comet is going to be much too slow. The overhead is also really depressing.
The way forward for that kind of thing will probably be WebSocket, with a custom server on the backend. But I don't think that means you need to resort to C[++] for this; the bottleneck is most likely going to be the network and not server processor power.
I'm using a Python SocketServer with a trivial message replication system — all the game logic in my case is on the client-side, with some complicated JavaScript maintaining a consistent game world in the face of lag — but even for a more complex server-side I think a scripting language will probably be just fine.
WebSocket isn't ready yet; there are no mainstream browser implementations. In the meantime I'm using a Flash Socket backup that emulates the WebSocket interface. Flash Sockets have their own problems in that they fail to negotiate proxies, but they are fast and hopefully the need for them will diminish as WebSocket arrives properly.
Reading your post sets alarm bells ringing.
How familiar are you with multi-threaded code? With C++? If the answer is "not very", then I fear you might be biting off a quite a large chunk. Why not take advantage of some existing (tried and tested) COMET server implementations rather than this barebones approach? Whatever application you have in mind, it should be quite separate from the comms implementation.
As someone who has implemented a such a server, I can tell you that it will take many design iterations and a helluva long time to get right. Testing such a product realisticly is also a very tricky process.
I need to set up a protocol for fast command/response interactions. My instinct tells me to just knock together a simple protocol with CRLF separated ascii strings like how SMTP or POP3 works, and tunnel it through SSH/SSL if I need it to be secured.
While I could just do this, I'd prefer to build on an existing technology so people could use a friendly library rather than the socket library interface the OS gives them.
I need...
Commands and responses passing structured data back and forth. (XML, S expressions, don't care.)
The ability for the server to make unscheduled notifications to the client without being polled.
Any ideas please?
If you just want request/reply, HTTP is very simple. It's already a request/response protocol. The client and server side are widely implemented in most languages. Scaling it up is well understood.
The easiest way to use it is to send commands to the server as POST requests and for the server to send back the reply in the body of the response. You could also extend HTTP with your own verbs, but that would make it more work to take advantage of caching proxies and other infrastructure that understands HTTP.
If you want async notifications, then look at pub/sub protocols (Spread, XMPP, AMQP, JMS implementations or commercial pub/sub message brokers like TibcoRV, Tibco EMS or Websphere MQ). The protocol or implementation to pick depends on the reliability, latency and throughput needs of the system you're building. For example, is it ok for notifications to be dropped when the network is congested? What happens to notifications when a client is off-line -- do they get discarded or queued up for when the client reconnects.
AMQP sounds promising. Alternatively, I think XMPP supports much of what you want, though with quite a bit of overhead.
That said, depending on what you're trying to accomplish, a simple ad hoc protocol might be easier.
How about something like SNMP? I'm not sure if it fits exactly with the model your app uses, but it supports both async notify and pull (i.e., TRAP and GET).
That's a great question with a huge number of variables to consider, and the question only mentioned a few them: packet format, asynchronous vs. synchronized messaging, and security. There are many, many others one could think about. I suggest going through a description of the 7-layer protocol stack (OSI/ISO) and asking yourself what you need at those layers, and whether you want to build that layer or get it from somewhere else. (You seem mostly interested in layer 6 and 7, but also mentioned bits of lower layers.)
Think also about whether this is in a safety-critical application or part of a system with formal V&V. Really good, trustworthy communication systems are not easy to design; also an "underpowered" protocol can put a lot of coding burden on application to do error-recovery.
Finally, I would suggest looking at how other applications similar to yours do the job (check open source, read books, etc.) Also useful is the U.S. Patent Office database, etc; one can get great ideas just from reading the description of the communication problem they were trying to solve.