Does anyone can provide any useful information to complete the publish/subscribe protocol "Scribe" using peersim? I already have the paper and the presentation, not very helpful for what I need, I understand the algorithm, but my problem is mainly with peersim.
I have some minor problems with handling the send and receive. I have already built a peer sampling service, over t-man. T-man always gives a list which includes the nodes, that a peer (or better myself, for each cycle) knows.
Related
It's well known that botting is one of the most great thread for MMORPG games. Since it's releatively easy to detect clint injection, I wonder how can MMORPG detect botting from server side. Thanks for any help.
By reading some papers, I figure this question by myself.
Here are two kind of major bot detection methods: detected by Sufficient Condition and detected by Necessary Condition.
For Sufficient Condition, it's always useful to detect behavioral action or social action.
For Necessary Condition, it's usually useful to detect Transaction Network Analysis.
The API provides the emit/send callback mechanism to acknowledge received messages. However, this callback doesn't get fired in case of disconnect or error. It appears to me that upon a disconnection one would need to go through some rather messy procedures to clean up outstanding sent messages (e.g. - assume a use case where you may want to store messages for forwarding later, etc.). Any simple ideas out here on how to accomplish this? Wondering if I'm missing something.... Thanks.
The Real Underlying Issue
This issue isn't just limited to socket.io. It is a well known problem called the Two Generals' Problem.
Two armies, each led by a general, are preparing to attack a fortified city. The armies are encamped near the city, each on its own hill. A valley separates the two hills, and the only way for the two generals to communicate is by sending messengers through the valley. Unfortunately, the valley is occupied by the city's defenders and there's a chance that any given messenger sent through the valley will be captured (this scenario assumes that while the two generals have agreed that they will attack, they haven't agreed upon a time for attack before taking up their positions on their respective hills).
You are trying to reach Common Knowledge over an unreliable link.
At any stage of the communication over socket.io the link can be broken, and a callback can be sent but the other side could not be sure that it arrived.
What Can Be Done
You need to embrace the fact this is always a possibility. There is no trivial solution for this. This problem and its generalization are still actively studied in fields like Multi-Agent Systems research.
What can still be done in your specific case
There are some common approaches to mitigate this issue.
What I did when designing an application using socket.io is attach IDs to messages, if a disconnect happens and one side tried to send an already-sent message, the receiving side will be aware that the message was already received.
Note that in practice you don't need to do this everywhere.
More Reading on the Issue
Related question here in SO
The Byzantine Generals Problem by Microsoft Research (LESLIE LAMPORT, ROBERT SHOSTAK, and MARSHALL PEASE) which discusses the problem and suggests some solutions
Acclaimed Book by Yoav Shoham that talks about Multiagent systems and how they address this issue.
Blog post about this problem in TCP.
I want to collect statistics from the spreading of a file in a new bittorrent swarm without actually downloading anything (or as little as possible). I need to know which peer has which pieces (to make file based statistics) knowing the number of seeders and leechers or percentages is not enough. Later when there are many peers I need to download the data to determine what it is. This part can be done with a regular torrent client.
I do not plan to implement the protocol myself so I looked at 2 implementations libtorrent and ktorrent's libbtcore. Neither is capable of collecting data while not downloading there are simply no connected peers when there is nothing to download. Libtorrent is simpler but ktorrent looks better commented.
I see 3 possibilities:
Use some application exactly for this. Are there any?
Modify a torrent implementation to do what I want. Is anyone familiar with them? Where to start?
Implement a small subset of the protocol. Just periodically ask the peers what they have. Is this feasible or would the program need to support almost the full protocol?
What do you recommend?
This is an old question, but perhaps this answer might be useful for others.
Use some application exactly for this. Are there any?
Not that I know of.
Modify a torrent implementation to do what I want. Is anyone familiar with them? Where to start?
I'm only familiar with the BitTornado core (that is used in e.g. ABC). It is written in Python, but it's an architectural mess.
However, you could just take any implementation and start stripping it from unnecessary functionality.
Implement a small subset of the protocol. Just periodically ask the peers what they have. Is this feasible or would the program need to support almost the full protocol?
Note that you cannot "ask" a peer what they have. The other peer informs you whenever it wants about the pieces it has (so it's push instead of pull). After the BitTorrent handshake, a peer may send a bitfield of pieces it has. Afterwards it may send HAVE messages informing you it has acquired a new piece. Also note that peers may lie about the pieces they have. Examples include superseeding peers and freeriding clients like BitThief.
If you want to implement a small subset of the protocol, you'd need at the bare minimum implement the BitTorrent handshake message and preferably the extended handshake message. The latter allows you to receive (and send) uTorrent PEX messages. PEX is useful to quickly discover other peers in the swarm.
For your statistics gathering purposes, you additionally need to support the bitfield and HAVE messages.
Background
Terminals are a compination of hardware and software. Terminal's main responsibility is to
- collect data (with it's sensors)
- process and transmit collected data to data server over the Internet.
The terminal has Internet access either via WLAN or GPRS. Terminal are running embedded Linux.
Things to consider, security perspective
Transmission of collected data over the air to data server.
Remote software updates over the air (is controlled by the data server),
Local software updates
Identification and authentication of terminal and server
What else should be considered in this type of system?
My question is divided in 3 parts.
Firstly, what kind of issues should be thought about when thinking about security with this kind of system.
Secondly, what ciphers, key exchange mechanism and security techniques could be applied in different parts of the answer of the first question.
And lastly, is there any good books/resources available covering this matter. Specifically targeting this type of application area or similar with practical advice on solutions.
I know my questions are little bit out there. I'm familiar with different ciphers (symmetric and asymmetric), but have found particularly difficult to find any pratical guidance in implementing security in real world systems. I hope this questions hits some traffic. I'm sure there are many of us out there facing similar challenges.
I can provide more details, just point me out where more information is required.
The actual important question is the first part of what your question. "What kind of Issues should be thought about." Only you can really determine that, via two tools: A threat model, and a security model.
A threat model is about who is trying to attack the system: Script Kiddies? Skilled Organized Crime Hackers? The evil overlord's government?
A security model describes what you are trying to protect. Should anybody be able to read the data? Should you be able to detect injection of false data?
First come up with a plan on what your requirements are, then look for technical solutions.
I need to set up a protocol for fast command/response interactions. My instinct tells me to just knock together a simple protocol with CRLF separated ascii strings like how SMTP or POP3 works, and tunnel it through SSH/SSL if I need it to be secured.
While I could just do this, I'd prefer to build on an existing technology so people could use a friendly library rather than the socket library interface the OS gives them.
I need...
Commands and responses passing structured data back and forth. (XML, S expressions, don't care.)
The ability for the server to make unscheduled notifications to the client without being polled.
Any ideas please?
If you just want request/reply, HTTP is very simple. It's already a request/response protocol. The client and server side are widely implemented in most languages. Scaling it up is well understood.
The easiest way to use it is to send commands to the server as POST requests and for the server to send back the reply in the body of the response. You could also extend HTTP with your own verbs, but that would make it more work to take advantage of caching proxies and other infrastructure that understands HTTP.
If you want async notifications, then look at pub/sub protocols (Spread, XMPP, AMQP, JMS implementations or commercial pub/sub message brokers like TibcoRV, Tibco EMS or Websphere MQ). The protocol or implementation to pick depends on the reliability, latency and throughput needs of the system you're building. For example, is it ok for notifications to be dropped when the network is congested? What happens to notifications when a client is off-line -- do they get discarded or queued up for when the client reconnects.
AMQP sounds promising. Alternatively, I think XMPP supports much of what you want, though with quite a bit of overhead.
That said, depending on what you're trying to accomplish, a simple ad hoc protocol might be easier.
How about something like SNMP? I'm not sure if it fits exactly with the model your app uses, but it supports both async notify and pull (i.e., TRAP and GET).
That's a great question with a huge number of variables to consider, and the question only mentioned a few them: packet format, asynchronous vs. synchronized messaging, and security. There are many, many others one could think about. I suggest going through a description of the 7-layer protocol stack (OSI/ISO) and asking yourself what you need at those layers, and whether you want to build that layer or get it from somewhere else. (You seem mostly interested in layer 6 and 7, but also mentioned bits of lower layers.)
Think also about whether this is in a safety-critical application or part of a system with formal V&V. Really good, trustworthy communication systems are not easy to design; also an "underpowered" protocol can put a lot of coding burden on application to do error-recovery.
Finally, I would suggest looking at how other applications similar to yours do the job (check open source, read books, etc.) Also useful is the U.S. Patent Office database, etc; one can get great ideas just from reading the description of the communication problem they were trying to solve.