Preliminary note
Yes, I am aware of the existence of DNS libraries and easy-to-use DNS servers. I am doing this purely for academic purposes, and to get a grasp of how DNS queries work.
The question
I was looking at RFC 1035 to get an idea of how DNS messages work. I think I more or less understood everything in that memo. There is one thing that I can't interpret autonomously, however. If you look at Section 4.1.3, here is how the resource record format is described:
1 1 1 1 1 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| |
/ /
/ NAME /
| |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| TYPE |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| CLASS |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| TTL |
| |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| RDLENGTH |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
/ RDATA /
/ /
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
In this scheme, NAME ends at the end of a two-bytes block. Does this mean that its length in bytes should always be a multiple of two? The format for names is described earlier in the RFC as a sequence of labels, terminated by a zero-length label. This means that, depending on the name, its length can arbitrarily be even or odd. So does NAME need padding to an even number of bytes? And if so, how should I pad?
I would have ignored the issue and assumed that no padding is required, if it wasn't that in Section 4.1.1 it specifies that QNAME does not need padding. Since they didn't specify the same for NAME in the answer RRs, I was wondering if I should assume that there is some difference.
There is no padding. If you have doubt, capture a query about the root zone (the name of which is a single NULL octet, so an odd length) from a root server (they can be trusted to get the protocol right as it is used in the real world) and look at it.
Related
I've been referring to RFC 4034 (https://www.rfc-editor.org/rfc/rfc4034#page-4), but there is something I quite don't understand about the flags field of a RDATA for a DNSKEY RR.
As stated in the RFC 2.1.1. The Flags Field, "Bits 0-6 and 8-14 are reserved: these bits MUST have value 0 upon creation of the DNSKEY RR and MUST be ignored upon receipt.".
But few lines under in 2.2. The DNSKEY RR Presentation Format, it states "The Flag field MUST be represented as an unsigned decimal integer. Given the currently defined flags, the possible values are: 0, 256, and 257."
How can bits 8-14 be required to have value of 0 if using 256 and 257 as the section value ?
Could someone explain to me if there is something I don't get please ?
I am currently building my own DNS server and I need to validate the format of a DNSKEY RR as well as craft packets on demand for them.
Thanks !
I am currently building my own DNS server
I hope it is mostly for learning, because otherwise this is a tricky matter!
You can (should) also study existing DNS libraries to understand implementation details.
As for the values, let us go back to the fields representation:
1 1 1 1 1 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0|0|0|0|0|0|1|0|0|0|0|0|0|0|0| => 256 (ZSK case)
|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|1| => 1 (SEP case)
|0|0|0|0|0|0|0|1|0|0|0|0|0|0|0|1| => 257 (KSK case)
Bits 8-14 are always 0, as are the 0-6 ones.
SEP alone can't happen, so value 1 is not possible, 256 and 257 are the usual one for DNSSEC, and 0 (which means bit 7 is 0 so a key but "not DNSSEC case").
Remember that most (all?) of IETF protocols are MSB, left-most bit is most significant, so bit 15 is 2⁰, bit 14 is 2¹, etc. from right to left up to bit 0 at left which encodes 2¹⁵.
You can find that repeated in RFC 1035 §2.3.2:
Whenever an octet represents a numeric quantity, the left most bit in
the diagram is the high order or most significant bit.
[..]
Similarly, whenever a multi-octet field represents a numeric quantity the left most bit of the whole field is the most significant bit. When a multi-octet quantity is transmitted the most significant octet is transmitted first.
When reading the Mark-Compact chapter on The Garbage Collection Handbook, a sequence of alternatives were presented, but most of them looked old / theoretical (for instance, the 2-finger compaction and the Lisp2 3-pass approach requiring an extra header word per object).
Is anyone aware of what algorithm does HotSpot uses when running Mark-Compact (in its old-generation, I assume)?
Thanks
Big disclaimer: I am not a GC expert/writer; all the things that are written bellow are subject to changes and some of them might be way too simplistic. Please take this with a grain of salt.
I will only speak about Shenandoah, as I think I understand it; which is not a generational GC.
There are two phases here actually: Mark and Compact. I would strongly emphases here that both are concurrent and do happen while your application is running (with some very short STW events).
And now to the details. I have explained a bit of things here, but because that answer is related to somehow a different question; I'll explain more here. I assume that traversing the graph of live objects is no news for you, after all you are reading a book about GC. As that answer explains, when the application is fully stopped (also called brought to safe-points), identifying live objects is easy. No one is changing anything under you feet, the floor is rigid and you control everything. Parallel collectors do this.
The really painful way is to do things concurrent. Shenandoah employs an algorithm called Snapshot at the beginning (that book explains it AFAIK), will call it SATB for short. Basically this algorithm is implemented like this: "I will start to scan concurrently the graph of objects (from GC roots), if anything changes while I scan, I will not alter the heap, but will record these changes and deal with them later".
The very first part that you need to question is : while I scan. How is that achieved? Well, before doing the concurrent mark, there is a STW event called Initial Mark. One of the things that gets done in that phase is to set a flag that concurrent marking has started. Later, while executing code, that flag is checked (Shenandoah thus employs changes in the interpreter). In pseudo-code:
if(!concurrentMarkingActive) {
// do whatever you were doing and alter the heap
} else {
// shenandoah magic
}
In machine code that might look like this:
test %r11, %r11 (test concurrentMarkingActive flag)
jne // concurrent marking is currently active
Now GC knows when concurrent marking is happening.
But how is concurrent marking even implemented. How can you scan the heap while the heap itself is mutated (not stable)? The floor under your feet adds more holes and removes them also.
That is the "shenandoah magic". Changes to the heap are "intercepted" and not persisted directly. So if GC performs a concurrent mark at this point in time, and application code tries to mutate the heap, those changes are recorded in each threads SATB queues (snapshot at the beginning). When concurrent mark is over, those queues are drained (via a STW event called Final Mark) and those changes that were drained are analyzed again (remember under a STW event now).
When this phase Final Mark is over GC knows what is alive and thus what is implicitly garbage.
Compact phase is next. Shenandoah is now supposed to move live objects to different regions (in a compact fashion) and mark the current region as one where we can allocate again. Of course, in a simple STW phase, this would be easy : move the object, update references pointing to it. Done. When you have to do it concurrently...
You can't take the object and simply move it to a different region and then update your references one by one. Think about it, let's suppose this is the first state we have:
refA, refB
|
---------
| i = 0 |
| j = 0 |
---------
There are two references to this instance: refA and refB. We create a copy of this object:
refA, refB
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
We created a copy, but have not updated any references yet. We now move a single reference to point to the copy:
refA refB
| |
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
And now the interesting part: ThreadA does refA.i = 5, while ThreadB does refB.j = 6 so your state becomes:
refA refB
| |
--------- ---------
| i = 5 | | i = 0 |
| j = 0 | | j = 6 |
--------- ---------
How do you merge these objects now? I'll be honest - I have no idea if that would even be possible and neither is this a route that Shenandoah took.
Instead, the solution from Shenandoah does a very interesting thing IMHO. An extra pointer added to each instance, also called forwarding pointer:
refA, refB
|
fwdPointer1
|
---------
| i = 0 |
| j = 0 |
---------
refA and refB points to fwdPointer1, while fwdPointer1 to the real Object. Let's create the copy now:
refA, refB
|
fwdPointer1 fwdPointer2
| |
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
And now, we want to switch all references (refA and refB) to point to the copy. If you look closely, this requires only a single pointer change - fwdPointer1. Make fwdPointer1 point to fwdPointer2 and you are done. This means one single change as opposed to two (in this set-up) of refA and refB. The bigger win here is that you don't need to scan the heap and find out references that point to your instance.
Is there a way to atomically update a reference? Of course : AtomicReference (at least in java). The idea here is almost the same, we atomically change the fwdPointer1 via a CAS (compare and swap), as such:
refA, refB
|
fwdPointer1 ---- fwdPointer2
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
So, refA and refB point to fwdPointer1, which now points to the copy we have created. Via a single CAS operation, we have switched concurrently all references to the newly created copy.
Then, GC can simply (concurrently) update all references refA and refB to point to the fwdPointer2. In the end having this:
refA, refB
|
fwdPointer1 ---- fwdPointer2
|
--------- ---------
| i = 0 | | i = 0 |
| j = 0 | | j = 0 |
--------- ---------
So, the Object on the left is now garbage: there are no references pointing to it.
But, we need to understand the drawbacks, there is no free lunch.
First, is obvious : Shenandoah adds a machine header that each instance in the heap (read further, as this is false; but makes understanding easier).
Each of these copies will generate an extra object in the new region, thus at some point there will be at least two copies of the same object (extra space required for Shenandoah to function, as such).
When ThreadA does refA.i = 5 (from the previous example), how does it know if it should try to create a copy, write to that copy and CAS that forwarding pointer vs simply do a write to the object? Remember that this happens concurrently. Same solution as with concurrentMarkingActive flag. There is a flag isEvacuationToADifferentRegionActive (not the actual name). If that flag is true => Shenandoah Magic, else simply do the write as it.
If you really understood this last point, your natural question should be :
"WAIT A SECOND! Does this mean that Shenandoah does an if/else against isEvacuationToADifferentRegionActive for EACH AND SINGLE write to an instance - be that primitive or reference? Also does that mean that EACH read must be accessed via the forwarding pointer?"
The answer used to be YES; but things have changed: via this issue (though I make it sound a lot worse than it really is). Now they use Load barriers for the entire Object, more details here. Instead of having a barrier on each write (that if/else against the flag) and a dereference via the forwarding pointer for each read, they moved to a load barrier. Basically do that if/else only when you load the object. Since writing to it implies reading first, they thus preserve "to-space invariant". Apparently this is simpler, better and easier to optimize. Hooray!
Remember that forwarding pointer? Well, it does not exist anymore. I don't understand the details in its entire glory (yet), but it has to do something with the possibility to use the mark word and the from space that, since the addition of load barriers, is not used anymore. A lot more details here. Once I understand how that really works internally, will update the post.
G1 is not VERY much different than what Shenandoah is, but the devil is in the details. For example Compact phase in G1 is a STW event, always. G1 is always generational - even if you want that or not (Shenandoah can be sort of like that - there is a setting to control this), etc.
I am looking to send data through a uart communication link that is very noisy (BER can reach up to 1E-13). I was thinking of sending data that is 64 bytes long.
However, these packets are of varying lengths and should be self decodable using reed solomon with X FEC bytes that are set according to another function in the program.
The packet should be divided according to the following scheme:
byte | 1 | 2..64 | 64..X |
meaning | Seq number | DATA | RS FEC bytes |
What I am now thinking about is how I can be able to delimit the packet to be recognizable by the receiver. I have thought about two major choices:
COBS
Using COBS sounds like a good option, however, since it's a noisy channel, I am afraid that an error that affects the delimiting characters will break the whole packet.
Add a header
Adding a header telling how big is the packet feels somewhat bad, since it would be only one byte long, there is no option for error correcting with reed solomon, and writing another error correcting algorithm is overkill.
What are other options I have at my disposal for this problem?
Formatting a CoAP packet
[RFC 7252 CoAP][1]
In RFC 7252 Section 3, Figure 7 the third row, bytes 9 ... 16 or maybe more, is the Options field. I am unable to find anything that specifies how long the options field is. I understand that it can change, but unlike the Token field who's length is specified by field TKL, I cannot recognize where the length of the Options is specified.
Yes, I see sections 3.1 and 3.2 but am not able to understand what they are telling me. The document states to reference the previous options. OK, what do you do for the first message where there is no previous packet and no previous option?
When my code needs to send a CoAP message, how do I determine what options can be sent? What values must be loaded into the packet to send, for example, no options?
If you see Figure 8 in sec 3.1 on the RFC, bits 4-7 denote the length of the option value.
0 1 2 3 4 5 6 7
+---------------+---------------+
| Option Delta | Option Length | 1 byte
+---------------+---------------+
Bits 0-3 will tell you which option it is. This nibble only gives you the delta compared to the previous option encoded in this message. For the first option in the message, there is no previous option so the bits 0-3 give you the Option number.
Lets consider an example where you need to encode 2 options Uri-Port with value 7000 and Uri-Path with value /temp in a CoAP message. Options are always encoded in increasing order of the Option numbers. So you first encode Uri-Port which has Option number 7 and then Uri-Path with Option number 11.
Uri-Port
As this is the first option in the message, the Option delta will be same as the Option number so Option delta = 0x7. Port value 7000 will take 2 bytes (0x1B58) so Option length = 0x2. So this Option will be encoded get encoded as 72 1b 58.
Uri-Path
This is not the first Option in this message. Option delta for this option will be this option number - prev option number i.e. 11 - 7 = 4. Encoding temp will take 4 bytes so Option length = 4. So this option would get encoded as 44 74 65 6d 70
Note that this was for a simplified case where the Option number and length are not more than 12 bytes. When either of these is more than 12 bytes, you encode using the extended option delta/length as specified in the RFC.
I appreciate that this is a moving target, however.
I have attempted to manage my VSO backlog using both the Drag and Drop functionality and the Stack Rank Approach. On 2015 09 25 i noted along with a coworker that either manner resulted in sparcification which did not leave the backlog in the same order in which i had placed it.
http://blogs.msdn.com/b/visualstudioalm/archive/2014/05/14/behind-the-scenes-the-backlog-priority-or-stack-rank-field.aspx?wa=wsignin1.0
The issue from MSDN does not indicate that this is designed behavior however when my backlog is consistently 300 items having a random order applied to my backlog causes havoc on a daily basis. In fact i was fairly certain it was sabotage.
Has anyone else noted this.
is there a way to over come this issue.
Links or other guidance will be truly appreciated
I have used both Excel and the "edit work item" feature to change the stack rank.
what i have observed is this
Lets say i have only 5 items in my backlog ( i actually have 300)+
I set the stack rank of the first 3 items to be equal so thay clump together
work item |stackrank| Order
| 1 | 1
| 1 | 2
| 1 | 3
| 2 | 4
| 9 | 5
For 5 minutes that is the order i see on the webpage ( if i was not clear my VSO is hosted on Azure)
After a certain time ( it happens almost immediately) the sparcification does this ( NOTE order as well as stackrank) *keeping number sane on purpose althougt i know that MS uses much larger ordination
work item |stackrank| Order
| 1 | 1
| 3 | 2
| 5 | 3
| 7 | 4
| 9 | 5
So why does it not respect my clumping strategy?
I have been watching the backlog with a skeptical eye for the last week and it is appears to be reordering items without human intervention.I cannot be completely certain , however i am the backlog owner and 99% of the time ppl fess up when they add items without prior conversation. ( it happens in my organization and wont be stopped shrug)
Firstly, the Stack Rank field is removed from work item form. Check this link for the details.
Secondly, just as Daniel mentioned above, you can use Excel to order the backlog as the field is still part of the work item, it is only removed from the form.