FreeBSD pkg suddenly stopped bootstrapping - freebsd

Question

I've set up a packer template to generate vagrant base image of FreeBSD 10.3 and it was working well at least Mon Oct 3 00:34:41 2016 +0300.
Yesterday I was going to continue my work on this project and it turned out this is not working anymore. So here come details.
Packer does what it have to do, then runs my script to install FreeBSD by using bsdinstall(8) with the following script:
PARTITIONS="ada0 { 29G freebsd-ufs /, 5G freebsd-swap, 10G freebsd-ufs /var }"
DISTRIBUTIONS="base.txz kernel.txz"
#!/bin/sh
echo 'WITHOUT_X11="YES"' >> /etc/make.conf
echo 'OPTIONS_UNSET=X11' >> /etc/make.conf
echo 'nameserver 8.8.8.8' >> /etc/resolv.conf
cat >> /etc/rc.conf <<EOF
ifconfig_em0="DHCP"
sshd_enable="YES"
dumpdev="NO"
EOF
env ASSUME_ALWAYS_YES=1 pkg bootstrap # <<stops here
pkg update
pkg install -y sudo
[.....snip.....]
reboot
This stops at bootstrapping pkg with the message:
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly, please wait...
Signature for pkg not available.
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly/Latest/pkg.txz.sig: Connection reset by peer
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
If I stop the bsdinstall script and chroot /mnt /bin/sh I can fetch pkg.txz.sig from the above URL without any problems.
Any ideas what could be the reason of the "connection reset by peer"? Something was changed on the pkg.FreeBSD.org recently?
I couldn't find anything about the issue.
UPD1
Looking at the captured traffic -- the site really answers 200OK and then drops the connection for the pkg.txz.sig file.
But this 200OK packet contains the signature file and they are identical for both manual fetch (which succeeds) and pkg bootstrap (which fails)
Both sessions are identical, so this is likely not a networking problem.
UPD2
The truss was not helpful either.
So as a workaround I've just modified my bsdinstall script to fetch files manually:
[.....snip.....]
#env ASSUME_ALWAYS_YES=1 pkg bootstrap
fetch http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly/Latest/pkg.txz
fetch http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly/Latest/pkg.txz.sig
pkg add pkg.txz
pkg update
[.....snip.....]
PS: The only thing that I can suspect now is the virtualbox version update... anyway downgrading is not an option. (ISO checksum is hardcoded into the template, the template and scripts are in git repository, so accidential changes are impossible)
UPD3
I've set up a debugging environment, for the moment I only isolated the function where the error is raised.
It's the second buffer refill from the http connection (while the first one has already read 727 bytes - it should be EOF)...
Here is small gdb log with backtrace and breakpoints to get there.
Added tcpdump capture made on the system (wireshark compatible).

Answer 1

As I found out, partially the problem was with pkg -- they try to read 10240 bytes from the connection, expecting the EOF if file will be smaller, but somehow on my system EOF is not set when whole remote file was already read out.
# /release/10.3.0/usr.sbin/pkg/pkg.c
185 char buf[10240];
242 while ((r = fread(buf, 1, sizeof(buf), remote)) > 0) {
and the following loops twice -- first time reading the file, second time getting connection reset error instead of EOF
# /release/10.3.0/lib/libc/stdio/fread.c
94 resid = count * size; # == 10240 here
100 while (resid > (r = fp->_r)) {
101 (void)memcpy((void *)p, (void *)fp->_p, (size_t)r);
102 fp->_p += r;
103 /* fp->_r = 0 ... done in __srefill */
104 p += r;
105 resid -= r;
106 if (__srefill(fp)) {
107 /* no more input: return partial result */
108 return ((total - resid) / size);
109 }
110 }
While manual fetch succeeds because the size is adjusted for small chunks and they only ask 727 bytes to read:
# /release/10.3.0/usr.bin/fetch/fetch.c
720 if (us.size != -1 && us.size - count < B_size &&
721 us.size - count >= 0)
722 size = us.size - count;
723 else
724 size = B_size;
733 if ((readcnt = fread(buf, 1, size, f)) < size) {
...but why EOF is not set is still a question.
Posted this to freebsd-pkg mailing list.
UPD1
Downgraded Virtualbox from 5.028 to 5.026 and EOF is set, _sread() on libc/stdio/refill.c:135 returns 0 and it sets EOF on line 138.
So something was changed in Virtualbox networking too. Added pcap file for Virtualbox 5.026 to the gist. 5.028 really was the culprit of connection reset - here is captures comparison.
Virtualbox 5.1.8 has this bug too. Version 5.1.6 works ok.
Opened ticket #16141 in their bugtracker.