Why does Perforce populate command slow? - perforce

Question

Situation:
Network bandwidth to the Perforce server is limited (I'm working remotely).
The repository I wanted to branch contains lots of binary files and it's many gigabytes in size.
The count of files is large (about 500000 files).
When I issue a p4 populate command to populate a new dev branch, the operation takes 1-2 hours to finish, why?
As far as I know populate does the branching in place and doesn't pull the entire repo through the network, so I don't think it's a network issue.
Creating a million entries in a database shouldn't take long as well.
Does it copy the binary files into the new repository, instead of just referencing them?
EDIT here is the log after I branched again using -Ztrace:
57055 files branched (change 2779417).
--- lapse 6408s
--- usage 3573+2619us 0+0io 0+0net 65672k 0pf
--- rpc msgs/size in+out 2+57059/0mb+13mb himarks 785732/280100 snd/rcv 3.38s/.047s
--- db.counters
--- pages in+out+cached 11+16+2
--- locks read/write 0/4 rows get+pos+scan put+del 3+0+0 4+0
--- db.logger
--- pages in+out+cached 6+4+4
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 2+0
--- db.user
--- pages in+out+cached 4+0+3
--- locks read/write 1/0 rows get+pos+scan put+del 1+0+0 0+0
--- db.group
--- pages in+out+cached 5+0+4
--- locks read/write 1/0 rows get+pos+scan put+del 0+3+5 0+0
--- db.domain
--- pages in+out+cached 7+0+5
--- locks read/write 2/0 rows get+pos+scan put+del 2+0+0 0+0
--- db.view
--- pages in+out+cached 7+0+7
--- locks read/write 4/0 rows get+pos+scan put+del 0+4+57 0+0
--- db.integed
--- pages in+out+cached 42486+51286+192
--- pages reordered internal+leaf 180+5972
--- pages split internal+leaf 182+4377
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 114110+0
--- total lock wait+held read/write 0ms+0ms/0ms+5175ms
--- db.archmap
--- pages in+out+cached 28+47+26
--- pages split internal+leaf 1+12
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 502+0
--- total lock wait+held read/write 0ms+0ms/0ms+1174ms
--- db.revdx
--- pages in+out+cached 3+0+1
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 0+0
--- total lock wait+held read/write 0ms+0ms/0ms+1174ms
--- db.revhx
--- pages in+out+cached 2888+8526+96
--- pages split internal+leaf 85+2783
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 57055+0
--- total lock wait+held read/write 0ms+0ms/0ms+1174ms
--- db.revpx
--- pages in+out+cached 233571+357872+96
--- pages split internal+leaf 79+2395
--- locks read/write 0/114110 rows get+pos+scan put+del 0+0+0 57055+57055
--- total lock wait+held read/write 0ms+0ms/5846ms+2128ms
--- max lock wait+held read/write 0ms+0ms/99ms+9ms
--- db.revcx
--- pages in+out+cached 1414+4160+96
--- pages split internal+leaf 43+1356
--- locks read/write 0/1 rows get+pos+scan put+del 0+0+0 57055+0
--- total lock wait+held read/write 0ms+0ms/0ms+1174ms
--- db.rev
--- pages in+out+cached 10977+7868+96
--- pages split internal+leaf 76+2570
--- locks read/write 2/1 rows get+pos+scan put+del 57055+7+184634 57055+0
--- total lock wait+held read/write 0ms+571ms/0ms+1174ms
--- max lock wait+held read/write 0ms+571ms/0ms+1174ms
--- peek count 1 wait+held total/max 0ms+0ms/0ms+0ms
--- db.trigger
--- pages in+out+cached 4+0+2
--- locks read/write 2/0 rows get+pos+scan put+del 0+2+35 0+0
--- db.change
--- pages in+out+cached 13+12+4
--- locks read/write 0/3 rows get+pos+scan put+del 3+0+0 2+1
--- total lock wait+held read/write 0ms+0ms/0ms+1175ms
--- max lock wait+held read/write 0ms+0ms/0ms+1175ms
--- db.changex
--- pages in+out+cached 11+8+3
--- locks read/write 0/3 rows get+pos+scan put+del 0+0+0 1+1
--- total lock wait+held read/write 0ms+0ms/0ms+1175ms
--- max lock wait+held read/write 0ms+0ms/0ms+1175ms
--- db.changeidx
--- pages in+out+cached 5+0+1
--- locks read/write 0/2 rows get+pos+scan put+del 0+0+0 0+0
--- total lock wait+held read/write 0ms+0ms/1ms+1174ms
--- max lock wait+held read/write 0ms+0ms/1ms+1174ms
--- db.desc
--- pages in+out+cached 13+12+4
--- locks read/write 0/3 rows get+pos+scan put+del 0+0+0 2+1
--- total lock wait+held read/write 0ms+0ms/0ms+1174ms
--- max lock wait+held read/write 0ms+0ms/0ms+1174ms
--- db.protect
--- pages in+out+cached 9+0+8
--- locks read/write 1/0 rows get+pos+scan put+del 0+1+410 0+0
--- db.monitor
--- pages in+out+cached 2+6+256
--- locks read/write 4/2 rows get+pos+scan put+del 4+0+0 1+1

Answer 1

If you run a command with the global flag -Ztrack, e.g.:
p4 -Ztrack populate (args...)
you'll get a bunch of performance debugging information that includes where the server was spending its time. (This information is probably already stored in the server log, if you don't want to have to run the command again; you just need to track it down in there.)
In the general case, p4 populate is a metadata-only command (copying rows in the revision tables and adding integed rows to connect them), and all of the actual file contents are simply "lazy copied" by virtue of having additional references to the same underlying archive.
The exception to this is if there are revisions that can't be lazy-copied. Off the top of my head the two cases of that I can think of are remote depots and +S files (maybe some of those large binaries you mentioned).
In any event, if you can get the performance tracking information, you can see whether the server was spending all of its time on creating database entries (in which case I'd look at the disk that the database is on and see if there's any way to speed it up -- is it network-mounted, is it shared by a bunch of other processes, etc) or if it was copying archives around (in which case you'll want to figure out why it needs to do that and if there's any way to avoid it -- for example, I always recommend against using the +S filetype flag for files that are going to be branched).