Cannot add members to MongoDB Replica Set - linux

Question

I'm trying to configure a MongoDB Replica Set but every time I try to add another member it fails.
I have 3 members I'm trying to configure. Their mongod.conf files all look like this:
# mongo.conf
#where to log
logpath=/log/mongod.log
logappend=true
# fork and run in background
fork = true
smallfiles=true
rest=true
port = 27017
replSet=KidzpaceReplSet
dbpath=/data
With the acception of the ports. They are 27017(Primary), 27018(Secondary) and 27019(Arbiter) respectively.
I have verified that the members can see each other:
[ec2-user#domU-12-31-39-06-C4-74 ~]$ mongo --host 174.129.232.170 --port 27018
MongoDB shell version: 2.4.3
connecting to: 174.129.232.170:27018/test
>
[ec2-user#domU-12-31-39-0A-30-E8 ~]$ mongo --host 174.129.230.20 --port 27017
MongoDB shell version: 2.4.3
connecting to: 174.129.230.20:27017/test
>
When adding the second member to the set it returns OK:
KidzpaceReplSet:PRIMARY> rs.add("174.129.232.170:27018")
{ "ok" : 1 }
However whatever the next command I run is, In this case it's adding my Arbiter, the set fails with this error:
KidzpaceReplSet:PRIMARY> rs.add("174.129.232.177:27019", true)
Tue May 28 20:24:07.139 DBClientCursor::init call() failed
Tue May 28 20:24:07.140 trying reconnect to 127.0.0.1:27017
Tue May 28 20:24:07.141 reconnect 127.0.0.1:27017 ok
reconnected to server after rs command (which is normal)
This is the the log file:
Tue May 28 20:44:06.173 [rsStart] replSet I am domU-12-31-39-06-C4-74:27017
Tue May 28 20:44:06.173 [rsStart] replSet STARTUP2
Tue May 28 20:44:07.175 [rsSync] replSet SECONDARY
Tue May 28 20:44:07.175 [rsMgr] replSet info electSelf 0
Tue May 28 20:44:08.174 [rsMgr] replSet PRIMARY
Tue May 28 20:44:29.813 [conn1] replSet replSetReconfig config object parses ok, 2 members specified
Tue May 28 20:44:29.817 [conn1] replSet replSetReconfig [2]
Tue May 28 20:44:29.817 [conn1] replSet info saving a newer config version to local.system.replset
Tue May 28 20:44:29.834 [conn1] replSet saveConfigLocally done
Tue May 28 20:44:29.834 [conn1] replSet info : additive change to configuration
Tue May 28 20:44:29.834 [conn1] replSet replSetReconfig new config saved locally
Tue May 28 20:44:39.835 [rsHealthPoll] DBClientCursor::init call() failed
Tue May 28 20:44:39.835 [rsHealthPoll] replset info 174.129.232.170:27018 heartbeat failed, retrying
Tue May 28 20:44:40.834 [rsHealthPoll] DBClientCursor::init call() failed
Tue May 28 20:44:40.834 [rsHealthPoll] replSet info 174.129.232.170:27018 is down (or slow to respond):
Tue May 28 20:44:40.835 [rsHealthPoll] replSet member 174.129.232.170:27018 is now in state DOWN
Tue May 28 20:44:40.835 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote
Tue May 28 20:44:40.835 [rsMgr] can't see a majority of the set, relinquishing primary
Tue May 28 20:44:40.835 [rsMgr] replSet relinquishing primary state
Tue May 28 20:44:40.835 [rsMgr] replSet SECONDARY
Tue May 28 20:44:40.835 [rsMgr] replSet closing client sockets after relinquishing primary
Tue May 28 20:44:42.044 [conn1] end connection 127.0.0.1:58727 (0 connections now open)
Tue May 28 20:44:46.150 [rsHealthPoll] replSet member 174.129.232.170:27018 is up
Tue May 28 20:44:46.151 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
Tue May 28 20:44:52.156 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes
UPDATE
I'm wondering if maybe the problem is when I run rs.initiate(). It gives me this output:
{
"set" : "KidzpaceReplSet",
"date" : ISODate("2013-05-28T20:59:05Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "domU-12-31-39-06-C4-74:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 23,
"optime" : {
"t" : 1369774732,
"i" : 1
},
"optimeDate" : ISODate("2013-05-28T20:58:52Z"),
"self" : true
}
],
"ok" : 1
}
Notice the name of the member? "name" : "domU-12-31-39-06-C4-74:27017" Where does this name come from? It's not my IP Address. I'm not sure but maybe this could be the source of the problem.

Answer 1

So it turns out rs.initiate() might give the member that launches it some kind of internal alias for it's IP address. In my case it was: domU-12-31-39-06-C4-74.
The initial connection to the secondary is fine because the primary instigates it. However since the secondary now has this alias to use when it tries to talk back to the primary, it fails.
The solution was a to copy the existing configuration:
cfg = rs.conf()
manually change the name(host) of the primary node:
cfg.members[0].host = 666.666.666.666:27017
And reconfigure the replica set:
rs.reconfig(cfg)