I have 2 gluster clusters of type distributed-replicated:
Cluster 1 (C1): with bricks in machines M1 and M2.
Cluster 2 (C2): with bricks in machines M3 and M4.
I want to expand C1 by adding M4 (already part of C2) and another machine M5.
For adding the bricks, first I need to add M4 and M5 into C1 by probing M4 and M5 from either M1 or M2. When I do peer probe, I am able to add machine M5, but when I try to add M4 to C1 by
sudo gluster peer probe M4
I get:
peer probe: failed: M4 is either already part of another cluster or having volumes configured
I have two questions:
Is it even possible to achieve what I want (since I am mixing two different clusters)?
If yes, how to do it?
PS: I have read the following links but my issue still is not resolved:
redhat mailing list, redhat mailing list-2
At any point of time, a node can be in only one cluster.
You should first do a gluster peer detach M4 of M4 from C2 and then you can add to C1.
Check this link:
http://docs.gluster.org/en/latest/Administrator%20Guide/Storage%20Pools/#removing-servers-from-the-trusted-storage-pool
# gluster peer detach <server>
Please run 'gluster peer detach ' on every hosts. Then try 'gluster peer probe
On the rejected peer:
Stop glusterd
In /var/lib/glusterd, delete everything except glusterd.info (the UUID file)
rm -rf !(glusterd.info)
Start glusterd
Probe one of the good peers
Restart glusterd,
check 'gluster peer status'
You may need to restart glusterd another time or two, keep checking peer status.
Related
Background
We have a production fabric cluster setup and has been been running for a year. Now most of the certs expire and the cluster crash, including both tls and identity certs.
I tried to fix by completely removing old certs and private keys, generate and enroll new identities for peer, peer admin, orderer, orderer admin.
Everything works again, but I cannot instantiate/upgrade chaincode in existing channel because the channel was configured with old admin certs.
Problem
So now look like I'm stuck in a deadlock. In order to update channel config with new cert, I need to sign the update with matching old cert, which is already expired and blocked by orderer.
I find out that we can disable expired cert check in orderer using ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true. But now I don't have the old admin private key so I still cannot update the channel config.
Questions
I already replaced old private keys with new one so there is no way to use the old cert again.
Can I do something to resolve this channel issue?
Suggestions are greatly appreciated.
[!] What I'm suggesting is an idea. I haven't tested it.
[!] It seems to be feasible enough, but side-effect is not considered.
[!] It's just a trick, it's correct that it should never be done.
The conclusion is that the orderer and peer's binary can be artificially manipulated and updated.
For fabric, refer to $GOROOT/src/crypto when building binary.
Build in the fabric repository after artificially modifying all ecdsa verify functions in crypto to return true immediately.
cd $GOROOT/src/crypto
vi ecdsa/ecdsa.go # modify `Verify` function
cd $GOPATH/src/github.com/hyperledger/fabric
make peer
make orderer
Back up the binaries of the currently running docker container, and rerun after planting the newly built binaries in the container.
docker cp <peer_container_name>:/usr/local/bin/peer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/peer <peer_container_name>:/usr/local/bin/peer
docker cp <orderer_container_name>:/usr/local/bin/orderer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/orderer <orderer_container_name>:/usr/local/bin/orderer
docker-compose -f <your_docker_compose_file_path> restart
Now all verify is valid unconditionally. so, update all recent status.
Afterwards, the backed up binary is replanted into the container to solve this problem.
docker cp ./peer <peer_container_name>:/usr/local/bin/peer
docker cp ./orderer <orderer_container_name>:/usr/local/bin/orderer
docker-compose -f <your_docker_compose_file_path> restart
My condition right now is
i have 4 server, all of them were centos 8-minimal based
i have created a volume named gv0 and replicated it to 2 other server (total 3 nodes, GFS-1 | GFS-2 | GFS-3) and it works normal, i can store/read files from another client node
i want to create a geo replication for gv0 from the GFS-1 node to another node named GFS-4 and it's on different LAN network
i saw this tutorial and followed it till executing this command on the GFS-1 node
gluster system:: execute gsec_create
it gives me an error said: gsync peer_gsec_create command not found.
what i can do with this? i haven't found any solution to this on Google, please help me
Please try yum install glusterfs-geo-replication
Thanks
How can I join a channel with a peer, using the latest config block ?
The orderers in config block 0 do no longer exist, the dns names have changed.
When I fetch the latest config for the channel, and try to join with that I get the following error:
peer channel fetch config ...
peer channel join ...
Error: proposal failed (err: bad proposal response 500: cannot create ledger from genesis block: Expected block number=0, received block number=11276)
command terminated with exit code 1
However when I fetch config block 0 and join, it does so succesfully, but the peer never 'syncs' up as it can't connect to the orderers (as they no longer exist under that domain)
peer channel fetch 0 ...
peer channel join ...
...
in logs
Could not connect to any of the endpoints: [{orderer-3.orderers.svc.cluster.local:7050 [...]} {orderer-1.orderers.svc.cluster.local:7050 [...]} {orderer-2.orderers.svc.cluster.local:7050 [...]}]
Try with block 0 or oldest. As DNS names have changed, you have to make some trick.
TRICK 1: Override name resolution in /etc/hosts.
In your peer (inside the docker container itself), edit /etc/hosts.
First, get the new domain IP:
# apt update
# apt install dnsutils -y
host new.svc.cluster.local
Take note of the IP, let's say X.Y.W.Z.
Now, edit /etc/hosts inside the peer container and associate the new IP to the old domain:
X.Y.W.Z old.svc.cluster.local
Do it for every domain that has changed. Now you should be able to join. Even if the peer joint before, now it is able to synchronize. Whenever your peer container is relocated, /etc/hosts changes are lost, but it doesn't mind once it has been synchronized
An alternative trick would be to use iptables, but it is only useful if your old domain still resolves to an IP.
I have unexpectedly executed sudo rm -rf */ on my amazon EC2 linux instance.
As soon as i saw the files deleting so i have terminated the command using ctrl+c.
My website instance is working fine. But i am unable to connect to the instance through ssh. Its giving connection has closed. On another pc there is an already existing connection, so when i do ls or sudo commands Its giving usr/bin/ls not found command. cd is working fine to switch the directories.
How can i be able to connect through the ssh and restore the deleted directories? OR How can i solve this issue?
Im sorry to tell you, but i don't think there is much you can do now. Setting it up completely new would be the easiest and i think most possible way to get it up working fine again. And there's this one rule, never use "sudo rm -rf" without checking your Location twice and a third time. You can delete your whole system if you do this command in your Root. So good luck.
Nico
Let us say you deleted the / in the instance I1 which has a volume attached V1. Now create another instance with same configuration and say it I2 with volume V2.
So now stop I1 detach V1 from it and attach it to I2.
Mount the V1 on I2.
Since both instances are from same Ami. They have same uuid so to mount by mount -o nouuid /V1 /V2/mountpoint.
Copy files in the directory / in V2 to / in V1.
Detach it from I2 and re-attach it I1.
I hope it works.
CoreOS was with the beta channel:
coreos-install -d /dev/sda -C beta -c ./cloud-config.yaml
SSH into the CoreOS host, and check the version:
$ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=991.2.0
VERSION_ID=991.2.0
BUILD_ID=2016-03-26-0329
PRETTY_NAME="CoreOS 991.2.0 (Coeur Rouge)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
Question 1: How frequent CoreOS checks for update? Would it continue within the same Release Channel?
Question 2: Is the Release Channel information (which was used to install CoreOS) written somewhere on the CoreOS host?
It checks for an update roughly every hour. You can verify this with journalctl -u update-engine to watch the logs.
Correct, the channel is stored in /etc/coreos/update.conf
Reading Switching Release Channels It seems that the information is written in /etc/coreos/update.conf
In my case. The content of that file is:
$ cat /etc/coreos/update.conf
GROUP=beta
REBOOT_STRATEGY=reboot
I suppose this answers Q2. But how about the frequency CoreOS checks for update?