Background
We have a production fabric cluster setup and has been been running for a year. Now most of the certs expire and the cluster crash, including both tls and identity certs.
I tried to fix by completely removing old certs and private keys, generate and enroll new identities for peer, peer admin, orderer, orderer admin.
Everything works again, but I cannot instantiate/upgrade chaincode in existing channel because the channel was configured with old admin certs.
Problem
So now look like I'm stuck in a deadlock. In order to update channel config with new cert, I need to sign the update with matching old cert, which is already expired and blocked by orderer.
I find out that we can disable expired cert check in orderer using ORDERER_GENERAL_AUTHENTICATION_NOEXPIRATIONCHECKS=true. But now I don't have the old admin private key so I still cannot update the channel config.
Questions
I already replaced old private keys with new one so there is no way to use the old cert again.
Can I do something to resolve this channel issue?
Suggestions are greatly appreciated.
[!] What I'm suggesting is an idea. I haven't tested it.
[!] It seems to be feasible enough, but side-effect is not considered.
[!] It's just a trick, it's correct that it should never be done.
The conclusion is that the orderer and peer's binary can be artificially manipulated and updated.
For fabric, refer to $GOROOT/src/crypto when building binary.
Build in the fabric repository after artificially modifying all ecdsa verify functions in crypto to return true immediately.
cd $GOROOT/src/crypto
vi ecdsa/ecdsa.go # modify `Verify` function
cd $GOPATH/src/github.com/hyperledger/fabric
make peer
make orderer
Back up the binaries of the currently running docker container, and rerun after planting the newly built binaries in the container.
docker cp <peer_container_name>:/usr/local/bin/peer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/peer <peer_container_name>:/usr/local/bin/peer
docker cp <orderer_container_name>:/usr/local/bin/orderer ./
docker cp $GOPATH/src/github.com/hyperledger/fabric/build/bin/orderer <orderer_container_name>:/usr/local/bin/orderer
docker-compose -f <your_docker_compose_file_path> restart
Now all verify is valid unconditionally. so, update all recent status.
Afterwards, the backed up binary is replanted into the container to solve this problem.
docker cp ./peer <peer_container_name>:/usr/local/bin/peer
docker cp ./orderer <orderer_container_name>:/usr/local/bin/orderer
docker-compose -f <your_docker_compose_file_path> restart
Related
It would be grateful if someone helps me out with this. I do wanted to access the certificates and keys from the common storage . if i give the path it is creating a folder near msp and creating the certificates.
I tired by changing the path in docker-compose.yml file. The path changed but it is creating near the msp folder.
I wanted to know , where the default path must be changed.
fabric-ca-client allows you to specify the directory in which keys/certificates will be stored created by using the -M option:
fabric-ca-client enroll -u http://enrollid:enrollsecret#myca:7054 -M /path/to/myfolder
If you are using Docker to run the fabric-ca-client and want to make the key/certs available outside the Docker container, you will need to mount an external volume.
So let's say you want to store the key/certs in the /var/mycerts directory on your host system. You can do:
docker run --rm -v /var/mycerts:/var/mycerts hyperledger/fabric-ca fabric-ca-client enroll -u http://enrollid:enrollsecret#myca:7054 -M /var/mycerts
I have a sawtooth 1.1 dockerized network, and i'm trying to backup the database from the validators so i can put down every container and then recover in case of a disaster.
Trying to achieve this i proceed as followed:
Stopped the all containers;
Backed up all the files of one of the validators on /var/lib/sawtooh/ using
cp --sparse=always [file] [file_backup]
Removed all the containers using docker-compose down
Started a fresh network with docker-compose up
Stopped all containers using docker-compose stop
Copied the files backed up on step 2 to the new validators using the command of that same step
Restarted all network using docker-compose restart
After this i could repare that the states were correct, users on the blockchain have the same balance as before of the docker-compose down. But the blockchain doesn't process new transactions.
The only error that i've found in the logs, was in the sawtooth-poet-engine
i believe during the consensus as it show on the this image,
ERROR_IMAGE.
So my question is, does anybody tried to do this with success or have any idea of what i'm doing wrong?
I just tried the same thing and it worked for me.
One possible problem is file permissions and ownership.
Use the cp -p option to preserve ownership and permissions:
cp -p --sparse=always [file] [file_backup]
Also verify the ownership and permissions are correct with ls -l /var/lib/sawtooth .
They should be all read/write by owner and owned by user/group sawtooth. If not, fix the ownership with something like
chown sawtooth:sawtooth /var/lib/sawtooth /var/lib/sawtooth/*
chmod u+rw /var/lib/sawtooth /var/lib/sawtooth/*
chmod ugo+r /var/lib/sawtooth/block-* /var/lib/sawtooth/txn_receipts-00.lmdb*
I'm newbie in the HyperLedger Fiber Technology and I'm following this official tutorial to build my first network. I replaced example.com with the domain of my company. After generating certificates I ran the follow command to up my network:
./byfn.sh up -c ttchannel
I get the following error during channel creation:
Error: got unexpected status: FORBIDDEN -- Failed to reach implicit threshold of 1 sub-policies, required 1 remaining: permission denied
!!!!!!!!!!!!!!! Channel creation failed !!!!!!!!!!!!!!!!
Then I verified from PEERs containers that peers are already connected with this channel (don't know why and when this connection got established), So, I restarted this network and then got the following error:
Error: failed to create deliver client: failed to load config for OrdererClient: unable to load orderer.tls.rootcert.file: open /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/vodworks.com/orderers/orderer.vodworks.com/msp/tlscacerts/tlsca.vodworks.com-cert.pem: no such file or directory
!!!!!!!!!!!!!!! Channel creation failed !!!!!!!!!!!!!!!!
And then I verified in the CLI container that crypto directory wasn't accessible so I have to restart that container to make it accessible. but still I was unable to run it.
Can anyone please suggest that what these error means and how can I run my network?
Where exactly did you change "example.com"? I'm pretty sure the setup script also creates all the crypto material based on configtx.yaml file, that has "example.com" as domain everywhere. The docker-compose files then map the created folders as volume into the container (see docker-compose-base.yml):
../crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp:/var/hyperledger/orderer/msp
So the docker-compose file also expects the certs to be in this folders. From your description I would expect that you missed to change the domain at some point.
The Reason of this error was that network was not going down properly. All the peers were connected with the channel already as I verified by running the command peer channel list inside the container of each peer which was resulting in the following output:
Channels peers has joined:
ttchannel
And with this situation, the command peer channel create ... was actually throwing this error. So, I'm able to run the network successfully with the following command:
**./byfn.sh restart -c ttchannel -s couchdb -t 60 -d 10**
Which actually first put the network down and then start it again.
I am trying to get hyperledger explorer to work with docker-compose, hyperledger blockchain explorer did not connect to the hyperledger blockchain network. It says "Failed to connect client peer, please check the configuration and peer status". However I followed all instructions I could find to get it to connect.
The Hyperledger explorer starts up and I can access it at http://localhost:8090/ but it does not connect to the blockchain.
Here is the github project where it is not working.
https://github.com/contractpendev/hyperledger-basic-network
Blockchain Explorer comes from here.
https://github.com/hyperledger/blockchain-explorer
Finally I think my question is not a good question as no-one else is likely to have this problem anymore.
I would add a comment asking for more info, but since I don´t have enough rep yet to comment, I guess I'll try an answer instead!
All of the following assume that Blockchain Explorer is run in a docker container, using the supplied deploy_explorer.sh script. All examples below should work with the first-network sample network from fabric-samples, assuming that fabric-samples has been cloned to ~/fabric/fabric-samples.
Firstly, are there any more detailed errors reported in either the docker logs (docker logs {container-name}) or the log file in the container (/opt/logs/app/app.log)?
Have you done the following?
Confirm that you are using the correct Blockchain Explorer version for your fabric version - e.g. explorer 3.7 for fabric 1.2.
Clean-up any existing Blockchain Explorer installation - if you had a previous version installed, you should remove it first using the supplied deployment script, i.e.
cd ~/fabric/blockchain-explorer
./deploy_explorer.sh --clean
Clone and/or check-out the correct release of the blockchain explorer repository - e.g.
cd ~/fabric
git clone https://github.com/hyperledger/blockchain-explorer.git
cd blockchain-explorer
git checkout release-3.7
Create a folder under "blockchain-explorer/examples" for your network - e.g.
cd ~/fabric/blockchain-explorer/examples
mkdir first-network
Create a config.json file for your network (in the folder created above). You can take a copy of the sample file from ~/fabric/blockchain-explorer/examples/net1/config.json and modify it accordingly.
Create a symbolic link to crypto-config (in the same folder), e.g.
cd ~/fabric/blockchain-explorer/examples/first-network
ln -fs ~/fabric/fabric-samples/crypto-config crypto
Deploy Blockchain Explorer - specifying the name of the folder containing your config, and the name of your docker network, e.g.
cd ~/fabric/blockchain-explorer
./deploy_explorer.sh first-network net_byfn
You should now be able to connect to explorer at http://localhost:8080/. If the page still doesn't load, check the docker logs for any errors (docker logs blockchain-explorer).
See https://github.com/hyperledger/blockchain-explorer/blob/master/README.md for more details (replacing "master" with the label for release you are using, e.g. "release-3.7").
Also, please note that the port exposed when using deploy_explorer.sh has changed between release-3.7 and release-3.8 from 8080 to 8090. This one caught me out for quite some time!
I used Mac OS.
When I was running Fabric java SDK，I met this Error：New channel foo error. StatusValue 400. Status BAD_REQUEST。How to fix it？
Checking the docker logs for the orderer will help. to do this, get the containerid of the orderer by running
$ docker ps
Then, check the logs for the by running
$ docker logs <containerid>
Some potential reasons are:
that the channel name has illegal characters (e.g. channelAllowedChars = "[a-z][a-z0-9.-]*", from https://github.com/hyperledger/fabric/blob/0631ccd2b1e30f56088dd3905a78bfb73a93ed51/common/configtx/manager.go).
The client that's being used to create the channel does not have the proper user context and thus fails certificate validation (needs to be a valid peer organization user with an MSPID to allow the certifcate and key to be verified).