Not sure how good this use case finally is, but that's what I'm working on right now. I need from time to time to store files on the hyperledger blockchain with CouchdDB as state database. Sometimes json, sometimes csv but sometimes binary. We also updated the couchdb, from the default fabric version to the latest available version CouchDB 3.3.0. At this stage we convert binaries to base64 ans store in the JSON body of the payload.
I see that the default supported document size for CouchDB 3.3.0 is 8MB, which we increased for our purposes. But this will be the hard limit in the future CouchDB updates. Attachments in CouchDB support up to 4GB....
Is it possible to make Hyperledger chaincode to write some data as an attachment?
Related
Since Hyperledger Fabric has a workaround to avoid double spending by checking the read set versioning with the world state versioning.
Assuming there is a client that issues transaction to update the same asset then the coming transactions by my knowledge will be invalidated by the peers.
To my understanding this means in one block there should only be one transaction for one asset update and note more.
Any help, could you please confirm my assumptions.
Highly appreciated.
Thanks.
Hyperledger Fabric uses Multi-Version Concurrency Control (MVCC) is used to address double-spending problems.
Fabric’s underlying databases use MVCC to make sure no double-spending or inconsistency in data will occur. In understandable terms, this means it’s using versioned documents or records (Since it’s a key-value store, we call each row a document or record). When updating the same state, a new version of an existing document will be made to overwrite the old one. Any transaction executed between this time will not be cleared and an MVCC error will be triggered. The advantage of using versioning overlocking like other databases is that even under high load, the DB can run at full speed asynchronously. This is why in modern databases MVCC is more widely used.
I am currently working on a project in Hyperledger, where I want to upload files like pdf and docs into hyperledger blockchain by distributing the document file across the node of the network, and retrieving the document file back. Please help me with how I can do and how should I approves. If it cannot be achieved in Hyperledger then please let me know by which blockchain it can be achieved. Thanks in advance.
As a general rule, blockchain technologies are not suitable to store large documents. Blockchain demands too much process and storage replication. Moreover, there are some more pitfalls, such as block size and confidentiality.
One common approach is storing the files in a distributed P2P storage system such as IPFS (https://ipfs.io/) and store the file's hash (hashes are used as file references in IPFS, so that integrity is assured) in the blockchain's state.
You may need also IPFS-Cluster (https://cluster.ipfs.io/) to ensure persistence and replication.
Encryption is not supported by IPFS, so, if needed, it should be applied end to end outside IPFS, encrypting before storing and decrypting after accessing.
From your comment it sounds like you are thinking of using Hyperledger Fabric. With Fabric you should be able to write chaincode and a client that stores files. It still might not be the best approach and you should think carefully about separating storage of large files from the blockchain consensus based chaincode, especially when storing a hash of the file on-chain would suffice. Each node will need to store the file and come to consensus that all nodes stored the same file (have the same state). This is expensive in terms of compute and network i/o.
With Fabric, you should be able to do just about anything in code, as long as it is deterministic and not a long-running process. In my experience, minimizing code and state on any blockchain is best practice.
A complete example of a Hyperledger Fabric implementation is here: https://fabric-chaintool.readthedocs.io/en/latest/getting-started/
I agree with Kekomal, don't put large files in a blockchain, as it bloats it. Rather store binary/large files offchain. One option in Hyperledger Fabric(HLF) are private data collections where you can store binary files in CouchDB as base64 text, and just store the checksum/hash onchain, but that would increase storage requirements - based on base64.guru a 20KB pdf in base64 text will be 26KB - probably significant with very large binary files, as well as improper use of a document db. Private data collections are nice as HLF let's you share files with various options
Please advise if it is possible to somehow remove old blocks from hyperledger fabric ?
I understand that it must be immutable, but what if we do not want to store years old data?
There is no "Archive" feature in Fabric at the moment, but there is an "Epic" in the jira system for an Archive feature. It has been around for quite a while but it now looks like a high priority.
Here are the details.
No, you can not remove transaction from the chain itself. Since otherwise you would destroy the whole thing (merkle tree properties).
But only 'recent' data will be stored in the StateDBs. This is data which is labeled as not deleted. Deleted data will be moved out of this DB. So you could actually think of this as a cache where you can 'quickly' access up to date data. (maybe read docs about StateDB and Ledger etc in Fabric to get more insights on this)
I am just a beginner in HyperLedger Composer and Fabrics.
Following IBM tutorial under 'https://www.coursera.org/learn/ibm-blockchain-essentials-for-developers'
I have one quick question:
How to create a File Server using Hyperledger Composer?
Is it possible now or not?
any feedback will be very helpful.
Using Composer and Fabric as a File Server would be unusual, and due to the nature of Fabric as a Distributed Ledger, it may well perform badly as a File Server.
There are other Questions and Answers in Stack Overflow about storing images etc using base64 encoding, such as this one.
Depending on your use case, it may be more appropriate to store some hash of the file on the Fabric enabling you to prove the validity of the file, whilst storing the file itself on a dedicated File Server.
With the nature of technology, saving a file in hyperledger fabric and using it as a file server would not be a good strategy.
You Hash the original file and Use hyperledger fabric to store the Hash value and other data you need for the system and to Validate. But you should use a different server to Store the Files.
Something similar to this would work I guess: A Sample System Design : Image
Any changes, updates or a version change, you store the new file in the server, create the new hash and add the data in to new entry in hyperledger fabric.
We are building a solution and we are modeling a network using Fabric and Composer
Regarding "not" storing any personal data (GDPR complience) on the blockchain, we would like to hash/map the personal data so that a GUID och Hash is stored in the Ledger instead (Anonymized data)
Does Hyperledger provide any solution to solve this kind of issues (ie a distributed DB that is around the ledger peers for example?)
Or is this something that is needed to be implemented outside the Hypeledger network topology?
Prior to Fabric v1.1, you would need to provide the database yourself and then just write the hashes to the blockchain as normal transactions. There are people who do this today for database records as well as for documents (store the document outside and just write the hash and metadata to the blockchain).
In Fabric v1.1, there is an experimental featured known as "private data". With this feature, the actual state is kept local to the peers in a private state database and is not included in the actual blockchain itself. The ledger actually contains hashes of the key and value.
There are new chaincode APIs (Get/PutPrivateData) which are used to do this automatically for you. You can then either delete the data manually or use the DeletePrivateState function in chaincode to delete the actual records (the hash will stay on the channel ledger).
This feature is experimental in v1.1 so you will need to build the peer from source with -tags experimental.
Since this feature is experimental, it is not currently supported in Composer.
We will be hardening the feature as part of the 1.2 release which is under development