I am looking for the details of how Hyperledger Fabric get transaction history for a specific asset, including what data structures and algorothms are used. In theory, the transactions of a specific asset should be stored in different blocks not in a sequence. That is, a linear search on all blocks in blockchain may be required to get all transactions of the specific asset. Is that right? Is there any document and code about this issue recommended to read? Thank you.
History is not fetched from the ledger.
1.Unlike state data which can be stored in leveldb or couchdb History data is stored in a separate datastore and it is leveldb.
2.History data store should be enabled for each peer.
3.That can be done by enabling it in the core.yaml file in
ledger->enableHistoryDatabase set to true
or else pass the env variable
CORE_LEDGER_HISTORY_ENABLEHISTORYDATABASE=true
to the container for which you want to enable history database.
How History for the same key is stored in history database.
1.Now as it is a nosql database it cannot store a duplicate key.So some unique data is appended to the key and then the data is stored in the leveldb.
2.Querying the history database the gethistoryforkey API uses a query something like this
leveldb::Iterator* it = db->NewIterator(leveldb::ReadOptions());
for (it->Seek("theKey");
it->Valid() && it->key().ToString() < "theKey~";
it->Next()) {
...
}
As you can see the key is split from the unique data and then we get all the data related to it from the db.
Hope this helps.
Related
I am trying to run test_network on hyperledger fabric node release 2.4. See here: https://hyperledger-fabric.readthedocs.io/en/latest/test_network.html#interacting-with-the-network.
I can query the network using ReadAsset and UpdateAsset function.
However, the DeleteAsset function using below query doesn't work.
peer chaincode query -C mychannel -n basic -c '{"Args":["DeleteAsset","idOfasset"]}'
where, idOfasset is the ID or key of asset I am trying to delete from ledger.
Second, if the command would execute, would I be deleting the asset from the world database or the state database?
Or let's put it the other way when I execute ReadAsset where does it read from i.e from the world state or the channel state database?
Help would be great, thanks.
Deleting an asset changes the state of the asset. For any kind of operations which involve modification of the state of the variables, the invoke command is used. Each "Invoke" transaction will be added to the "block" in the ledger. query is used when you just want to query the chaincode for reading the current value of an asset. So use, invoke in lieu of query in your command.
Coming on to your second question
Second, if the command would execute, would I be deleting the asset from the world database or the state database? Or let's put it the other way when I execute ReadAsset where does it read from i.e from the world state or the channel state database?
World and state databases are the same things which are your CouchDB/LevelDB.
From Hyperledger Fabric's documentation:
there’s a world state – a database that holds current values of a set of ledger states. The world state makes it easy for a program to directly access the current value of a state rather than having to calculate it by traversing the entire transaction log
You can read more about it here.
Hyperledger Fabric doesn't actually "remove" the asset when a delete operation is performed. Blockchain is immutable and no data on a blockchain will ever be deleted. A deletion is just another transaction saying certain data is deleted so that the world state database can remove that data.
You can refer to this for more info.
Just a general question, if I'm building a blockchain for a business I want to store 3 years of transactions but anything older than that I won't need and don't want actively in the working database. Is there a way to backup and purge a blockchain or delete items older than some time frame? I'm more interested in the event logic than the forever memory aspect.
I'm not aware of any blockchain technology capable of this yet, but Hyperledger Fabric in particular is planning to support data archiving (checkpointing). Simply put, participants need to agree on a block height, so that older blocks can be discarded. This new block then becomes the source of trust, similar to original genesis block. Also, snapshot needs to be taken and consented, which captures current state.
From serviceability point of view, it's slightly more complicated, i.e. you may have nodes that are down while snapshotting, etc.
If you just want to purge the data after a while, Fabric Private Data has an option which could satisfy your desire.
blockToLive Represents how long the data should live on the private database in
terms of blocks. The data will live for this specified number of
blocks on the private database and after that it will get purged,
making this data obsolete from the network so that it cannot be
queried from chaincode, and cannot be made available to requesting
peers
You can read more here.
Personally, I don't think there is a way to remove a block from the chain. It might destroy the Immutable property of blockchain.
There are 2 concepts which help you achieve your goals.
The one thing is already mentioned. It is about Private Data. Private data gives you the possibility to 'label' data with a time to live. Then only the private data hashes are stored on the chain (to be able to verify this transaction) but the data itself is stored in so called SideDBs and gets fully pruned (except the hashes on the chain of course). This is kind of the basis for using Fabric without workarounds and achieving GDPR.
The other thing, which was not mentioned yet and kind of is very helpful to this question
Is there a way to backup and purge a blockchain or delete items older than some time frame?
Every peer only stores the 'current state' of the ledger in his StateDB. The current state could be described as the data which is labeled 'active' and probably soon to be used again. You can think of the StateDB being like a Cache. Every Data is comes into this Cache by creating or updating a new key (invoking). To remove a key from the Cache you can use 'DelState'. So it is labeled 'deleted' and not in the Cache anymore. BUT it is still on the ledger! and you can retrieve the history and data to that key.
Conclusion: For 'real' deleting of data you have to use the concept of Private Data and for managing data in your StateDB (think of the 'Cache' analogy) you can simply use built in functions.
How does the state database(couchDB) store data if the putState(key,value) overrides the existing value? Since we are able to fetch the history using GetHistoryForKey()....this means the old value is still existing in the state DB?
Only the latest value of a key is stored in the CouchDB state database.
The full history of keys and values are stored in the blockchain data structure itself. When GetHistoryForKey() is called, an index is consulted that identifies all the transactions that have updated a key, and then these transactions are queried from the blockchain data structure to return the history of keys and values.
i'm looking at the high-throughput chaincode example and have a question regarding the composite keys.
In the code the key is created as follows
compositeIndexName := "varName~op~value~txID"
Is it possible to query by 'op' or 'value' omitting or using some wildcard for the 'varName'? Or would I need to create different index composite keys, like in the marbles_chaincode example, for each id I would want to query? The other option is using the couchDB for state database which supports more complex querying?
As I'm gonna be saving some JSON data onto the ledger, which I'll need to query by different keys (in the marbles example let's say Color or Size).
Best regards and happy holidays!
I would recommend considering using CouchDB as a state database since it provides you with quite comprehensive querying capabilities, which a way more expressive than you can achieve with composite keys on a top-level db. This could be especially useful in case you are storing your documents in JSON format. Anyway, check this CouchDB query syntax for more information.
// GetQueryResult performs a "rich" query against a state database. It is
// only supported for state databases that support rich query,
// e.g.CouchDB. The query string is in the native syntax
// of the underlying state database. An iterator is returned
// which can be used to iterate (next) over the query result set.
// The query is NOT re-executed during validation phase, phantom reads are
// not detected. That is, other committed transactions may have added,
// updated, or removed keys that impact the result set, and this would not
// be detected at validation/commit time. Applications susceptible to this
// should therefore not use GetQueryResult as part of transactions that update
// ledger, and should limit use to read-only chaincode operations.
GetQueryResult(query string) (StateQueryIteratorInterface, error)
erefore you can use following API to retrieve your
I have a scenario where I am holding personal data. For privacy reasons I need to ensure that I am not holding onto the personal data for too long. However, when a document is deleted a tombstone record is still kept on disk. After I delete a document, can I be sure that the personal data is completely destroyed?
Question: What information is stored in the 'tombstone' record?
The tombstone record only contains the following fields:
_deleted (boolean flag)
_id
_rev
Source: CouchDB document API
However, if you have stored sensitive data in the _id field, you may need to consider:
Purging the record (Single node CouchDB only)
Contacting Cloudant Support (Cloudant only)
Finally, because we are talking about deleting records, be careful of workloads that generate a high ratio of deleted:active documents as this is considered an anti-pattern.