What is the difference between the TransactionRegistry and the Historian? - hyperledger-fabric

Say I just wanted to get a list of all transactions that involved a specific asset (assuming I would need a query for this?). Should I use the TransactionRegistry or the Historian? What's the difference?

we have a current issue open for Historian to show history of changes/ deltas for a particular asset - https://github.com/hyperledger/composer/issues/991 As a workaround you can do the following - so for sample network trade-network with an asset Commodity (and a transaction class 'Trade') you could create a query eg:
query selectTransaction {description: "choose specific commodity asset"
statement: SELECT org.acme.biznet.Trade
WHERE (commodity == _$commodity ) }
On the difference:
Historian records all transaction activities (eg. create Asset, create participant, create identity etc etc - and also business network specific custom transactions like 'TransferAsset' or 'PlaceOrder') including (where assets / participants are concerned) what changed.
For the TransactionRegistry itself (ie a particular class - say 'TransferAsset' or 'PlaceOrder') this is stored in the Transaction registry for that class - you may have many Transaction classes in your business network. But nothing in here would be related to other activities, such as system activities, that also recorded in Historian records.
to query - you would do something like this (in a query file for example):
query myTransactions{
description: "return all transactions made (ie system transactions)"
statement: SELECT org.acme.sample.PlaceOrder
}
ie SELECT org.acme.sample.NAME_OF_TRANSACTION_CLASS
For Historian queries - eg
SELECT org.hyperledger.composer.system.HistorianRecord WHERE (transactionType == 'myTranType'
see more examples here -> https://www.ibm.com/developerworks/cloud/library/cl-create-powerful-blockchain-queries-with-hyperledger-composer/index.html

furthermore, to see the transaction data (deltas) for the asset id you're zoning in on - ie available through the transactionInvoked field of the transaction class (eg org.acme.trading.Trade transaction class). you could use REST APIs with loopback filters -eg either (both return a promise below):
return this.httpClient.get('http://localhost:3000/api/Trade?filter=%7B%22include%22%3A%22resolve%22%7D', {withCredentials: true}).toPromise();`
or
return this.httpClient.get('http://localhost:3000/api/Trade?filter=%7B%22include%22%3A%22resolve%22%7D').toPromise();
which has the {"include":"resolve"} filter to resolve relationships in the transaction class - each resolved transaction has the transaction deltas. Then you could look for the asset id in question.
Sample unresolved transaction class (below, followed by resolved txn class):
Not resolved:
[
{
"$class": "org.acme.mynetwork.Trade",
"commodity": "resource:org.acme.mynetwork.Commodity#1",
"newOwner": "resource:org.acme.mynetwork.Trader#2",
"transactionId": "354dca97fc6ac00aabbd923883e3ec2a3d09b8c75a54a8f536a88b6df31e8a0f",
"timestamp": "2018-03-23T12:02:11.228Z"
},
{
"$class": "org.acme.mynetwork.Trade",
"commodity": "resource:org.acme.mynetwork.Commodity#2",
"newOwner": "resource:org.acme.mynetwork.Trader#1",
"transactionId": "9da43acca718633ac8870e6ea34c3c9f481194e48bcdba42673570177091809f",
"timestamp": "2018-03-23T12:02:31.294Z"
}
]
Resolved with {"include":"resolve"} as a filter:
[
{
"$class": "org.acme.mynetwork.Trade",
"commodity": {
"$class": "org.acme.mynetwork.Commodity",
"tradingSymbol": "1",
"description": "werwer",
"mainExchange": "wrrewer",
"quantity": 10,
"owner": {
"$class": "org.acme.mynetwork.Trader",
"tradeId": "2",
"firstName": "tes2t",
"lastName": "test"
}
},
"newOwner": {
"$class": "org.acme.mynetwork.Trader",
"tradeId": "2",
"firstName": "tes2t",
"lastName": "test"
},
"transactionId": "354dca97fc6ac00aabbd923883e3ec2a3d09b8c75a54a8f536a88b6df31e8a0f",
"timestamp": "2018-03-23T12:02:11.228Z"
},
{
"$class": "org.acme.mynetwork.Trade",
"commodity": {
"$class": "org.acme.mynetwork.Commodity",
"tradingSymbol": "2",
"description": "Ut fugiat.",
"mainExchange": "ACE2",
"quantity": 10,
"owner": {
"$class": "org.acme.mynetwork.Trader",
"tradeId": "1",
"firstName": "test",
"lastName": "test"
}
},
"newOwner": {
"$class": "org.acme.mynetwork.Trader",
"tradeId": "1",
"firstName": "test",
"lastName": "test"
},
"transactionId": "9da43acca718633ac8870e6ea34c3c9f481194e48bcdba42673570177091809f",
"timestamp": "2018-03-23T12:02:31.294Z"
}
]

Related

Query only some attributes of Hyperledger Fabric

i've build Hyperledger Fabric channel and it was running well. But, i wanna get just some attributes only from that Hyperledger that have data structure like this
[
{
"name": "name-1",
"class": "IV-A",
"address": "dummy st.",
"handphone": "xxx-xxx-xxx-xxx",
"status": "inactive"
},
{
"name": "name-2",
"class": "IV-A",
"address": "dummy st. 7th",
"handphone": "xxx-xxx-xxx-xxx",
"status": "active"
},...
]
i wanna get just some attributes (name, address, and status), so we can get the return like this
[
{
"name": "name-1",
"address": "dummy st.",
"status": "inactive"
},
{
"name": "name-2",
"address": "dummy st. 7th",
"status": "active"
},...
]
is it possible to get those attributes via hyperledger directly without using couchdb? i know if we use couchdb we just need to create the view with some query in the "emit()", but is there any way to get that in hyperledger using chaincode or selector?
Thank you in advance.
I'm not sure about your Hyperledger Fabric environment and data.
But you can check out this chaincode.
OR if you are OK to use 'invoke function', you can return value what you want.
return shim.Success([]byte($value))

Spark streaming Dynamic Schema Evolution from Kafka Eventhub on Microbatch

We are streaming data from the Kafka Eventhub. The records may have a nested structure. The schema is inferred dynamically from the data and the Delta table is formed with the schema of the first incoming batch of data.
Note: The data read from Kafka topic will be a whole JSON string. Hence,
When we apply schema and convert to a dataframe, we lose the fields' values with mismatch datatype or newly added fields.
When we do spark.read.json, the entire field values are converted to String.
We encounter a situation where the Source data has some schema changes. Some of the scenarios we faced are :
The datatype changes at the parent level
The datatype changes at the nested level
There are duplicate keys in a different case
There are the addition of new fields
A sample Source data with the Actual schema
{
"Id": "101",
"Name": "John",
"Department": {
"Id": "Dept101",
"Name": "Technology",
"EmpId": "10001"
},
"Experience": 2,
"Organization": [
{
"Id": "Org101",
"Name": "Google"
},
{
"Id": "Org102",
"Name": "Microsoft"
}
]
}
A sample Source data addressing the 4 points mentioned above
{
"Id": "102",
"name": "Rambo", --- Point 3
"Department": {
"Id": "Dept101",
"Name": "Technology",
"EmpId": 10001 ---- Point 2
},
"Experience": "2", --- Point 1
"Organization": [
{
"Id": "Org101",
"Name": "Google",
"Experience": "2", --- Point 4
},
{
"Id": "Org102",
"Name": "Microsoft",
"Experience": "2",
}
]
}
We need a solution to overcome the above issues. Though it's difficult to embed the new schema to the existing delta table, at least we should be able to separate the records with schema changes without losing the original data.

Issue on Docusign Admin Api getUserProfile

Hi I am facing an issue while loading the endpoint point /v2/organizations/{organizationId}/users/profile
I am getting a 404 error
Attached below screenshots for the postman response and getting organisationId
The correct baseUrl for demo Admin API is https://api-d.docusign.net/management/v2/
Try to change that and see if that works.
In order for that endpoint to work you're going to need to provide some additional parameters in the URL. For retrieving a user's profile directly, you should be able to use:
https://api-d.docusign.net/management/v2/organizations/{OrgID}/users/profile?email=useremail#example.com
The result looks like this:
{
"users": [
{
"id": "28fbe7e4-xxxx-xxxx-xxxx-c26569aa827c",
"site_id": 1,
"site_name": "Demo",
"user_name": "Matt_SATest_1",
"first_name": "MattSAT_1",
"last_name": "K",
"user_status": "active",
"default_account_id": "70da9c0c-xxxx-xxxx-xxxx-7b3d6733328e",
"default_account_name": "Matt K",
"is_organization_admin": false,
"created_on": "2020-08-25T20:38:57.533",
"memberships": "",
"identities": "",
"is_device_verification_enabled": true
}
]
}
To retrieve a list of users for an org, you should be able to use:
https://api-d.docusign.net/management/v2/organizations/{OrgID}/users?URLParameters.
The URL parameters for this particular call require one of the following: account_id, organization_reserved_domain_id, or email. You're required to use at least one, but you should also be able to use more than one parameter at a time.
The result looks something like this:
{
"users": [
{
"id": "28fbe7e4-xxxx-xxxx-xxxx-c26569aa827c",
"user_name": "Matt_SATest_1",
"first_name": "Matt",
"last_name": "K",
"user_status": "active",
"email": "email#example.com",
"created_on": "2020-08-25T20:38:57.533"
}
],
"paging": {
"result_set_size": 1,
"result_set_start_position": 0,
"result_set_end_position": 0,
"total_set_size": 1
}
}

Merge documents by fields

I have two types of docs. Main docs and additional info for it.
{
"id": "371"
"name": "Mike",
"location": "Paris"
},
{
"id": "371-1",
"age": 20,
"lastname": "Piterson"
}
I need to merge them by id, to get result doc. The result should look like:
{
"id": "371"
"name": "Mike",
"location": "Paris"
"age": 20,
"lastname": "Piterson"
}
Using COLLECT / INTO, SPLIT(), and MERGE():
FOR doc IN collection
COLLECT id = SPLIT(doc.id, '-')[0] INTO groups
RETURN MERGE(MERGE(groups[*].doc), {id})
Result:
[
{
"id": "371",
"location": "Paris",
"name": "Mike",
"lastname": "Piterson",
"age": 20
}
]
This will:
Split each id attribute at any - and return the first part
Group the results into sepearate arrays (groups)
Merge #1: Merge all objects into one
Merge #2: Merge the id into the result
See REMOVE & INSERT or REPLACE for write operations.

How to search through data with arbitrary amount of fields?

I have the web-form builder for science events. The event moderator creates registration form with arbitrary amount of boolean, integer, enum and text fields.
Created form is used for:
register a new member to event;
search through registered members.
What is the best search tool for second task (to search memebers of event)? Is ElasticSearch well for this task?
I wrote a post about how to index arbitrary data into Elasticsearch and then to search it by specific fields and values. All this, without blowing up your index mapping.
The post is here: http://smnh.me/indexing-and-searching-arbitrary-json-data-using-elasticsearch/
In short, you will need to do the following steps to get what you want:
Create a special index described in the post.
Flatten the data you want to index using the flattenData function:
https://gist.github.com/smnh/30f96028511e1440b7b02ea559858af4.
Create a document with the original and flattened data and index it into Elasticsearch:
{
"data": { ... },
"flatData": [ ... ]
}
Optional: use Elasticsearch aggregations to find which fields and types have been indexed.
Execute queries on the flatData object to find what you need.
Example
Basing on your original question, let's assume that the first event moderator created a form with following fields to register members for the science event:
name string
age long
sex long - 0 for male, 1 for female
In addition to this data, the related event probably has some sort of id, let's call it eventId. So the final document could look like this:
{
"eventId": "2T73ZT1R463DJNWE36IA8FEN",
"name": "Bob",
"age": 22,
"sex": 0
}
Now, before we index this document, we will flatten it using the flattenData function:
flattenData(document);
This will produce the following array:
[
{
"key": "eventId",
"type": "string",
"key_type": "eventId.string",
"value_string": "2T73ZT1R463DJNWE36IA8FEN"
},
{
"key": "name",
"type": "string",
"key_type": "name.string",
"value_string": "Bob"
},
{
"key": "age",
"type": "long",
"key_type": "age.long",
"value_long": 22
},
{
"key": "sex",
"type": "long",
"key_type": "sex.long",
"value_long": 0
}
]
Then we will wrap this data in a document as I've showed before and index it.
Then, the second event moderator, creates another form having a new field, field with same name and type, and also a field with same name but with different type:
name string
city string
sex string - "male" or "female"
This event moderator decided that instead of having 0 and 1 for male and female, his form will allow choosing between two strings - "male" and "female".
Let's try to flatten the data submitted by this form:
flattenData({
"eventId": "F1BU9GGK5IX3ZWOLGCE3I5ML",
"name": "Alice",
"city": "New York",
"sex": "female"
});
This will produce the following data:
[
{
"key": "eventId",
"type": "string",
"key_type": "eventId.string",
"value_string": "F1BU9GGK5IX3ZWOLGCE3I5ML"
},
{
"key": "name",
"type": "string",
"key_type": "name.string",
"value_string": "Alice"
},
{
"key": "city",
"type": "string",
"key_type": "city.string",
"value_string": "New York"
},
{
"key": "sex",
"type": "string",
"key_type": "sex.string",
"value_string": "female"
}
]
Then, after wrapping the flattened data in a document and indexing it into Elasticsearch we can execute complicated queries.
For example, to find members named "Bob" registered for the event with ID 2T73ZT1R463DJNWE36IA8FEN we can execute the following query:
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "flatData",
"query": {
"bool": {
"must": [
{"term": {"flatData.key": "eventId"}},
{"match": {"flatData.value_string.keyword": "2T73ZT1R463DJNWE36IA8FEN"}}
]
}
}
}
},
{
"nested": {
"path": "flatData",
"query": {
"bool": {
"must": [
{"term": {"flatData.key": "name"}},
{"match": {"flatData.value_string": "bob"}}
]
}
}
}
}
]
}
}
}
ElasticSearch automatically detects the field content in order to index it correctly, even if the mapping hasn't been defined previously. So, yes : ElasticSearch suits well these cases.
However, you may want to fine tune this behavior, or maybe the default mapping applied by ElasticSearch doesn't correspond to what you need : in this case, take a look at the default mapping or, for even further control, the dynamic templates feature.
If you let your end users decide the keys you store things in, you'll have an ever-growing mapping and cluster state, which is problematic.
This case and a suggested solution is covered in this article on common problems with Elasticsearch.
Essentially, you want to have everything that can possibly be user-defined as a value. Using nested documents, you can have a key-field and differently mapped value fields to achieve pretty much the same.

Resources