My Question is:
Q1. What do we mean by 'label 1, 2' and how 4 Peers are contributing to it?
Q2. What do we mean by label 3, when we compare it with 'send rate' ?
Q3. What is difference between label 3 and lable 5 and why there is much gap in memory utilization of both?
Q1: Lable 1 is the number of transactions that were successfully processed and written on the ledger. Lable 2 is the number of transactions that are being submitted every second. Lable 2 has nothing to do with the number of peers but the number of peers (and their processing power) contributes to this as if a peer fails to do its job (endorsement, verification, etc.) the transaction would fail therefore this number would be different.
Q2: Lable 3 represents the number of transaction that has been processed vs send rate which is the per second rate of the transactions submitted to the blockchain. e.g., in your Test1 the 49 transactions per second were submitted but only 47 transactions per second were processed hence the 2.4 seconds Max Latency (It is more complex than what I said.)
Q3: Lable 5 represents a peer which is in charge of running and verifying the smart contracts and probably endorsement as well (depending on your endorsement policy) but the label 3 is a world state database peer (more here: https://vitalflux.com/hyperledger-fabric-difference-world-state-db-transaction-logs/ ) and running smart contracts uses more resources.
Related
I'm working on node api, and facing concurrent request problem.
pseudo code
1. Get user details along with credit balance.
2. Get contest maxEntry count (1 user = 1 count), total no. user joined contest so far, maxEntryPerUser, and contest entry fee.
3. Compare user credit balance and entry if balance is low then throw error
4. Compare maxEntry and totalJoined, if maxEntry is less or equal to totalJoined then throw error and not allow user to join the contest otherwise allow.
Problem 1: Suppose user have 2 credit balance and contest entryFee is 2, maxEntryPerUser is 1
user have credit balance 1, and user sent 100 request concurrently, all 100 request started processing parallelly, out of 100 more than 50 request get process successfully i.e. same user joined the contest more than 50 time with low credit balance
Expected result: 99 request must get failed and only 1 request should get success response.
Problem 2: Suppose maxEntry count 1
Multiple users sent 100 request concurrently, all 100 request started processing parallelly, out of 100 more than 50 request get process successfully i.e. more than 50 users joined the contest whereas there was only 1 entry allowed
Expected result: 99 request must get failed and only 1 request should get success response.
This can easily be solved by running the check and the data modification in the same REPEATABLE READ transaction. Then all but one transactions will get a serialization error and have to restart.
If you expect conflicts that often, it might be better to use SELECT ... FOR NO KEY UPDATE in the check to ensure that only one check can run on the same data at the same time.
I know the below is true for sequence number within a partition :-
Sequence Number follows a predictable pattern where numbering is contiguous and unique within the scope of a partition. So if message x has sequence number 500 , message y will have sequence number 501.
Say last sequence number when message was received was 5000, and then no more messages were received and after 7 days retention policy , on 10th day we receive a message on the partition , will the sequence number start from 5001 or will it be different this time?
The reason i am asking because I am seeing:
"The expected sequence number xxx is less than the received sequence number yyy."?
For example:-
The supplied sequence number '33508' is invalid. The last sequence number in the system is '583'
Just to update this post with answer, we found the concerned EventHubs are indeed recreated, but checkpoint store was not reset. Thus, SequenceNumber presented by client (read from checkpoint store) was greater than latest sequence number at service.
As Serkant confirmed above, Sequence number is always contiguous. It can only be interrupted by recreating the eventhub. Typically, you shouldn’t have a need to delete and recreate EventHub in production. But in case you run into the situation, you should also reset checkpoint store.
HTH,
Amit Bhatia
I'm trying to run a project using Hyperledger Fabric with a setup similar to the Fabcar example.
I'm surprised by the huge amount of time it takes to submit a transaction.
Just to make it simple and fully reproducible I measured the time needed submit the transaction createCar on the actual Fabcar project.
After setting up the network (startFabric.sh javascript) and enrolling the admin and a user, I run the invoke.js script. The whole script takes about 2.5 seconds!
As far as I can understand running the contract takes just few milliseconds. The same for sending the Transaction Proposal.
It mostly spend time having the eventHandler listening and waiting for the event (in the transaction.js library).
Is there a way of speeding up the process? I was expecting to be able to submit several transactions per second.
Short answer : decrease BatchTimeout in configtx.yaml
Long answer :
If you only run one query, like you describe, this is totaly normal.
If you look at your configtx.yaml, in the part 'Orderer', you can see this :
Orderer: &OrdererDefaults
# Orderer Type: The orderer implementation to start
# Available types are "solo" and "kafka"
OrdererType: solo
Addresses:
- orderer.example.com:7050
# Batch Timeout: The amount of time to wait before creating a batch
BatchTimeout: 2s
# Batch Size: Controls the number of messages batched into a block
BatchSize:
# Max Message Count: The maximum number of messages to permit in a batch
MaxMessageCount: 10
# Absolute Max Bytes: The absolute maximum number of bytes allowed for
# the serialized messages in a batch.
AbsoluteMaxBytes: 99 MB
# Preferred Max Bytes: The preferred maximum number of bytes allowed for
# the serialized messages in a batch. A message larger than the preferred
# max bytes will result in a batch larger than preferred max bytes.
PreferredMaxBytes: 512 KB
There are 2 important things :
BatchTimeout
MaxMessageCount
BatchTimeout define the maximum time before creating a block. This mean, when an invoke is made, the orderer compute the transaction and wait for 2 seconds before creating the block. This mean, each first transaction will take more than 2 seconds ! But if there is another invoke, lets say, at 1,5s after the first transaction, the second call will take less than 1s !
MaxMessageCount, speak for itself. It mean that if there are more than 10 invoke, a block will be created, even if the 2 seconds are not past. For example, 10 invoke in a row of 0.5s will result in a block creation in less than a second.
This settings are here to balance the load depending on your network. Let's say you have a low use application, with less than 10 tps (transaction per second), you can reduce the BatchTimeout to less than 2s to increased response time of invoke. If you have a hight tps, you can increased the MaxMessageCount to create larger block.
The others settings define the max size of a message.
Try to know how your network will be, simulate the estimated tps with a test case and tweak the parameters to find the configuration of your needs.
As described in this example each tserver started with 12 tablet as we set number of shards to 4.
And when we added a new node the number of tablet per tserver became 9. it seems the total number of tablet, which is 36, will not increase.
My question is:
How many node could we add while we have 36 total tablet(in this example)?
And Is it possible to increase shards count in a running universe to be able to add more node?
How many node could we add while we have 36 total tablet(in this example)?
In this example, you can expand to 12 nodes (each node would end up with 1 leader and 2 followers).
Reasoning: There are 36 total tablets for this table and the replication factor is 3. So there will 12 tablet leaders and 24 tablet followers. Leaders are responsible for handling writes and reads (unless you're doing follower reads, lets assume that is not the case). If you go to 12 nodes, each node would at least have one leader and be doing some work.
Ideally, you should create enough tablets upfront so that you end up with 4 tablets per node eventually.
And Is it possible to increase shards count in a running universe to be able to add more node?
This is currently not possible, but being worked on and getting close to the finish. expected to be released in Q1 2020. If you are interested in this feature, please subscribe to this GitHub issue for updates.
Until that is ready, as a workaround, you can split the table into sufficient number of tablets.
For Azure event hub 1 though put unit equals 1MB/sec ingress. So it can take 1000 messages of 1 KB. If I select 5 or more throughput units would I be able to ingest 5000 messages/ second of 1KB size with 4 partitions? What would be egress in that case? I am not sure about limitation on Event Hub partition, i read that it is also 1MB/sec. But then does that mean to use event hub effectively i need to have same number of partitions?
Great question.
1 Throughput Unit (TU) means an ingress limit of 1 MB/sec or 1000 msgs/sec - whichever happens first. You pay for TUs and you can change TUs as per your load requirements. This is your knob to control the bill. And TUs are set on a given Event Hubs Namespace!
When you buy 1 TU for an EventHubs Namespace and create a number of EventHubs in it, the the limit of 1 MB/sec or 1000 msgs/sec applies cumulatively across them. The limit also applies to each partition individually. Although, sometimes you might get lucky in some regions where load is low.
Consider these principles while deciding on no. of partitions in eventhub for your service:
The intent of Partitions is to offer high-availability. If you are sending to Eventhubs and you want the sends to succeed NO MATTER WHAT you should create multiple partitions and send using EventHubClient.Send (which doesn't confine the send to a particular partition).
The no. of partitions will determine how fat the event pipe is & how fast/parallelly you can receive & process the events. If you have 10 partitions on your EventHub - it's capacity is effectively capped at 10 TUs. You can create 10 epoch receivers in parallel & consume & process events. If you envision that the EventHub that you are currently creating now can quickly grow 10-fold create as many partitions and keep the TU's matching the current load. Analogy here is having multiple lanes on a freeway!
Another thing to note is, a TU is configured at namespace level. And, one Event Hubs namespace can have multiple EventHubs in it and each EventHub can have a different no. of partitions.
Answers:
If you select 5 or more TUs on the Namespace and have only 1 EventHub with 4 partitions you will get a max. of 4 MB/sec or 4K msgs/sec.
Egress max will be 2X of ingress (8 MBPS or 8K msgs/sec). In other words, you could create 2 patterns of receives (e.g. slow and fast) by creating 2 consumer groups. If you need more than 2X parallel receives then you will need to by more TUs.
Yes, ideally you will need more partitions than TUs. First model your partition count as mentioned above. Start with 1 TU while you are developing your solution. Once done, when you are doing load testing or going live, increase TUs in tune with your load. Remember, you could have multiple EventHubs in a Namespace. So, having 20 TUs at Namespace level and 10 EventHubs with 4 partitions each can deliver 20 MB/sec across the Namespace.
More on EventHubs
One partition goes to one TPU. Think of TPUs as a processing engine. You can't take advantage of more TPUs than you have partitions. If you have 4 partitions, you can't use more than 4 TPUs.
It's typical to have more partitions than TPUs, for the following reasons
You can scale the number of TPUs up if you have a lot of traffic, but you can't change the number of partitions
You can't have more concurrent readers than you have partitions. If you want to have 5 concurrent readers, you need 5 partitions.
As for throughput, the limits are 1 MB ingerss/2 MB egress per TPU. This covers the typical scenario where each event is sent both to cold storage (eg a database) and Stream analytics or an event processor for analysis, monitoring etc.