Can OPC-UA write data to a tag in blocks / chunks? Let's say I have a sensor/UA client that samples at 100Hz and needs to send those data to a UA server, I have an OPC UA tag on that server to take in that sensor's data. Can I have a "blocked" write, to write 100 samples all together in one transaction to that tag every second? Each sample has a source timestamp. So for offline application it does not matter how the data were sent.
Currently, I transmit the samples one by one, therefore 100 write transactions for 1 second of data, is not efficient.
Unless that server is configured to store history for that tag then whether you write 100 values at once (which yes, is possible) or 100 values in a row to the same tag the end result is the same - the tag's value attribute will be the last value of the 100.
Maybe you should consider configuring the tag in that server to be an array and then writing an array of values to it?
Related
I am using hazelcast jet to aggreagte(sum) stream of data
Source is kafka where i receive integer and jet stream simply adds each incoming number.
I have few questions
1. When it receives each number along with a it saves the data in IMap, how can i access that snapshot?
#Abhishek, Hazelcast-Jet takes snapshots if you configure it, and not with each number, with a time period. If you want to access map, you cannot & even if you access, the data stored in that map uses an internal data structure, you cannot just view your numbers there.
If you can share what kind of information you're trying to get, I can help you more. (Along with your job definition to understand it a bit if possible)
My data is stored among multiple partitions. I was to send this data to the client but I want to paginate the response. So say my 1st partition has 100 rows and 2nd partition has 100 rows. I want to send 10 rows per page along with PagingState. The client would send PagingState back to server and I'll use it to fetch next 10 records running the same query. Once I have exhausted 100 rows of 1st partition, I'll have to change the query. Is it possible to find which query was executed from PagingState so that I could read the PagingState, find for which partition it was for and using this information, I can determine what should be next partition
Its possible, but not straight forward or safe. The content changes between (protocol and cassandra) versions. Its also not very trivial to parse, as latest uses var ints to mark size of both partition key and row marker. On older versions it requires to send a cell level marker as well which it still sends for backwards compatibility in some scenarios so should really handle both. And with new versions of C* you will need to check to see if it changes.
You can always do paging on client side which will give you control over it and knowledge of the state that wont change on versions.
I have the following scenario:
Mobile app produces events that are sent to Event Hub which is input stream source to a Stream Analytics query. From there they are passed through a sequential flow of queries that splits the stream into 2 streams based on criteria, evaluates other conditions and decides whether or not to let the event keep flowing through the pipeline (if it doesn't it is simply discarded). You could classify what we are doing is noise reduction/event filtering. Basically if A just happened don't let A happen again unless B & C happened or X time passes. At the end of the query gauntlet the streams are merged again and the "selected" events are propagated as "chosen" outputs.
My problem is that I need the ability to compare the current event to the previous "chosen" event (not just the previous input event) so in essence I need to join my input stream to my output stream. I have tried various ways to do this and so far none have worked, I know that other CEP engines support this concept. My queries are mostly all defined as temporary results sets inside of a WITH statement (that's where my initial input stream is pulled into the first query and each following query depends on the one above it) but I see no way to either join my input to my output or to join my input to another temporary result set that is further down in the chain. It appears that join only supports inputs?
For the moment I am attempting to work around this limitation with something I really don't want to do in production, but I actually have an output defined going to an Azure Queue then an Azure Function triggered by events on that queue that wakes up and posts it to a different Event hub that is mapped as a recirc feed input back into my queries which I can join to. Still wiring all of that up so not 100% sure it will work but thinking there has to be a better option for this relatively common pattern?
The WITH statement is indeed the right way to get a previous input joined with some other data.
You may need to combine it with the LAG operator, that gets the previous event in a data stream.
Let us know if it works for you.
Thanks,
JS - Azure Stream Analytics
AFAIK, the stream analytics job supports two distinct data input types: data stream inputs and reference data inputs. Per my understanding, you could leverage Reference data to perform a lookup or to correlate with your data stream. For more details, you could refer to the following tutorials:
Data input types: Data stream and reference data
Configuring reference data
Tips on refreshing your reference data
Reference Data JOIN (Azure Stream Analytics)
I am trying to build a real-time stock application.
Every seconds I can get some data from web service like below:
[{"amount":"20","date":1386832664,"price":"183.8","tid":5354831,"type":"sell"},{"amount":"22","date":1386832664,"price":"183.61","tid":5354833,"type":"buy"}]
tid is the ticket ID for stock buying and selling;
date is the second from 1970.1.1;
price/amount is at what price and how many stock traded.
Reuirement
My requirement is show user highest/lowest price at every minute/5 minutes/hour/day in real-time; show user the sum of amount in every minute/5 minutes/hour/day in real-time.
Question
My question is how to store the data to redis, so that I can easily and quickly get highest/lowest trade from DB for different periods.
My design is something like below:
[date]:[tid]:amount
[date]:[tid]:price
[date]:[tid]:type
I am new in redis. If the design is this is that means I need to use sorted set, will there any performance issue? Or is there any other way to get highest/lowest price for different periods.
Looking forward for your suggestion and design.
My suggestion is to store min/max/total for all intervals you are interested in and update it for current ones with every arriving data point. To avoid network latency when reading previous data for comparison, you can do it entirely inside Redis server using Lua scripting.
One key per data point (or, even worse, per data point field) is going to consume too much memory. For the best results, you should group it into small lists/hashes (see http://redis.io/topics/memory-optimization). Redis only allows one level of nesting in its data structures: if you data has multiple fields and you want to store more than one item per key, you need to somehow encode it yourself. Fortunately, standard Redis Lua environment includes msgpack support which is very a efficient binary JSON-like format. JSON entries in your example encoded with msgpack "as is" will be 52-53 bytes long. I suggest grouping by time so that you have 100-1000 entries per key. Suppose one-minute interval fits this requirement. Then the keying scheme would be like this:
YYmmddHHMMSS — a hash from tid to msgpack-encoded data points for the given minute.
5m:YYmmddHHMM, 1h:YYmmddHH, 1d:YYmmdd — window data hashes which contain min, max, sum fields.
Let's look at a sample Lua script that will accept one data point and update all keys as necessary. Due to the way Redis scripting works we need to explicitly pass the names of all keys that will be accessed by the script, i.e. the live data and all three window keys. Redis Lua has also JSON parsing library available, so for the sake of simplicity let's assume we just pass it JSON dictionary. That means that we have to parse data twice: on the application side and on the Redis side, but the performance effects of it are not clear.
local function update_window(winkey, price, amount)
local windata = redis.call('HGETALL', winkey)
if price > tonumber(windata.max or 0) then
redis.call('HSET', winkey, 'max', price)
end
if price < tonumber(windata.min or 1e12) then
redis.call('HSET', winkey, 'min', price)
end
redis.call('HSET', winkey, 'sum', (windata.sum or 0) + amount)
end
local currkey, fiveminkey, hourkey, daykey = unpack(KEYS)
local data = cjson.decode(ARGV[1])
local packed = cmsgpack.pack(data)
local tid = data.tid
redis.call('HSET', currkey, tid, packed)
local price = tonumber(data.price)
local amount = tonumber(data.amount)
update_window(fiveminkey, price, amount)
update_window(hourkey, price, amount)
update_window(daykey, price, amount)
This setup can do thousands of updates per second, not very hungry on memory, and window data can be retrieved instantly.
UPDATE: On the memory part, 50-60 bytes per point is still a lot if you want to store more a few millions. With this kind of data I think you can get as low as 2-3 bytes per point using custom binary format, delta encoding, and subsequent compression of chunks using something like snappy. It depends on your requirements, whether it's worth doing this.
I'm designing a HTTP-service, with capacity of up to 500 million requests per day (served by more than one independent machine).
For each request I have to generate unique ID and return it to user. ID must be 100% unique within a window of 10 minutes. (1 day is preferred, globally unique IDs are ideal.) No server-server communication must be needed to generate that ID.
Silly pseudo-session example:
Client: GET /foo
Server: Content-Type: text/xml
<root>
<id>ab9d1972-2844-11e0-86b2-000c29544403</id>
<other_data/>
</root>
In previous generation of this HTTP service I used UUIDs.
I'm happy with UUIDs, but there is one problem: they are too long. On that number of requests, this extra size in noticeable in disk space waste for log files.
What is the best way to create a short, but unique identifier? To make things worthwhile, I guess, algorithm should produce at most half of UUID length while being unique for all day long (10 minutes should be even shorter).
Ideally, suggested algorithm would have sane, lightweight production-quality implementation in plain C.
Update: Generated ID should not require URI-encoding when passed in the GET request.
Give each machine a unique prefix. Give each machine a counter. To generate an ID, increment the counter, and append its value to the prefix.
If you want to obfuscate the IDs, encrypt them - a cipher is a reversible transformation, so applying it to unique values will produce unique values.
A few thoughts:
500 million requests a day. Really?
Use UUIDs.
If required, don't use HTTP (as that's the more significant overhead) and transfer the UUID in a binary form.
You need a certain amount of bytes to guarantee that your server returns a truly unique ID.
How about using UDP?
Anyway, what the heck are you trying to do?