I am using ditto and while creating some things I obtain this error org.eclipse.ditto.things.model.ThingTooLargeException
I searched on ditto source code and found only this property
<ditto.limits.things.max-size>102400</ditto.limits.things.max-size>
on file ditto\things\model\pom.xml
I am not very experienced, how can I change this limit?
The default limit of the Eclipse Ditto managed Thing entity is 100k.
This is configured in the ditto-limits.conf.
You can overwrite this limit by setting the environment variable LIMITS_THINGS_MAX_SIZE to something higher, e.g. LIMITS_THINGS_MAX_SIZE=200k.
Depending how you start Ditto (e.g. using Kubernetes or Docker Compose) you have to set this environment variable for all Ditto services.
I however want to mention that this limit is there for a reason and that it e.g. does not make sense to configure this much higher.
The stability, throughput, memory consumption, as well as the usage of the database, e.g. regarding search indexes, is influenced by this limit.
So I would rather rethink if a Ditto "Thing" is really the place where you want to store >100k data in.
As a side note: comparable commercial IoT services like AWS IoT Core have a "shadow size limit" (which is the equivalent to the Ditto "thing") of 8k.
Related
Tech Stack
Azure WebApp
.Net core 2.1 WebApi
We have around 4k reference data which is used during auto suggest lookup, so in this i was wondering whether i should cache this data on WebApp or should always get it from database / 3rd party API.
I know i can use RedisCache to solve this issue, but i would like to know how Azure WebApp works when it comes to caching, it will have memory pressure? When? Yes then scale-up is the only solution?
We are using IMemoryCache in .net Core to store reference data and it expires on daily basis or when Azure WebApp is restarted (So 1st user will get delay till it gets all data in cache).
Data size is in range of 500KB - 1MB & sometimes goes till 3MB+.
What is the best approach?
iMemoryCache is not suggested when using WebApps because it is tightly bound to your application instance, so if you try to scale out your app (in case of load surges during the day) your caching mechanism will be broken.
RedisCache is pretty much a dictionary, key-value pairs.
It is very fast on look-ups but it could be very slow in some other operations like a GetAllKeys when it has to run through the whole cache. That will bring your cache server to its knees, so it needs to be handled carefully.
It will not put any significant pressure in the memory consumption of your app, you only need to have a static client. The rest is handled by the redis server.
If you plan to scale up your application (give more RAM and CPU resources to your one running instance) the iMemory cache is probably fine.
If you plan to scale out (create multiple instances of your application), that is strongly suggested for all stateless applications, then RedisCache (or any other distributed cache) is an one way for you if you need a caching mechanism.
Value and key max size is 512MB so you are on the safe side regarding value data size.
Attention
Be sure to use the Connection multiplexer as it is suggested in the official documentation because it automatically re-establishes the connection in case it is lost. That was a bug earlier, when redis cache server was going into maintenance your calls where redirected to the fail over instance but the connection was failing, so you needed to restart your application.
In IMap configuration there is an attribute read-backup-data that can be set as true which enables a member to read the value from the backup copy, if available, in case the owner of the key is some other member.
http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Data_Structures/Map/Backing_Up_Maps.html#page_Enabling+Backup+Reads
Then there is nearcache which will start caching results for a few datastructures locally.
http://docs.hazelcast.org/docs/latest-development/manual/html/Performance/Near_Cache/Hazelcast_Data_Structures_with_Near_Cache_Support.html
If we have 2 kinds of cluster setup:
2 members, and async-backup-count for a map is 1, and read-backup-data is true
2 members, nearcache enabled for this map
Would there be differences in these 2 approaches?
1st setup will probably use less memory, and will not be configurable. But in terms of read performance?
For two member cluster setup; enabling backup reads will provide you to access all the data locally, since both members hold all the entries as either primary or backup. This setup is not much different than using a Replicated Map (see here for details: http://docs.hazelcast.org/docs/latest-development/manual/html/Distributed_Data_Structures/Replicated_Map.html). So; when your cluster have only two members (also no clients), enabling backup reads can be more advantageous in terms of performance.
However; near cache has a bunch of configuration options, and you can decide how much data you need to access locally at any type of setup (including client-server topology). You can also decide the in-memory data format in near cache. These options can provide you more performance than enabling backup reads.
Both options are not so much different in single entry read performance (I assume near cache contains valid entry), since both don't perform a remote operation.
I've had a few outages of 10 to 15 minutes, because apparently Microsoft had a 'blip' on their storages. They told me that it is because of a shared file system between the instances (making it a single point of failure?)
I didn't understand it and asked how file share is involved, because I would assume a really dumb stateless IIS app that communicates with SQL Azure for its data.
I would assume the situation below:
This is their reply to my question (I didn't include the drawing)
The file shares are not necessarily for your web app to communicate to
another resources but they are on our end where the app content
resides on. That is what we meant when we suggested that about storage
being unavailable on our file servers. The reason the restarts would
be triggered for your app that is on both the instances is because the
resources are shared, the underlying storage would be the same for
both the instances. That’s the reason if it goes down on one, the
other would also follow eventually. If you really want the
availability of the app to be improved, you can always use a traffic
manager. However, there is no guarantee that even with traffic manager
in place, the app doesn’t go down but it improves overall availability
of your app. Also we have recently rolled out an update to production
that should take care of restarts caused by storage blips ideally, but
for this feature to be kicked it you need to make sure that there is
ample amount of memory needs to be available in the cases where this
feature needs to kick in. We have couple of options that you can have
set up in order to avoid any unexpected restarts of the app because of
a storage blip on our end:
You can evaluate if you want to move to a bigger instance so that
we might have enough memory for the overlap recycling feature to be
kicked in.
If you don’t want to move to a bigger instance, you can always use
local cache feature as outlined by us in our earlier email.
Because of the time differences the communication takes ages. Can anyone tell me what is wrong in my thinking?
The only thing that I think of is that when you've enabled two instances, they run on the same physical server. But that makes really little sense to me.
I have two instances one core, 1.75 GB memory.
My presumption for App Service Plans was that they were automatically split into availability sets (see below for a brief description) Largely based on Web Apps sales spiel which states
App Service provides availability and automatic scale on a global data centre infrastructure. Easily scale applications up or down on demand, and get high availability within and across different geographical regions.
Following on from David Ebbo's answer and comments, the underlying architecture of Web apps appears to be that the VM's themselves are separated into availability sets. However all of the instances use the same fileserver to share the underlying disk space. This file server being a significant single point of failure.
To mitigate this Azure have created the WEBSITE_LOCAL_CACHE_OPTION which will cache the contents of the file server onto the individual Web App instances. Using caching in lieu of solid, high availability engineering principles.
The problem here is that as a customer we have no visibility into this issue, we've no idea if there is a plan to fix it, or if or when it will ever be fixed since it seems unlikely that Azure is going to issue a document that admits to how badly this has been engineered, even if it is to say that it is fixed.
I also can't imagine that this issue would be any different between ASM and ARM. It seems exceptionally unlikely that there was originally a high availability solution at the backend that they scrapped when ARM came along. So it is very likely that cloud services would suffer the exact same issue.
The small upside is that now that we know this is an issue, one possible solution would be to deploy multiple web apps and have a traffic manager between them. Even if they are in the same region, different apps should have different backend file servers.
My first action would be to reply to that email, with a link to the Web Apps page, (and this question) with a copy of the quote and ask how to enable high availability within a geographic region.
After that you'll likely need to rearchitect your solution!
Availability sets
For virtual machines Azure will let you specify an availability set. An availability set will automatically split VMs into separate update and fault domains. Meaning that servers will end up in different server racks, and those server racks won't get updates at the same time. (it is a little more complex than that, but that's the basics!)
Azure Web Apps do used a shared file storage. The best way to think about it is that all the instances of your app map to the same network share that have your files. So if you modify the files by any mean (e.g. FTP, msdeploy, git, ...), all the instances instantly get the new files (since there is only one set of files).
And to answer your final question, each instance does run on a separate VM.
So I have implemented ELK using REDIS as a caching layer.
I am using REDIS 3.0.4 from an RPM I found for Redhat EL6.
I am also running jemalloc 3.6.0.
I believe the configuration for REDIS is largely vanilla with the exception of a max memory cap and also non-default eviction policy.
maxmemory 500mb
and
maxmemory-policy allkeys-random
though I feel that the eviction policy is probably not required.
Now I have verified that the store is generally empty, i.e. my logstash indexer is doing its job well and the data is making it into Elastic Search.
What concerns me is the used memory for REDIS continues to trend upwards and from what I have seen, if used memory hits max, which it has done, then REDIS stop working, i.e. no more log entries flow through.
So, what am I missing:
Am I being paranoid and can dismiss what I am seeing.
Should I avoid the pre-packaged RPMs.
Is there some additional settings I need to change.
Everything I have read to date about REDIS and ELK suggests that out of the box should be fine.
Be aware that this is a lightweight implementation which I hope will provide impetus for a more broader, bullet proof implementation, which is the reason for the 500MB limit.
We want to save a few bucks and share our 1GB dedicated Azure Redis Cache between Development, Test, QA and maybe even production.
Is there a better way than prefixing all keys with an environment string like "Dev_[key]", "Test_[key]" etc.
We are using the StackExchange Redis client for .NET.
PS: We tried using the cheap 250GB (Shared infrastructure), but had very slow performance. Read operations were consistent between 600-800ms... without any load (for a ~300KB object). Upgrading to dedicated 1GB services changed that to 30-40ms. See more here: StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors
One approach is to use multiple Redis databases. I'm assuming this is available in your environment :)
Some advantages over prefixing your keys might be:
data is kept separate, you can flushdb in test and not touch the production data
keys are smaller and consume less memory
The main disadvantage would be not taking advantage of multiple cores, like you could do if you ran multiple instances of Redis on the same server. Obviously not an issue in this case. Also note that this feature is not deprecated, like one of the answers suggests.
Another thing I've seen people complain about is that databases are numbered, they don't have meaningful names. Some people create a hash in database 0 that maps each number to a name.
Here is another idea to save some bucks: use separate Redis cache machines for each environment - so no problems with the keys, but stop them when you don't use them, like in the weekend and during nights. Probably more than 50% of the time you are not using them. I think it would be easy to start and stop them with some PowerShell script, we are using AWS and here it is possible.
Now from what I see the Redis persistence in Azure is not enabled, but they started working on it http://feedback.azure.com/forums/169382-cache/status/191763 - it would be nice to do a RDB snapshot before stopping and then on start to load it. So if you need to save some values and reload them on start you should do it manually (with your own service).