I am working on a project in which I need a set of data frequently and currently for getting that data I have to make call to 3rd party Service which is taking lot of time.So what I want is to maintain a local cache.The Data is modified very infrequently or is almost constant.What is the best way of implementing this in Azure Service Fabric.I am currently thinking of making the Microservice stateful. Is is the best way to do this?When node goes down it should copy its local cache to other node.If making it stateful is good than How should i go on implementing this?
Well, you have two options:
If you need performance and geografical cache data replication, you can use a redis cache.
The other option is use reliable dictionary's. It's a service fabric feature, and reliable dictionary's are replicated to other nodes.
You can only access the reliable dictionary in a service fabric statefull context.
Example bellow:
IReliableDictionary<string, string> Dictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<string, string>>("stringlistcache");
using (var tx = this.StateManager.CreateTransaction())
{
await pingDictionary.AddOrUpdateAsync(tx, "testkey", "testvalue", (key, value) => "testvalue");
cachedValue = await Dictionary.TryGetValueAsync(tx, "testkey");
await Dictionary.TryRemoveAsync(tx, "testkey");
await tx.CommitAsync();
}
I know this post is quite old, but for those looking for a solution today, you can use this open source project:
http://service-fabric-distributed-cache.socreate.it
you can also find it on GitHub here:
https://github.com/SoCreate/service-fabric-distributed-cache
Related
What if I put multiple function inside a single cloud function so that its instance lives at max and that I will have to deal with cold start once?
Why is this a bad idea?
export const shop = functions.https.onCall(async (data, context) => {
switch (data.type) {
case "get_fruits":
return await getFruits();
case "place_order":
return await placeOrder();
case "add_to_cart":
return await addToCart();
default:
return;
}
});
It will work but, IMO, it's not a good thing to do. There are many principles and patterns that exist today and that you do not enforce your solution.
Microservice
One of them is the split in microservices. There is no problem to build a monolith, but when I'm seeing your example (get_fruit, place_order, add_to_cart), I'm seeing different roles and responsibilities. I love the separation of concern: 1 service does 1 thing.
Routing
But, maybe your service is only a service for the routing and call functions deployed independently (and you enforce the microservice principle). If so, your service can become a bottleneck, if there are a lot of entries and a lot of queries.
In addition, there are services dedicated for routing: load balancers. They use the URL path of the requests and reach the correct microservices to serve them
Developer usage
Yes a URL, not a field in the body of your message to route the traffic. Today, the developers are familiar with the REST API. To get the fruit, they perform a GET request to the /fruit URL and they know they will get the fruits. If they want to add to the cart, they perform a POST request to the /cart URL and it works!
You USE URL, standard REST definition, load balancers and microservices.
You can imagine other benefits:
Each microservice can scale independently (you can have more get_fruit request than place_order, the service scale differently)
The security is easier to control (no security to get the catalog (fruits)), but you have to be authenticated to place an order
Evolution velocity can be decoupled between the services
...
I'm looking to use the Change Feed Processor SDK to monitor for changes to an Azure Cosmos DB collection, however, I have not seen clear documentation about whether the host can be run as an Azure Web Job. Can it? And if yes, are there any known issues or limitations versus running it as a Console App?
There are a good number of blog posts about using the CFP SDK, however, most of them vaguely mention running the host on a VM, and none of them or any examples running the host as an azure web job.
Even if it's possible, as a side question is, if such a host is deployed as a continuous web job and I select the "Scale" setting of the web job to Multi Instance, what are the approaches or recommendations to make the extra instances run with a different instance name, which the CFP SDK requires?
According to my research,Cosmos db trigger could be implemented in the WebJob SDK.
static async Task Main()
{
var builder = new HostBuilder();
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddCosmosDB(a =>
{
a.ConnectionMode = ConnectionMode.Gateway;
a.Protocol = Protocol.Https;
a.LeaseOptions.LeasePrefix = "prefix1";
});
});
var host = builder.Build();
using (host)
{
await host.RunAsync();
}
}
But it seems only Nuget for c# sdk could be used,no clues for other languages.So,you could refer to the Compare Functions and WebJobs to balance your needs and cost.
The Cosmos DB Trigger for Azure Functions it's actually, a WebJobs extension: https://github.com/Azure/azure-webjobs-sdk-extensions/tree/dev/src/WebJobs.Extensions.CosmosDB
And it uses the Change Feed Processor.
Functions run over WebJob technology. So to answer the question, yes, you can run Change Feed Processor on WebJobs, just make sure that:
Your App Service is set to Always On
If you plan to use multiple instances, make sure to set the InstanceName accordingly and not a static/fixed value. Probably something that identifies the WebJob instance.
We are designing an Azure Website which will allow users to Upload content(MP4,Docx...MSOffice Files) which can then be accessed.
Some video content we will encode to provide several differing quality formats, before it will be streamed (using Azure Media Services).
We need to add an intermediate step so we can scan uploaded files for potential virus risk. Is there functionality built into azure (or third party) which will allow us to call an API to scan content before processing it? We are ideally looking for an API rather than just a background service on a VM, so we can get feedback potentially for use in a web or worker role.
Had a quick look at Symantec Endpoint and Windows Defender but not sure these offer an API
I have successfully done this using the open source ClamAV. You don't specify what languages you are using, but as it's Azure I'll assume .Net.
There is a .Net wrapper that should provide the API that you are looking for:
https://github.com/tekmaven/nClam
Here is some sample code (note: this is copied directly from the nClam GitHub repo page and reproduced here just to protect against link rot)
using System;
using System.Linq;
using nClam;
class Program
{
static void Main(string[] args)
{
var clam = new ClamClient("localhost", 3310);
var scanResult = clam.ScanFileOnServer("C:\\test.txt"); //any file you would like!
switch(scanResult.Result)
{
case ClamScanResults.Clean:
Console.WriteLine("The file is clean!");
break;
case ClamScanResults.VirusDetected:
Console.WriteLine("Virus Found!");
Console.WriteLine("Virus name: {0}", scanResult.InfectedFiles.First().VirusName);
break;
case ClamScanResults.Error:
Console.WriteLine("Woah an error occured! Error: {0}", scanResult.RawResult);
break;
}
}
}
There are also APIs available for refreshing the virus definition database. All the necessary ClamAV files can be included in the deployment package and any configuration can be put into the service start-up code.
ClamAV is a good idea, specially now that 0.99 is about to be released with YARA rule support - it will make it really easy for you to write custom rules and allow clamav to use tons of good YARA rules in the open today.
Another route, and a bit of shameless plugging, is to check out scanii.com, it's a SaaS for malware/virus detection and it integrates quite nicely with AWS and Azures.
There are a number of options to achieve this:
Firstly you can use ClamAV as already mentioned. ClamAV doesn't always receive the best press for its virus databases but as others have pointed out it's easy to use and is expandable.
You can also install a commercial scanner, such as avg, kaspersky etc. Many of these come with a C API that you can talk to directly, although often getting access to this can be expensive from a licensing point of view.
Alternatively you can make calls to the executable directly using something like the following to capture the output:
var proc = new Process {
StartInfo = new ProcessStartInfo {
FileName = "scanner.exe",
Arguments = "arguments needed",
UseShellExecute = false,
RedirectStandardOutput = true,
CreateNoWindow = true
}
};
proc.Start();
while (!proc.StandardOutput.EndOfStream) {
string line = proc.StandardOutput.ReadLine();
}
You would then need to parse the output to get the result and use it within your application.
Finally, now there are some commercial APIs available to do this kind of thing such as attachmentscanner (disclaimer I'm related to this product) or scanii. These will provide you with an API and a more scalable option to scan specific files and receive the response from at least one virus checking engine.
New thing coming Spring / Summer 2020. Advanced threat protection for Azure Storage includes Malware Reputation Screening, which detects malware uploads using hash reputation analysis leveraging the power of Microsoft Threat Intelligence, which includes hashes for Viruses, Trojans, Spyware and Ransomware. Note: cannot guarantee every malware will be detected using hash reputation analysis technique.
https://techcommunity.microsoft.com/t5/Azure-Security-Center/Validating-ATP-for-Azure-Storage-Detections-in-Azure-Security/ba-p/1068131
How I can refresh spring cache when I inserted data into Database through my services and when I added data directly into database.Can we achieve this?.
Note:
I am using following libs
1)net.sf.json-lib
2)spring-support-context
through my services
This is typically achieved in your application's services (e.g. #Service application components) using the Spring #Cacheable, #CachePut annotations, for example...
#Service
class BookService {
#Cacheable("Books")
Book findBook(ISBN isbn) {
...
return booksRepository().find(isbn);
}
#CachePut(cacheNames = "Books", key = "#book.isbn")
Book update(Book book) {
...
return booksRepository.save(book);
}
}
Both #Cacheable and #CachePut will update the cache provider as the underlying method may callback through to the underlying database.
and when I added data directly into database
This is typically achieved by the underlying cache store. For example, in GemFire, you can use a CacheLoader to "read-through" (to your underlying database perhaps) on "cache misses". See GemFire's user guide documentation on "How Data Loaders Work" as an example and more details.
So, back to our example, if the "Book (Store)" database was updated independent of the application (using Spring's Caching Annotation support and infrastructure), then a developer just needs to define a strategy on cache misses. And, in GemFire that could be a CacheLoader, and when...
bookService.find(<some isbn>);
is called resulting in a "cache miss", GemFire's CacheLoader will kick in, load the cache with that book and Spring will see it as a "cache hit".
Of course, our implementation of bookService.find(..) went to the underlying database anyway, but it only retrieves a "single" book. A loader could be implemented to populate an entire set (or range) of books based on some criteria (such as popularity), where the application service expects those particular set of books to be searched for by potential customers, using the application, first, and pre-cache them.
So, while Spring's Cache annotations typically work per entry, a cache store specific strategy can be used to prefetch and, in-a-way, "refresh" the cache, lazily, on the first cache miss.
In summary, while the former can be handled by Spring, the "refresh" per say is typically handled by the caching provider (e.g. GemFire).
Hopefully this gives you some ideas.
Cheers,
John
I'm getting started with ServiceStack and I've got to say I'm very impressed with all it has under the bonnet and how easy it is to use!
I am developing a predominantly read-only application with it. There will likely be updates to the database 3 or 4 times a year but the rest of the time the solution will be displaying data on an electronic information board (large touch screen monitor).
The database structure is well normalised with a few foreign keyed tables and with this in mind I think it may be best to separate the read only API from the CRUD API. The CRUD API can be used to create and modify the relational data with POCO classes matching the database tables. I would then ensure the read-only API flattens the relational data into a few POCOs spanning a few db tables making the data easier to handle on the read-only UIs.
I'm just looking for ideas and advice really on whether this separation of concerns is wasted effort or if there is a better way of achieving what I need? Has anyone had similar thoughts / ideas?
Having developed a similar read only application (a gazetteer, updated quarterly/yearly) using ServiceStack we went with optimizing the API for reads, making use of the built in caching:
// For cached responses this has to be an object
public object Any(CachedRequestDto request)
{
string cacheKey = request.CacheKey;
return this.RequestContext.ToOptimizedResultUsingCache(
base.Cache, cacheKey, () =>
{
using (var service = this.ResolveService<RequestService>())
{
return service.Any(request.TranslateTo<RequestDto>()).TranslateTo<CachedResponseDto>();
}
});
}
Where CacheKey is just:
public string CacheKey
{
get
{
return UrnId.Create<CachedRequestDto>(string.Format("{0}_{1}", this.Field1, this.Field2));
}
}
We did start creating a CRUD / POCO service, but for speed went with using bulk import tools such SQL Server DTS/SSIS or console apps which suffices for now, and will revisit this later if required.
Might want to consider something like CQRS.
https://gist.github.com/kellabyte/1964094 (or Google for CQRS Martin Fowler, can only post 2 links).
Also found the following article valuable recently when starting to implement additional search type services: https://mathieu.fenniak.net/stop-designing-fragile-web-apis/