How to re-purpose existing instances of Redis and RabbitMQ with ServiceStack - servicestack

After successfully developing an application with multiple ServiceStack services, we are moving to other testing environments, lots of them due to us running a SAAS model (aka multi-tenant). I'd like to reuse some of the base infrastructure services, primarily Redis and RabbitMQ across a few of these environments.
We're using the IAppSetting interface to pull our configuration from multiple sources into one cohesive settings object at run-time, which is then filtered tier. Since tier drives the configuration per environment it made sense to use Tier to prefix any RabbitMQ messages queues, and prefix any generated cache keys that will be used by Redis, thus providing collision protection per environment.
Below is an example:
RabbitMQ => "Some MQ method here" => "mq:qa1.Outbound.inq"
Redis => "Some Redis method here" => "urn:qa1.somePoco:123"
Here is an example configuration and the various enviroments
<appSettings>
<add key="Tier" value="qa1" />
<!--<add key="Tier" value="dev" />-->
<!--<add key="Tier" value="tst" />-->
<!--<add key="Tier" value="stg" />-->
<!--<add key="Tier" value="prod" />-->
</appSettings>
Thank you,
Stephen

Some examples on how to modify Queue Names are in MqNameTests, e.g:
QueueNames.SetQueuePrefix("site1.");
Will add a prefix on QueueNames, e.g:
site1.mq:TestPrefix.inq
Otherwise you can use QueueNames.ResolveQueueNameFn to have complete control over the MQ name, e.g:
QueueNames.ResolveQueueNameFn = (typeName, suffix) =>
"SITE.{0}{1}".Fmt(typeName, suffix.ToUpper());
QueueNames<TestFilter>.In.Print(); // SITE.TestFilter.INQ
Note the same configuration also needs to applied on the client so the same MQ names gets used.
Configuring ServiceStack with AppSettings
ServiceStack is a code-first framework which means all configuration is done in code, but has a rich and versatile configuration model where you can get the behavior your after by reading App Settings in AppHost.Configure():
QueueNames.SetQueuePrefix(AppSettings.Get("Tier","dev"));
Where if Tier doesn't exist in your Web.config (e.g. in Unit Tests) it will use dev otherwise will use the value in your appSettings:
<appSettings>
<add key="Tier" value="qa1" />
</appSettings>

Related

No performanceCounters/Requests Data in azure Application Insights

I have a web app running on prem (IIS) and I am trying to collect performance counters from it, although I have everything setup on my local and I am seeing all other types of telemetry in my Azure resource (requests, exceptions, traces) performance Counters are just not there,
it is worth mentioning that I am running 2 more web applications (they all work together) under the same site and same applicationPool in IIS and they are collecting Performance counters just fine, they all use the same package that contains our Application Insights implementation. Something strange I have noticed is that, when I am serving the application from Visual Studio, all the telemetry goes through, even performance counters, but, when I try with an installed Instance of my webapplications (using our inhouse installer ) any request or perfCounter for this particular web application just won't work.
I don't know what else to check, all 3 web applications should be collecting performance counters the same way since they are pretty much alike, and running under the same AppPool, same goes for requests.
this is what my appInsights.config looks like:
<Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
<Counters>
<Add PerformanceCounter="\Processor(_Total)\% Processor Time" ReportAs="Processor Total - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_W3SVC_PROC??)\% Processor Time" ReportAs="W3SVC - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\% Processor Time" ReportAs="Win32 - % Processor Time"/>
<Add PerformanceCounter="\Process(??APP_CLR_PROC??)\% Processor Time" ReportAs="CLR - % Processor Time"/>
<Add PerformanceCounter="\Process(_Total)\Private Bytes" ReportAs="Process Total - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_W3SVC_PROC??)\Private Bytes" ReportAs="W3SVC - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Private Bytes" ReportAs="Win32 - Private Bytes"/>
<Add PerformanceCounter="\Process(??APP_CLR_PROC??)\Private Bytes" ReportAs="CLR - Private Bytes"/>
</Counters>
<!--
Use the following syntax here to collect additional performance counters:
<Counters>
<Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Handle Count" ReportAs="Process handle count" />
...
</Counters>
PerformanceCounter must be either \CategoryName(InstanceName)\CounterName or \CategoryName\CounterName
NOTE: performance counters configuration will be lost upon NuGet upgrade.
The following placeholders are supported as InstanceName:
??APP_WIN32_PROC?? - instance name of the application process for Win32 counters.
??APP_W3SVC_PROC?? - instance name of the application IIS worker process for IIS/ASP.NET counters.
??APP_CLR_PROC?? - instance name of the application CLR process for .NET counters.
-->
</Add>
few more details: AppInsights sdk 2.4
.NetFramework 4.6.1
Thanks, all help is welcome

How to log HTTP requests sent by ODataQueryBuilder API / VDM API?

Using latest version of Java SAP Cloud SDK
We have some code which uses ODataQueryBuilder API and VDM API as well. We want to log the HTTP requests that are being sent by these API's. We want to log whole of the HTTP request - headers, body everything. Please note that our application is running on SAP Cloud Platform's Cloud Foundry PAAS offering and using cf set-logging-level doesn't seem to work.
I've been using this Java arg when debugging my requests, but I've been doing it locally.
-Dorg.slf4j.simpleLogger.log.org.apache.http.wire=debug
If you can pass it withing CF environment I think you should start seeing all the payloads. I'll research a bit more to provide a better guidance if this won't work for you.
For applications deployed on SCP CF, there are different setups for which recommend other logging practices. The goal is to configure individual log levels for specific packages of your application and third-party dependencies, e.g. SAP Cloud SDK or SAP Service SDK or Apache HTTP components.
TomEE based application:
Edit the manifest.yml to include the following env entry for environment variable:
SET_LOGGING_LEVEL: '{ROOT: INFO, com.sap.cloud.sdk: INFO, org.apache.http.wire: DEBUG}'
Feel free to customize.
Spring Boot based application:
We expect the logback framework.
Edit/Create the file: application/src/main/resources/logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProfile name="!cloud">
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<root level="INFO"/>
<logger name="org.springframework.web" level="INFO"/>
</springProfile>
<springProfile name="cloud">
<appender name="STDOUT-JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="com.sap.hcp.cf.logback.encoder.JsonEncoder"/>
</appender>
<logger name="org.springframework.web" level="INFO"/>
<logger name="com.sap.cloud.sdk" level="INFO"/>
<logger name="org.apache.http.wire" level="DEBUG"/>
<root level="INFO">
<appender-ref ref="STDOUT-JSON"/>
</root>
</springProfile>
</configuration>
Feel free to customize.
Notice the different profile settings. Make sure the cloud profile is active for deployed applications. Edit the manifest.yml to include the following env entry for environment variable:
SPRING_PROFILES_ACTIVE: 'cloud'

ServiceStack Utility to read from Custom Config Sections

Is there any ServiceStack utility that can read from custom config sections. ServiceStack has IAppSettings which makes it easy to read from appSettings in a config file. I am wondering if ServiceStack has a similar utility to read from Custom Config Sections.
Thanks
rudrvij
No there isn't, we actively discourage usage of Config Sections which is XML encumbered, in-flexible, non-portable and non-substitutable.
Our preference (when needed) is instead to embed Complex Type configuration in a single AppSetting Text Value which is transparently supported using the human-friendly JSV Format, e.g:
<appSettings>
<add key="RedisConfig"
value="{Host:localhost,Port:6379,Database:1,Timeout:10000}" />
</appSettings>
Which is supported by every App Settings Provider and can be easily serialized into any POCO with:
RedisConfig redisConf = appSettings.Get<RedisConfig>("RedisConf");

Configuring Jersey Test Framework with Security

I am writing a REST web service using Jersey, and I'm trying to write a set of unit tests to test the service using the Jersey Test Framework.
However, I use HTTP Authentication and SecurityContext as part of my web service, and I'm having issues setting up JTF to allow me to test these aspects. I can send authentication information in the request, but how do I configure it to know about the different roles and users I wish to set up?
I'm currently using Jetty (via JettyTestContainerFactory), but can switch to different test containers if needed.
The specific configuration I am trying to achieve is two roles, and four users with the combinations of those possible roles (e.g. No roles, role a, role b, roles a and b). The web service will handle giving access to different URLs, so that doesn't need to be specified in the configuration.
I have done this by implementing my own Jetty Test container similar to the one provided by Jersey. We use an embedded Jetty for testing our application in development normally and by creating our own test container based on that embedded Jetty it loads our web application as it would if it was started by a Java main process.
We use a custom Jetty Security Handler configured in a jetty-env.xml file which the embedded Jetty uses to configure the security.
<Set name="securityHandler">
<New class="com.example.DevelopmentSecurityHandler">
<Set name="loginService">
<New class="com.example.DevelopmentLoginService">
<Set name="name">LocalRealm</Set>
<Set name="config">src/main/webapp/WEB-INF/users.properties</Set>
<Call name="start" />
</New>
</Set>
<Set name="authenticator">
<New class="com.example.DevelopmentAuthenticator"></New>
</Set>
<Set name="checkWelcomeFiles">true</Set>
</New>
</Set>
That Jetty env file is loaded by embedded Jetty:
XmlConfiguration configuration = null;
if (jettyEnvFile.exists()) {
try {
configuration = new XmlConfiguration(jettyEnvFile.toURI().toURL());
} catch (Exception e) {
throw new ProcessingException(String.format("Exception loading jetty config from %s", jettyEnvFile));
}
} else {
LOG.warn("No jetty-env.xml found.");
}
The users.properties file referenced in that xml is a simple user to role mapping e.g.
USERNAME=PASSWORD,ROLE_NAME1,ROLE_NAME2
Depending how you configure your Jetty security this may or may not work for you. You can also configure this programmatically, there's lots of examples of embedded Jetty here. The SecuredHelloHandler.java example there could be a good start for you.
For the test container you can basically start by copying org.glassfish.jersey.test.jetty.JettyTestContainerFactory and org.glassfish.jersey.jetty.JettyHttpContainerFactory essentially changing the
public static Server createServer(final URI uri, final SslContextFactory sslContextFactory, final JettyHttpContainer handler, final boolean start)
method to create your version of an embedded Jetty server with security configured however you require.

azure cache preview

I tried the new azure preview that came with the new sdk on my computer.
I put a worker role with cache preview and put co-located role with 30% cache size.
on my controller i put this code:
[OutputCache(Duration=int.MaxValue, VaryByParam="none")]
public ActionResult Index()
{
ViewBag.Message = "Welcome to ASP.NET MVC!";
ViewBag.Id = Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CurrentRoleInstance.Id;
return View();
}
now i ran the worker role via the emulator with 4 instances. the result was that every time i saw a different id - which mean the output cache never work with all the 4 instances ( to be clear i configure the output cache to work with the cache preview).
Only when i put an extra cache worker role as dedicated role everything start to work like it should be.
My questions is:
Do i need the extra worker role to actually make the cache preview to work ok? - which mean the trade off of not working with azure appfabric cache is putting extra machine
Did i do something work and it should work with the web roles as co located roles?
thanks
edit:
this another section of my web.config
<dataCacheClients>
<tracing sinkType="DiagnosticSink" traceLevel="Error" />
<dataCacheClient name="default">
<autoDiscover isEnabled="true" identifier="NugetTest" />
<!--<localCache isEnabled="true" sync="TimeoutBased" objectCount="100000" ttlValue="300" />
</dataCacheClient>
if my identifier have NugetTest ( which is my web roles - which i have 4) every time i switch machine i get a different cache. if i change the identifier to my worker role i get the result
Can you add applicationName tag in the provider configuration in web.config of you app? If this is not added, instances will not share the cache across. Please note the applicationName tag.
This should be added for the web.config of webrole in both dedicated or colocated cache scenario.
Please reply if this solves your issue.
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache" type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache" cacheName="<cacheName>" applicationName ="<anyName>" dataCacheClientName="<dataCacheClientName>" />
</providers>
</outputCache>
</caching>
I'm unable to reproduce this issue. I always see the same instance, and I'm using Ctrl+F5 in the browser (thus rule out browser cache). PLease make sure you've configured output cache provider as described on http://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/.
<!-- If output cache content needs to be saved in a Windows Azure
cache, add the following to web.config inside system.web. -->
<caching>
<outputCache defaultProvider="DistributedCache">
<providers>
<add name="DistributedCache"
type="Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider, Microsoft.Web.DistributedCache"
cacheName="default"
dataCacheClientName="default" />
</providers>
</outputCache>
</caching>
Best Regards,
Ming Xu.

Resources