Hi everyone!
I have "ELK" (6.4.2) working perfectly with filebeat, metricbeat, packetbeat and winlogbeat in CentOS 7 x86_64 (Kernel 3.10.0-862.11.6.el7.x86_64).
I'm trying to integrate zipkin + elk (see https://logz.io/blog/zipkin-elk/), but Elasticsearch does not create indices with Kibana.
When trying to create the indices in Kibana, the process does not end. (Follow logs below).
I suspect the zipkin connection drivers are not compatible with elk 6.4.2. Has anyone had the same problem and has a "light at the end of the tunnel"?
Tks for all!
Java version:
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
Zipkin startup:
java -DSTORAGE_TYPE=elasticsearch -DES_HOSTS=http://localhost:9200 -jar /opt/zipkin.io/bin/zipkin.jar
Error log in Elasticsearch:
[2018-10-24T11:31:59,933][WARN ][o.e.d.i.m.MapperService ] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2018-10-24T11:31:59,936][WARN ][o.e.d.i.m.MapperService ] [_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-10-24T11:31:59,954][WARN ][o.e.d.i.m.MapperService ] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2018-10-24T11:32:00,033][WARN ][o.e.d.c.m.MetaDataCreateIndexService] index or alias name [zipkin:span-2018-10-24] containing ':' is deprecated and will not be supported in Elasticsearch 7.0+
[2018-10-24T11:32:00,245][WARN ][o.e.d.i.m.MapperService ] Setting index.mapper.dynamic is deprecated since indices may not have more than one type anymore.
[2018-10-24T11:33:47,717][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
Here is the related issue:
we also mentioned recently that for data to appear, applications need to be
sending traces
https://github.com/openzipkin/zipkin#quick-start
you can tell also by hitting the /metrics endpoint and look at stats named
collector
https://github.com/openzipkin/zipkin/issues/1939
I opened a issue on the zipkin github, a theme already being treated as a bug.
Initial thread:
https://github.com/openzipkin/zipkin/issues/2218#issuecomment-432876510
Bug track:
https://github.com/openzipkin/zipkin/issues/2219
Tks for all!
Related
Does anyone have an idea on how I can do this (what the top comments suggests): Keycloak logging to logstash
But for the newest keycloak version 17+ which uses Quarkus instead of Wildfly?
Edit 1:
After I have successfully added logstash feature to my keycloak+quarkus build, and logging-gelf shows under installed features
I can still not see logs on kibana, as if the logs are not being sent but the feature is installed.
Am I correct in thinking that this configuration is a correct "replacement" for the the Wildfly cli from the link:
quarkus.log.handler.gelf.enabled=true
quarkus.log.handler.gelf.json=true
quarkus.log.handler.gelf.json.key-overrides={timestamp="#timestamp"}
quarkus.log.handler.gelf.json.exception-output-type=formatted
quarkus.log.handler.gelf.json.additional-field."appname".value=${env.LOGSTASH_APPNAME:myApplicationName}
quarkus.log.handler.gelf.json.additional-field."#version".value=1
quarkus.log.handler.gelf.host=tcp:${env.LOGSTASH_DESTINATION:someDefaultDestination}
quarkus.log.handler.gelf.port=${env.LOGSTASH_PORT:5044}
#Default vale is tcp
#quarkus.log.handler.gelf.protocol=tcp
quarkus.log.handler.gelf.block-on-reconnect=true
quarkus.log.handler.gelf.async=true
#Default value is ALL
#quarkus.log.handler.gelf.level=ALL
quarkus.log.handler.gelf.async.queue-length=${env-LOGSTASH_QUEUE:1000}
#Default value is block
#quarkus.log.handler.gelf.async.overflow=block
Or is there some configurations that I just have "made up"?
First of all: at the moment keycloak has no native support for gelf/logstash directly. That said, it is possible to do for sure using the quarkus logging gelf extension. In order for this to work, it is a bit complicated, but in general You have to:
download the jars for the quarkiverse-logging-gelf - both, the "deployment" and the runtime jar, for the quarkus version your keycloak version uses (2.7.0? Not sure, iirc I updated to 2.7.5 in 17.0.1). Also you need to download the transient dependency of this quarkus extension, namely the logstash-gelf jar in the right version. Put these jars inside the "providers" directory of your kc.
create a quarkus.properties file in the conf directory and add your desired configuration. There is a centralized log management guide on the quarkus homepage.
run a "build"
start Keycloak. Your log should now show
[io.quarkus] (main) Installed features: [..., logging-gelf, ...]
And it should work to get your logs out the way you want it.
These steps are also described with examples e.g. here.
Note: I am in the Keycloak team. There is no out of the box support right now directly, bc. we are thinking about leveraging quarkus' extension approach better (so you do not have to download the jars and transient dependencies manually), but I guess for this specific case we will do something the future :)
edit: open pr for support using keycloak config: -> https://github.com/keycloak/keycloak/pull/12843
edit 2: pr merged, firstclass support should be available from the next major version (19 i guess)
I am experiencing difficulties to get started using StormCrawler using the StormCrawler+ElasticSearch archetype. On the StormCrawler website, I see two versions, namely 1x and 2x. Similarly, Apache Storm comes in version 1 and 2.
Should I install StormCrawler using the version 1x or 2x?
What version of JDK does StormCrawler require? Is there a need to use Oracle JDK or can the OpenJDK be used as well?
I want to use StormCrawler to identify and process images and documents. At what place in the topology can these tasks best be added?
Update: According to the following URL (Storm Crawler with Java 11), StormCrawler 2 is advised. What StormCrawler+ElasticSearch archetype should be used when using StormCrawler 2?
SC 1.x is stable and the current version, 2.x is less tested but will be the main version at some point.
The thread mentioned in the question does not advise you to use SC2 as such, it mentions that you should use it if you need Java 11. If you are on Java 8, then you can use whichever version you want.
SC works fine with openjdk.
As for question #3, it depends what you want to do. Can you please elaborate?
I've used CTL in ExecuteGroovyScript processor to connect to database which is selected by database.name due to this problem DBCPConnectionPoolLookup problem in ExecuteGroovyScript. after upgrading NiFi to new version (1.10.0), the processors show this error:
and this is my processor configuration:
Please let me know the problem.
I started testing version 1.10.0 myself recently and that was one of the things I ran in to as well. Try using SQL.* (ex: SQL.db) for your property name as that is used specifically for DBCP services. If you need to access other services that still can be done by looking up the service by name in your Groovy script as a workaround but based on Jira activity, I see this bug is already being worked on and hopefully will be fixed in next release. I hope this helps.
I'm trying to look at using the Stratios Lucene index plugin (on Windows)installation of Cassandra (Datastax v3.5) but can't get Cassandra to recognize it.
I'm aware that you must use the corresponding version to Cassandra and have tried with 3.0.5 & 3.5 but both with the same results. The service is stopped, the index .jar file is copied to the lib directory & then the service is restarted. Then using CQLSH, I can create the relevant keyspace & table (as described in the Stratio documentation) but when attempting to create the index it fails with the following message:
Query invalid because of configuration issue: message="Unable to find custom indexer class 'com.stratio.cassandra.lucene.Index'"
https://github.com/Stratio/cassandra-lucene-index/tree/branch-3.5
Does anyone have any idea how to get this implemented & working?
Is there a central forum or a point of contact for Stratios Lucene index support?
This resource https://github.com/Stratio/cassandra-lucene-index/issues/118#issuecomment-211796434 suggests that only open source Apache Cassandra is officially supported by this plugin. It might work with DSE, might not. I checked 3.5.0 version works on Linux with Apache Cassandra but does not work on Windows with DSE :( According to Datastax docs, it should support custom secondary indexes. So, it might be the plugin does not run on Windows?
I've been trying to work through an issue that developed while attempting to upgrade our testing environment from 12.04 to 14.04 ubuntu on aws. Prior to this, the Package repository version of solr was 1.4.1 which matched our 1.4.1 solrj client integrated with our application.
Changing the base AMI to the 14.04 latest and running our default deploy caused solr 3.6.2 server to be installed. It appears it was accepting our configs without issue, however when our client tried to connect we received different errors:
The first was an unknown custom field, which we traced back to our deployment scripts not moving our schema.xml and solrconfig.xml to /etc/solr/conf/ but keeping it in the base directory.
We corrected this issue, and then ran into the following:
'exception: Invalid version or the data in not in 'javabin' format'
This was generated by a wrapper ontop of solrj, but I'll be honest and say I know nothing regarding Solr and that this may be on our end. I've asked our dev team to look at 2 options:
1) enabling: 'server.setParser(new XMLResponseParser());'
Which is the recommendation on the backwards compatibility for an older client.
2) updating our client in the application to 3.6.2
-I know less about the requirements on this.
My fall back is to revert to 1.4.1, but it appears it hasn't been touched since 2011, which makes me hesitant.
Any thoughts / suggestions would be appreciated!
Thanks!
I think the best option is to maintain the same version of Solr and Solrj.
I used for a lot of time Solr 1.4.1 and, while as you said, the most part of it works with newer versions without any problem, actually a lot of things have been changed since 1.4.*
I did your same porting last year, (from 1.4.1 to 3.6.1) and I can confirm you that the 2nd way is the right one: all changes you must do in your client code are just "formal" and very very quick.
Any workaround you could do for being able to communicate with a different version (between Solrj and Solr) is just, as the word says, a "workaround" and it could lead to unexpected (hidden) side-effects later.