Building a Spatial Index - geospatial

I want to build spatial index on my Jena Fuseki server.
I try to follow the doc here : https://jena.apache.org/documentation/query/spatial-query.html
But when I restart my server I have this :
[2018-05-24 17:05:03] Server INFO Apache Jena Fuseki 3.7.0
[2018-05-24 17:05:03] Config INFO FUSEKI_HOME=C:\Users\i3mainz\Desktop\JenaFusekiProject\apache-jena-fuseki-3.7.0.
[2018-05-24 17:05:03] Config INFO FUSEKI_BASE=C:\Users\i3mainz\Desktop\JenaFusekiProject\apache-jena-fuseki-3.7.0\run
[2018-05-24 17:05:03] Config INFO Shiro file: file://C:\Users\i3mainz\Desktop\JenaFusekiProject\apache-jena-fuseki-3.7.0\run\shiro.ini
[2018-05-24 17:05:04] Config INFO Configuration file: C:\Users\i3mainz\Desktop\JenaFusekiProject\apache-jena-fuseki-3.7.0\run\config.ttl
[2018-05-24 17:05:04] riot WARN [line: 32, col: 1 ] Bad IRI: Code: 4/UNWISE_CHARACTER in PATH: The character matches no grammar rules of URIs/IRIs. These characters are permitted in RDF URI References, XML system identifiers, and XML Schema anyURIs.
[2018-05-24 17:05:04] Server ERROR Exception in initialization: No rdf:type for dataset :spatial_dataset
[2018-05-24 17:05:04] WebAppContext WARN Failed startup of context o.e.j.w.WebAppContext#ca27722{/,file:///C:/Users/i3mainz/Desktop/JenaFusekiProject/apache-jena-fuseki-3.7.0/webapp/,UNAVAILABLE}
org.apache.jena.fuseki.FusekiConfigException: No rdf:type for dataset :spatial_dataset
at org.apache.jena.fuseki.build.FusekiBuilder.getDataset(FusekiBuilder.java:119)
at org.apache.jena.fuseki.build.FusekiBuilder.buildDataServiceCustom(FusekiBuilder.java:73)
at org.apache.jena.fuseki.build.FusekiBuilder.buildDataAccessPoint(FusekiBuilder.java:65)
at org.apache.jena.fuseki.build.FusekiConfig.servicesAndDatasets(FusekiConfig.java:121)
at org.apache.jena.fuseki.server.FusekiSystem.processServerConfigFile(FusekiSystem.java:266)
at org.apache.jena.fuseki.server.FusekiSystem.initServerConfiguration(FusekiSystem.java:241)
at org.apache.jena.fuseki.server.FusekiSystem.initializeDataAccessPoints(FusekiSystem.java:207)
at org.apache.jena.fuseki.server.FusekiServerListener.serverInitialization(FusekiServerListener.java:96)
at org.apache.jena.fuseki.server.FusekiServerListener.contextInitialized(FusekiServerListener.java:53)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1514)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1476)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:449)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:416)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.jena.fuseki.jetty.JettyFuseki.start(JettyFuseki.java:137)
at org.apache.jena.fuseki.cmd.FusekiCmd.runFuseki(FusekiCmd.java:367)
at org.apache.jena.fuseki.cmd.FusekiCmd$FusekiCmdInner.exec(FusekiCmd.java:351)
at jena.cmd.CmdMain.mainMethod(CmdMain.java:93)
at jena.cmd.CmdMain.mainRun(CmdMain.java:58)
at jena.cmd.CmdMain.mainRun(CmdMain.java:45)
at org.apache.jena.fuseki.cmd.FusekiCmd$FusekiCmdInner.innerMain(FusekiCmd.java:103)
at org.apache.jena.fuseki.cmd.FusekiCmd.main(FusekiCmd.java:67)
[2018-05-24 17:05:04] Server INFO Started 2018/05/24 17:05:04 CEST on port 3030
I really don't understand where i'm wrong.
Here my config:
# Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0
## Fuseki Server configuration file.
#prefix : <http://base/#> .
#prefix fuseki: <http://jena.apache.org/fuseki#> .
#prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
#prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
#prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#> .
[] rdf:type fuseki:Server ;
# Example::
# Server-wide query timeout.
#
# Timeout - server-wide default: milliseconds.
# Format 1: "1000" -- 1 second timeout
# Format 2: "10000,60000" -- 10s timeout to first result,
# then 60s timeout for the rest of query.
#
# See javadoc for ARQ.queryTimeout for details.
# This can also be set on a per dataset basis in the dataset assembler.
#
# ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "30000" ] ;
# Add any custom classes you want to load.
# Must have a "public static void init()" method.
# ja:loadClass "your.code.Class" ;
# End triples.
.
<#service_spatial_tdb> rdf:type fuseki:Service ;
rdfs:label "TDB/spatial service" ;
fuseki:name "ds" ;
fuseki:serviceQuery "query" ;
fuseki:serviceQuery "sparql" ;
fuseki:serviceUpdate "update" ;
fuseki:serviceUpload "upload" ;
fuseki:serviceReadGraphStore "get" ;
fuseki:serviceReadWriteGraphStore "data" ;
fuseki:dataset :spatial_dataset ;
Can someone help me ?

Related

DataDog GKE NESTJS integration using DataDog's helm chart

Im trying to deploy my service and read my local logfile from the inside pod.
Using DataDog's helm chart values with the following configs :
## Default values for Datadog Agent
## See Datadog helm documentation to learn more:
## https://docs.datadoghq.com/agent/kubernetes/helm/
## #param image - object - required
## Define the Datadog image to work with.
#
image:
## #param repository - string - required
## Define the repository to use:
## use "datadog/agent" for Datadog Agent 6
## use "datadog/dogstatsd" for Standalone Datadog Agent DogStatsD6
#
repository: datadog/agent
## #param tag - string - required
## Define the Agent version to use.
## Use 6.13.0-jmx to enable jmx fetch collection
#
tag: 6.13.0
## #param pullPolicy - string - required
## The Kubernetes pull policy.
#
pullPolicy: IfNotPresent
## #param pullSecrets - list of key:value strings - optional
## It is possible to specify docker registry credentials
## See https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
#
# pullSecrets:
# - name: "<REG_SECRET>"
nameOverride: ""
fullnameOverride: ""
datadog:
## #param apiKey - string - required
## Set this to your Datadog API key before the Agent runs.
## ref: https://app.datadoghq.com/account/settings#agent/kubernetes
#
apiKey: "xxxxxxxx"
## #param apiKeyExistingSecret - string - optional
## Use existing Secret which stores API key instead of creating a new one.
## If set, this parameter takes precedence over "apiKey".
#
# apiKeyExistingSecret: <DATADOG_API_KEY_SECRET>
## #param appKey - string - optional
## If you are using clusterAgent.metricsProvider.enabled = true, you must set
## a Datadog application key for read access to your metrics.
#
appKey: "xxxxxx"
## #param appKeyExistingSecret - string - optional
## Use existing Secret which stores APP key instead of creating a new one
## If set, this parameter takes precedence over "appKey".
#
# appKeyExistingSecret: <DATADOG_APP_KEY_SECRET>
## #param securityContext - object - optional
## You can modify the security context used to run the containers by
## modifying the label type below:
#
# securityContext:
# seLinuxOptions:
# seLinuxLabel: "spc_t"
## #param clusterName - string - optional
## Set a unique cluster name to allow scoping hosts and Cluster Checks easily
#
# clusterName: <CLUSTER_NAME>
## #param name - string - required
## Daemonset/Deployment container name
## See clusterAgent.containerName if clusterAgent.enabled = true
#
name: datadog
## #param site - string - optional - default: 'datadoghq.com'
## The site of the Datadog intake to send Agent data to.
## Set to 'datadoghq.eu' to send data to the EU site.
#
# site: datadoghq.com
## #param dd_url - string - optional - default: 'https://app.datadoghq.com'
## The host of the Datadog intake server to send Agent data to, only set this option
## if you need the Agent to send data to a custom URL.
## Overrides the site setting defined in "site".
#
# dd_url: https://app.datadoghq.com
## #param logLevel - string - required
## Set logging verbosity, valid log levels are:
## trace, debug, info, warn, error, critical, and off
#
logLevel: INFO
## #param podLabelsAsTags - list of key:value strings - optional
## Provide a mapping of Kubernetes Labels to Datadog Tags.
#
# podLabelsAsTags:
# app: kube_app
# release: helm_release
# <KUBERNETES_LABEL>: <DATADOG_TAG_KEY>
## #param podAnnotationsAsTags - list of key:value strings - optional
## Provide a mapping of Kubernetes Annotations to Datadog Tags
#
# podAnnotationsAsTags:
# iam.amazonaws.com/role: kube_iamrole
# <KUBERNETES_ANNOTATIONS>: <DATADOG_TAG_KEY>
## #param tags - list of key:value elements - optional
## List of tags to attach to every metric, event and service check collected by this Agent.
##
## Learn more about tagging: https://docs.datadoghq.com/tagging/
#
# tags:
# - <KEY_1>:<VALUE_1>
# - <KEY_2>:<VALUE_2>
## #param useCriSocketVolume - boolean - required
## Enable container runtime socket volume mounting
#
useCriSocketVolume: true
## #param dogstatsdOriginDetection - boolean - optional
## Enable origin detection for container tagging
## https://docs.datadoghq.com/developers/dogstatsd/unix_socket/#using-origin-detection-for-container-tagging
#
# dogstatsdOriginDetection: true
## #param useDogStatsDSocketVolume - boolean - optional
## Enable dogstatsd over Unix Domain Socket
## ref: https://docs.datadoghq.com/developers/dogstatsd/unix_socket/
#
# useDogStatsDSocketVolume: true
## #param nonLocalTraffic - boolean - optional - default: false
## Enable this to make each node accept non-local statsd traffic.
## ref: https://github.com/DataDog/docker-dd-agent#environment-variables
#
nonLocalTraffic: true
## #param collectEvents - boolean - optional - default: false
## Enables this to start event collection from the kubernetes API
## ref: https://docs.datadoghq.com/agent/kubernetes/event_collection/
#
collectEvents: true
## #param leaderElection - boolean - optional - default: false
## Enables leader election mechanism for event collection.
#
# leaderElection: false
## #param leaderLeaseDuration - integer - optional - default: 60
## Set the lease time for leader election in second.
#
# leaderLeaseDuration: 60
## #param logsEnabled - boolean - optional - default: false
## Enables this to activate Datadog Agent log collection.
## ref: https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup
#
logsEnabled: true
## #param logsConfigContainerCollectAll - boolean - optional - default: false
## Enable this to allow log collection for all containers.
## ref: https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup
#
logsConfigContainerCollectAll: true
## #param containerLogsPath - string - optional - default: /var/lib/docker/containers
## This to allow log collection from container log path. Set to a different path if not
## using docker runtime.
## ref: https://docs.datadoghq.com/agent/kubernetes/daemonset_setup/?tab=k8sfile#create-manifest
#
containerLogsPath: /var/lib/docker/containers
## #param apmEnabled - boolean - optional - default: false
## Enable this to enable APM and tracing, on port 8126
## ref: https://github.com/DataDog/docker-dd-agent#tracing-from-the-host
#
apmEnabled: true
## #param processAgentEnabled - boolean - optional - default: false
## Enable this to activate live process monitoring.
## Note: /etc/passwd is automatically mounted to allow username resolution.
## ref: https://docs.datadoghq.com/graphing/infrastructure/process/#kubernetes-daemonset
#
processAgentEnabled: true
## #param env - list of object - optional
## The dd-agent supports many environment variables
## ref: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#environment-variables
#
# env:
# - name: <ENV_VAR_NAME>
# value: <ENV_VAR_VALUE>
## #param volumes - list of objects - optional
## Specify additional volumes to mount in the dd-agent container
#
# volumes:
# - hostPath:
# path: <HOST_PATH>
# name: <VOLUME_NAME>
## #param volumeMounts - list of objects - optional
## Specify additional volumes to mount in the dd-agent container
#
# volumeMounts:
# - name: <VOLUME_NAME>
# mountPath: <CONTAINER_PATH>
# readOnly: true
## #param confd - list of objects - optional
## Provide additional check configurations (static and Autodiscovery)
## Each key becomes a file in /conf.d
## ref: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#optional-volumes
## ref: https://docs.datadoghq.com/agent/autodiscovery/
#
confd:
conf.yaml: |-
init_config:
instances:
logs:
- type: "file"
path: "/app/logs/service.log"
service: nodejs
source: nodejs
sourcecategory: sourcecode
# kubernetes_state.yaml: |-
# ad_identifiers:
# - kube-state-metrics
# init_config:
# instances:
# - kube_state_url: http://%%host%%:8080/metrics
## #param checksd - list of key:value strings - optional
## Provide additional custom checks as python code
## Each key becomes a file in /checks.d
## ref: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#optional-volumes
#
# checksd:
# service.py: |-
## #param criSocketPath - string - optional
## Path to the container runtime socket (if different from Docker)
## This is supported starting from agent 6.6.0
#
# criSocketPath: /var/run/containerd/containerd.sock
## #param dogStatsDSocketPath - string - optional
## Path to the DogStatsD socket
#
# dogStatsDSocketPath: /var/run/datadog/dsd.socket
## #param livenessProbe - object - optional
## Override the agent's liveness probe logic from the default:
## In case of issues with the probe, you can disable it with the
## following values, to allow easier investigating:
#
# livenessProbe:
# exec:
# command: ["/bin/true"]
## #param resources - object -required
## datadog-agent resource requests and limits
## Make sure to keep requests and limits equal to keep the pods in the Guaranteed QoS class
## Ref: http://kubernetes.io/docs/user-guide/compute-resources/
#
resources: {}
# requests:
# cpu: 200m
# memory: 256Mi
# limits:
# cpu: 200m
# memory: 256Mi
## #param clusterAgent - object - required
## This is the Datadog Cluster Agent implementation that handles cluster-wide
## metrics more cleanly, separates concerns for better rbac, and implements
## the external metrics API so you can autoscale HPAs based on datadog metrics
## ref: https://docs.datadoghq.com/agent/kubernetes/cluster/
#
clusterAgent:
## #param enabled - boolean - required
## Set this to true to enable Datadog Cluster Agent
#
enabled: false
containerName: cluster-agent
image:
repository: datadog/cluster-agent
tag: 1.3.2
pullPolicy: IfNotPresent
## #param token - string - required
## This needs to be at least 32 characters a-zA-z
## It is a preshared key between the node agents and the cluster agent
## ref:
#
token: ""
replicas: 1
## #param metricsProvider - object - required
## Enable the metricsProvider to be able to scale based on metrics in Datadog
#
metricsProvider:
enabled: true
## #param clusterChecks - object - required
## Enable the Cluster Checks feature on both the cluster-agents and the daemonset
## ref: https://docs.datadoghq.com/agent/autodiscovery/clusterchecks/
## Autodiscovery via Kube Service annotations is automatically enabled
#
clusterChecks:
enabled: true
## #param confd - list of objects - optional
## Provide additional cluster check configurations
## Each key will become a file in /conf.d
## ref: https://docs.datadoghq.com/agent/autodiscovery/
#
# confd:
# mysql.yaml: |-
# cluster_check: true
# instances:
# - server: '<EXTERNAL_IP>'
# port: 3306
# user: datadog
# pass: '<YOUR_CHOSEN_PASSWORD>'
## #param resources - object -required
## Datadog cluster-agent resource requests and limits.
#
resources: {}
# requests:
# cpu: 200m
# memory: 256Mi
# limits:
# cpu: 200m
# memory: 256Mi
## #param priorityclassName - string - optional
## Name of the priorityClass to apply to the Cluster Agent
# priorityClassName: system-cluster-critical
## #param livenessProbe - object - optional
## Override the agent's liveness probe logic from the default:
## In case of issues with the probe, you can disable it with the
## following values, to allow easier investigating:
#
# livenessProbe:
# exec:
# command: ["/bin/true"]
## #param podAnnotations - list of key:value strings - optional
## Annotations to add to the cluster-agents's pod(s)
#
# podAnnotations:
# key: "value"
## #param readinessProbe - object - optional
## Override the cluster-agent's readiness probe logic from the default:
#
# readinessProbe:
rbac:
## #param created - boolean - required
## If true, create & use RBAC resources
#
create: true
## #param serviceAccountName - string - required
## Ignored if rbac.create is true
#
serviceAccountName: default
tolerations: []
kubeStateMetrics:
## #param enabled - boolean - required
## If true, deploys the kube-state-metrics deployment.
## ref: https://github.com/kubernetes/charts/tree/master/stable/kube-state-metrics
#
enabled: true
kube-state-metrics:
rbac:
## #param created - boolean - required
## If true, create & use RBAC resources
#
create: true
serviceAccount:
## #param created - boolean - required
## If true, create ServiceAccount, require rbac kube-state-metrics.rbac.create true
#
create: true
## #param name - string - required
## The name of the ServiceAccount to use.
## If not set and create is true, a name is generated using the fullname template
#
name: coupon-service-account
## #param resources - object - optional
## Resource requests and limits for the kube-state-metrics container.
#
# resources:
# requests:
# cpu: 200m
# memory: 256Mi
# limits:
# cpu: 200m
# memory: 256Mi
daemonset:
## #param enabled - boolean - required
## You should keep Datadog DaemonSet enabled!
## The exceptional case could be a situation when you need to run
## single DataDog pod per every namespace, but you do not need to
## re-create a DaemonSet for every non-default namespace install.
## Note: StatsD and DogStatsD work over UDP, so you may not
## get guaranteed delivery of the metrics in Datadog-per-namespace setup!
#
enabled: true
## #param useDedicatedContainers - boolean - optional
## Deploy each datadog agent process in a separate container. Allow fine-grained
## control over allocated resources and better isolation.
#
# useDedicatedContainers: false
containers:
agent:
## #param env - list - required
## Additionnal environment variables for the agent container.
#
# env:
## #param logLevel - string - optional
## Set logging verbosity, valid log levels are:
## trace, debug, info, warn, error, critical, and off.
## If not set, fall back to the value of datadog.logLevel.
#
logLevel: INFO
## #param resources - object - required
## Resource requests and limits for the agent container.
#
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 200m
memory: 256Mi
processAgent:
## #param env - list - required
## Additionnal environment variables for the process-agent container.
#
# env:
## #param logLevel - string - optional
## Set logging verbosity, valid log levels are:
## trace, debug, info, warn, error, critical, and off.
## If not set, fall back to the value of datadog.logLevel.
#
logLevel: INFO
## #param resources - object - required
## Resource requests and limits for the process-agent container.
#
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 100m
memory: 200Mi
traceAgent:
## #param env - list - required
## Additionnal environment variables for the trace-agent container.
#
# env:
## #param logLevel - string - optional
## Set logging verbosity, valid log levels are:
## trace, debug, info, warn, error, critical, and off.
## If not set, fall back to the value of datadog.logLevel.
#
logLevel: INFO
## #param resources - object - required
## Resource requests and limits for the trace-agent container.
#
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 100m
memory: 200Mi
## #param useHostNetwork - boolean - optional
## Bind ports on the hostNetwork. Useful for CNI networking where hostPort might
## not be supported. The ports need to be available on all hosts. It Can be
## used for custom metrics instead of a service endpoint.
##
## WARNING: Make sure that hosts using this are properly firewalled otherwise
## metrics and traces are accepted from any host able to connect to this host.
#
useHostNetwork: true
## #param useHostPort - boolean - optional
## Sets the hostPort to the same value of the container port. Needs to be used
## to receive traces in a standard APM set up. Can be used as for sending custom metrics.
## The ports need to be available on all hosts.
##
## WARNING: Make sure that hosts using this are properly firewalled otherwise
## metrics and traces are accepted from any host able to connect to this host.
#
useHostPort: true
## #param useHostPID - boolean - optional
## Run the agent in the host's PID namespace. This is required for Dogstatsd origin
## detection to work. See https://docs.datadoghq.com/developers/dogstatsd/unix_socket/
#
# useHostPID: true
## #param podAnnotations - list of key:value strings - optional
## Annotations to add to the DaemonSet's Pods
#
# podAnnotations:
# <POD_ANNOTATION>: '[{"key": "<KEY>", "value": "<VALUE>"}]'
## #param tolerations - array - optional
## Allow the DaemonSet to schedule on tainted nodes (requires Kubernetes >= 1.6)
#
# tolerations: []
## #param nodeSelector - object - optional
## Allow the DaemonSet to schedule on selected nodes
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
#
# nodeSelector: {}
## #param affinity - object - optional
## Allow the DaemonSet to schedule using affinity rules
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
#
# affinity: {}
## #param updateStrategy - string - optional
## Allow the DaemonSet to perform a rolling update on helm update
## ref: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
#
# updateStrategy: RollingUpdate
## #param priorityClassName - string - optional
## Sets PriorityClassName if defined.
#
# priorityClassName:
## #param podLabels - object - optional
## Sets podLabels if defined.
#
# podLabels: {}
## #param useConfigMap - boolean - optional
# Configures a configmap to provide the agent configuration
#
# useConfigMap: false
deployment:
## #param enabled - boolean - required
## Apart from DaemonSet, deploy Datadog agent pods and related service for
## applications that want to send custom metrics. Provides DogStasD service.
#
enabled: false
## #param replicas - integer - required
## If you want to use datadog.collectEvents, keep deployment.replicas set to 1.
#
replicas: 1
## #param affinity - object - required
## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
#
affinity: {}
## #param tolerations - array - required
## Tolerations for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
#
tolerations: []
## #param dogstatsdNodePort - integer - optional
## If you're using a NodePort-type service and need a fixed port, set this parameter.
#
# dogstatsdNodePort: 8125
## #param traceNodePort - integer - optional
## If you're using a NodePort-type service and need a fixed port, set this parameter.
#
# traceNodePort: 8126
## #param service - object - required
##
#
service:
type: ClusterIP
annotations: {}
## #param priorityClassName - string - optional
## Sets PriorityClassName if defined.
#
# priorityClassName:
clusterchecksDeployment:
## #param enabled - boolean - required
## If true, deploys agent dedicated for running the Cluster Checks instead of running in the Daemonset's agents.
## ref: https://docs.datadoghq.com/agent/autodiscovery/clusterchecks/
#
enabled: false
rbac:
## #param dedicated - boolean - required
## If true, use a dedicated RBAC resource for the cluster checks agent(s)
#
dedicated: false
## #param serviceAccountName - string - required
## Ignored if rbac.create is true
#
serviceAccountName: default
## #param replicas - integer - required
## If you want to deploy the cluckerchecks agent in HA, keep at least clusterchecksDeployment.replicas set to 2.
## And increase the clusterchecksDeployment.replicas according to the number of Cluster Checks.
#
replicas: 2
## #param resources - object -required
## Datadog clusterchecks-agent resource requests and limits.
#
resources: {}
# requests:
# cpu: 200m
# memory: 500Mi
# limits:
# cpu: 200m
# memory: 500Mi
## #param affinity - object - optional
## Allow the ClusterChecks Deployment to schedule using affinity rules.
## By default, ClusterChecks Deployment Pods are forced to run on different Nodes.
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
#
# affinity:
## #param nodeSelector - object - optional
## Allow the ClusterChecks Deploument to schedule on selected nodes
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
#
# nodeSelector: {}
## #param tolerations - array - required
## Tolerations for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
#
# tolerations: []
## #param livenessProbe - object - optional
## Override the agent's liveness probe logic from the default:
## In case of issues with the probe, you can disable it with the
## following values, to allow easier investigating:
#
# livenessProbe:
# exec:
# command: ["/bin/true"]
## #param env - list of object - optional
## The dd-agent supports many environment variables
## ref: https://github.com/DataDog/datadog-agent/tree/master/Dockerfiles/agent#environment-variables
#
# env:
# - name: <ENV_VAR_NAME>
# value: <ENV_VAR_VALUE>
as you can see I expect my logs to be available at /app/logs/service.log and thats what Im supplying to my conf.d :
confd:
conf.yaml: |-
init_config:
instances:
logs:
- type: "file"
path: "/app/logs/service.log"
service: nodejs
source: nodejs
sourcecategory: sourcecode
In my service, I use WinstonLogger using file transport with the JSON format.
transports: [
new transports.File({
winston.format.json(),
filename: `${process.env.LOGS_PATH}/service.log`,
}),
]
process.env.LOGS_PATH = '/app/logs'
After all, exploring my pod and tail -f my service.log in the expected /app/logs folder I see that the application actually writes the logs in a JSON format as expected.
My DataDog doesn't pick up the logs and they are not showing in the log section ..
NOTE:: I do not mount any volume to and from my service ..
What am I missing?
Should I mount my local log to /var/log/pods/[service_name]/ ?
The main issue is that you would have to mount the volume in both the app container and the agent container in order to make it available. It also means you have to find a place to store the log file before it gets picked up by the agent. Doing this for every container could become difficult to maintain and time consuming.
An alternative approach would be to instead send the logs to stdout and let the agent collect them with the Docker integration. Since you configured logsConfigContainerCollectAll to true, the agent is already configured to collect the logs from every container output, so configuring Winston to output to stdout should just work.
See: https://docs.datadoghq.com/agent/docker/log/
To support rochdev comment, here are a few code snippets to help out (if you do not opt in for the STDOUT method which should be simpler). This is only to mount the right volume inside the container agent.
On your app deployment, add:
spec:
containers:
- name: your_nodejs_app
...
volumeMounts:
- name: abc
mountPath: /app/logs
volumes:
- hostPath:
path: /app/logs
name: abc
And on your agent daemonset:
spec:
containers:
- image: datadog/agent
...
volumeMounts:
...
- name: plop
mountPath: /app/logs
volumes:
...
- hostPath:
path: /app/logs/
name: plop

How to Generate Grok Patterns automatically using LogMine

I am trying to generate GROK patterns automatically using LogMine
Log sample:
Error IGXL error [Slot 2, Chan 16, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow. Please check the timing programming. Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.
Error IGXL error [Slot 2, Chan 18, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow. Please check the timing programming. Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.
For the above logs, I am getting the following pattern:
re.compile('^(?P<Event>.*?)\\s+(?P<Tester>.*?)\\s+(?P<State>.*?)\\s+(?P<Slot>.*?)\\s+(?P<Instrument>.*?)\\s+(?P<Content1>.*?):\\s+(?P<Content>.*?)$')
But I expect a Grok Pattern(Logstash) that looks like this:
%{LOGLEVEL:level} *%{DATA:Instrument} %{LOGLEVEL:State} \[%{DATA:slot} %{DATA:slot} %{DATA:channel} %{DATA:channel} %{DATA:Site}] %{DATA:Tester} : %{DATA:Content}
Code: LogMine is imported from the following link: https://github.com/logpai/logparser/tree/master/logparser/LogMine
import sys
import os
sys.path.append('../')
import LogMine
input_dir ='E:\LogMine\LogMine' # The input directory of log file
output_dir ='E:\LogMine\LogMine/output/' # The output directory of parsing results
log_file ='E:\LogMine\LogMine/log_teradyne.txt' # The input log file name
log_format ='<Event> <Tester> <State> <Slot> <Instrument><content> <contents> <context> <desc> <junk> ' # HDFS log format
levels =1 # The levels of hierarchy of patterns
max_dist =0.001 # The maximum distance between any log message in a cluster and the cluster representative
k =1 # The message distance weight (default: 1)
regex =[] # Regular expression list for optional preprocessing (default: [])
print(os.getcwd())
parser = LogMine.LogParser(input_dir, output_dir, log_format, rex=regex, levels=levels, max_dist=max_dist, k=k)
parser.parse(log_file)
This code returns only the parsed CSV file, I am looking to generate the GROK Patterns and use it later in a Logstash application to parse the logs.

Issue with xsd:decimal when estimating the radius

I want to extract items that are located within 20 km of the given latitude and longitude. Below I provide my query. The only issue that it has is related to ?lat and ?long. They have the format like this "41.3823035"^^xsd:decimal. In the ontology file, latitude and longitude have the format xsd:float.
When I run the query, I get the error:
Error 500: Infinite or NaN
Fuseki - version 2.4.1 (Build date: 2016-11-04T18:59:20+0000)
If I substitute ?lat by 41.38, and ?long by 2.22, then it works.
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX owl: <http://www.test.com/my-ontology.owl#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX oo: <http://www.w3.org/2002/07/owl#>
PREFIX oa: <http://www.w3.org/ns/oa#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX lfn: <http://www.dotnetrdf.org/leviathan#>
PREFIX fn: <http://www.w3.org/2005/xpath-functions#>
PREFIX afn: <http://jena.hpl.hp.com/ARQ/function#>
SELECT DISTINCT ?id ?kwds ?lat WHERE {
?p owl:id 123 .
?p rdf:type ?service .
?service rdfs:subClassOf ?family .
?otherService rdfs:subClassOf ?family .
?item rdf:type ?otherService .
?item owl:id ?id .
?item owl:latitude ?lat .
?item owl:longitude ?long .
BIND ((lfn:degrees-to-radians(?lat)) AS ?lat1) .
BIND ((lfn:degrees-to-radians(?long)) AS ?lon1) .
BIND ((lfn:degrees-to-radians(41.384697)) AS ?lat2) .
BIND ((lfn:degrees-to-radians(2.150849)) AS ?lon2) .
BIND ((?lon2-?lon1) AS ?dlon) .
BIND ((?lat2-?lat1) AS ?dlat) .
BIND ((lfn:sq(lfn:sin(?dlat/2))+lfn:cos(?lat1)*lfn:cos(?lat2)*lfn:sin(?dlon/2) ) AS ?a) .
BIND ((2*lfn:sin-1(lfn:sqrt(?a))) AS ?c) .
BIND ((xsd:float(fn:round((6371*?c)))) AS ?dist) .
FILTER ( ?dist < 25 ) .
}
UPDATE
The problem is caused by this line BIND ((2*lfn:sin-1(lfn:sqrt(?a))) AS ?c) .. For some values of lat and lon (0.0) it returns NaN.
I solved this issue by adding IF to the BIND statement:
BIND (IF(?c != "NaN"^^xsd:double,xsd:float(fn:round((6371*?c))),999999) AS ?dist)

AC_ARG_ENABLE in an m4_foreach_w loop: no help string

I wish to generate a lot of --enable-*/--disable-* options by something like:
COMPONENTS([a b c], [yes])
where the second argument is the default value of the automatic enable_* variable. My first attempt was to write an AC_ARG_ENABLE(...) within an m4_foreach_w, but so far, I'm only getting the first component to appear in the ./configure --help output.
If I add hand-written AC_ARG_ENABLEs, they work as usual.
Regardless, the --enable-*/--disable-* options work as they should, just the help text is missing.
Here's the full code to reproduce the problem:
AC_INIT([foo], 1.0)
AM_INIT_AUTOMAKE([foreign])
AC_DEFUN([COMPONENTS],
[
m4_foreach_w([component], [$1], [
AS_ECHO(["Processing [component] component with default enable=$2"])
AC_ARG_ENABLE([component],
[AS_HELP_STRING([--enable-[]component], [component] component)],
,
[enable_[]AS_TR_SH([component])=$2]
)
])
AC_ARG_ENABLE([x],
[AS_HELP_STRING([--enable-[]x], [component x])],
,
[enable_[]AS_TR_SH([x])=$2]
)
AC_ARG_ENABLE([y],
[AS_HELP_STRING([--enable-[]y], [component y])],
,
[enable_[]AS_TR_SH([y])=$2]
)
])
COMPONENTS([a b c], [yes])
for var in a b c x y; do
echo -n "\$enable_$var="
eval echo "\$enable_$var"
done
AC_CONFIG_FILES(Makefile)
AC_OUTPUT
And an empty Makefile.am. To verify that the options work:
$ ./configure --disable-a --disable-b --disable-d --disable-x
configure: WARNING: unrecognized options: --disable-d
...
Processing component a with default enable=yes
Processing component b with default enable=yes
Processing component c with default enable=yes
$enable_a=no
$enable_b=no
$enable_c=yes
$enable_x=no
$enable_y=yes
After I poked around in autoconf sources, I figured out this has to do with the m4_divert_once call in the implementation of AC_ARG_ENABLE:
# AC_ARG_ENABLE(FEATURE, HELP-STRING, [ACTION-IF-TRUE], [ACTION-IF-FALSE])
# ------------------------------------------------------------------------
AC_DEFUN([AC_ARG_ENABLE],
[AC_PROVIDE_IFELSE([AC_PRESERVE_HELP_ORDER],
[],
[m4_divert_once([HELP_ENABLE], [[
Optional Features:
--disable-option-checking ignore unrecognized --enable/--with options
--disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no)
--enable-FEATURE[=ARG] include FEATURE [ARG=yes]]])])dnl
m4_divert_once([HELP_ENABLE], [$2])dnl
_AC_ENABLE_IF([enable], [$1], [$3], [$4])dnl
])# AC_ARG_ENABLE
# m4_divert_once(DIVERSION-NAME, CONTENT)
# ---------------------------------------
# Output CONTENT into DIVERSION-NAME once, if not already there.
# An end of line is appended for free to CONTENT.
m4_define([m4_divert_once],
[m4_expand_once([m4_divert_text([$1], [$2])])])
I'm guessing that the HELP-STRING argument is remembered in it's unexpanded form, so it is added just once for all components. Manually expanding the AC_HELP_STRING does what I want:
AC_DEFUN([COMPONENTS],
[
m4_foreach_w([comp], [$1], [
AS_ECHO(["Processing component 'comp' with default enable=$2"])
AC_ARG_ENABLE([comp],
m4_expand([AS_HELP_STRING([--enable-comp], enable component comp)]),
,
[enable_[]AS_TR_SH([comp])=$2]
)
])
])
COMPONENTS([a b c x y], [yes])
I couldn't find a way to properly quote components so that it appears as a string, after being used as the loop variable in m4_foreach_w, so I just renamed it to spare me the trouble.

Splunk splitting xml log event

We have logs that log an event to a single file. Each log entry looks something like this:
<LogEntry>
<UserName>IIS APPPOOL\ASP.NET v4.0</UserName>
<TimeStamp>02/28/2014 13:54:17</TimeStamp>
<ThreadName>20</ThreadName>
<CorrelationId>7a0d464d-556c-4d47-820f-0cf01322e54c</CorrelationId>
<LoggerName>-Api-booking</LoggerName>
<Level>INFO</Level>
<Identity></Identity>
<Domain>API-1-130380690118132000</Domain>
<CreatedOn>02/28/2014 13:54:22</CreatedOn>
<ExceptionObject />
<RenderedMessage>"7a0d464d-556c-4d47-820f-0cf01322e54c" - "GET https://myapi.com/booking" - API-"Response":
"Unauthorized"</RenderedMessage>
</LogEntry>
When we import these logs into Splunk, the log entry is split up incorrectly into 3 parts e.g.
1-
<LogEntry>
<UserName>IIS APPPOOL\ASP.NET v4.0</UserName>
2-
<CreatedOn>02/28/2014 02:57:55</CreatedOn>
<ExceptionObject />
<RenderedMessage>"66d8cdda-ff62-480a-b7d2-ec175b151e5f" - "POST https://myapi.com/booking" - API-"Response":
"Bad Request"</RenderedMessage>
</LogEntry>
3-
<TimeStamp>02/28/2014 02:57:29</TimeStamp>
<ThreadName>21</ThreadName>
<CorrelationId>66d8cdda-ff62-480a-b7d2-ec175b151e5f</CorrelationId>
<LoggerName>-Api-booking</LoggerName>
<Level>INFO</Level>
<Identity></Identity>
<Domain>/LM/W3SVC/1/ROOT/Api-1-130380256918440000</Domain>
How can I configure Splunk to see these as a single log event?
props.conf (pay attention to LINE_BREAKER)
[your_xml_sourcetype]
TIME_PREFIX = <TimeStamp>
MAX_TIMESTAMP_LOOKAHEAD = 19
TZ = GMT
# A performance tweak is to disable SHOULD_LINEMERGE and then set the
# LINE_BREAKER to "line ending characters coming before a new time stamp"
# (note the direct link of the TIME_FORMAT to the regex of LINE_BREAKER).
TIME_FORMAT = %m/%d/%Y %T
LINE_BREAKER = ([\r\n]+)<LogEntry>
SHOULD_LINEMERGE = False
# 10000 is default, should be set on a case by case basis
TRUNCATE = 5000
# If the data does not have nice key=value pairs, (or some other readily
# machine parseable format, like JSON or XML), set KV_MODE = none so that
# Splunk doesn't spin its wheels on attempting to look for key = value
# pairs which don't exist.
KV_MODE = xml
# Leaving PUNCT enabled can impact indexing performance. Customers can
# comment this line if they need to use PUNCT
ANNOTATE_PUNCT = false
More information here: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Resources