i need to get kube state metrics with Mi, it default comes with Ki. can any one please help me
[root#dte-dev-1-bizsvck8s-mst harsha]# curl http://<server IP>:8088/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/hello-kubernetes-65bc74d4b9-qp9dc
{
"kind": "PodMetrics",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"name": "hello-kubernetes-65bc74d4b9-qp9dc",
"namespace": "default",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/hello-kubernetes-65bc74d4b9-qp9dc",
"creationTimestamp": "2020-04-17T12:31:59Z"
},
"timestamp": "2020-04-17T12:31:26Z",
"window": "30s",
"containers": [
{
"name": "hello-kubernetes",
"usage": {
"cpu": "0",
"memory": "20552Ki"
}
}
]
i want to get memory usage from Mi (megabytes) not the Ki. Please help me!
This unit is hardcoded in official kube-state-metrics code which shouldn't be changed. For example node metrics - especially memory usage is in Megabytes unit not Kilobytes.
To get memory usage of specific pod in Megabytes units simply execute:
kubectl top pod --namespace example-app
NAME CPU(cores) MEMORY(bytes)
app-deployment-76bf4969df-65wmd 12m 1Mi
app-deployment-76bf4969df-mmqvt 16m 1Mi
The kubectl top command returns current CPU and memory usage for a cluster’s pods or nodes, or for a particular pod or node if specified.
You can also convert received value:
1 KB = 0.001 MB (in decimal),
1 KB = 0.0009765625 MB (in binary)
Take a look: kube-state-metrics-monitoring.
Related
How could I change the docker-volume-device option after the creation, With out removing the container
Previous command used for creation
docker volume create \
--driver local \
--opt type=none \
--opt device="/home/userA/Containers/vol1/home" \
--opt o=bind \
$DOCKER_VOLUME_NAME
Current Status
$ docker volume inspect vol1
[
{
"CreatedAt": "XXX",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/vol1/_data",
"Name": "vol1-home",
"Options": {
"device": "/home/userA/Containers/vol1/home",
"o": "bind",
"type": "none"
},
"Scope": "local"
}
]
Is there any way we can safely alter the value of device "/home/userA/Containers/vol1/home" to "/Containers/vol1/home". I am not want to keep this in specific user.
I would like to change
" --opt device="/home/userA/Containers/vol1/home" "
I would like to see
$ docker volume inspect vol1
[
{
"CreatedAt": "XXX",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/vol1/_data",
"Name": "vol1-home",
"Options": {
"device": "/Containers/vol1/home",
"o": "bind",
"type": "none"
},
"Scope": "local"
}
]
Does any one have any suggestion for my problem?
Reason / Benefit of this:
Due to a lack of space in my disk, I moved all files from /var/lib/docker to some other disk and I mounted them as /var/lib/docker. I saved my OS space and separated docker from the os disk.
Now I am trying to separate the volume device also. so I can save more space, Unfortunately, I might lose my container data for doing so.
I know I can increase memory allocation for my flows using code. But can I do the same thing on Prefect Cloud somewhere?
To some extent, you can - it depends on what you are asking. What is currently a bottleneck for you?
E.g. you can allocate more memory to your VM, Kubernetes job or a serverless container used as infrastructure blocks. Here is an example for KubernetesJob:
from prefect.infrastructure import KubernetesJob
k8s_job = KubernetesJob(
namespace="prefect",
customizations=[
{
"op": "add",
"path": "/spec/template/spec/resources",
"value": {"limits": {"memory": "8Gi", "cpu": "4000m"}},
},
{
"op": "add",
"path": "/spec/template/spec/resources",
"value": {"requests": {"memory": "2Gi", "cpu": "1000m"}},
}
],
)
k8s_job.save("prod")
and here is one for ECSTask:
from prefect_aws.ecs import ECSTask
from prefect_aws.credentials import AwsCredentials
aws_credentials_block = AwsCredentials(
aws_access_key_id="xxx",
aws_secret_access_key="yyy",
)
ecs = ECSTask(
aws_credentials=aws_credentials_block,
cpu="256",
memory="512",
)
ecs.save("prod")
I need to get memory used by azure VM, but i am not getting .
Tried this https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXX/providers/Microsoft.Compute/virtualMachines/XXXXXXX/providers/microsoft.insights/metrics?timespan=2019-03-31T11:30:00.000Z/2020-09-14T11:00:00.000Z&interval=P1D&metricnames=\Memory\% Committed Bytes In Use&aggregation=Average&api-version=2018-01-01&metricnamespace=azure.vm.windows.guestmetrics
Response I am getting
{
"cost": 0,
"timespan": "2020-08-14T11:00:00Z/2020-09-14T11:00:00Z",
"interval": "P1D",
"value": [
{
"id": "/subscriptions/xxxxxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxxxxxx/providers/Microsoft.Compute/virtualMachines/xxxxxxx/providers/Microsoft.Insights/metrics/\Memory\% Committed Bytes In Use",
"type": "Microsoft.Insights/metrics",
"name": {
"value": "\Memory\% Committed Bytes In Use",
"localizedValue": "\Memory\% Committed Bytes In Use"
},
"unit": "Unspecified",
"timeseries": [],
"errorCode": "Success"
}
],
"namespace": "azure.vm.windows.guestmetrics",
"resourceregion": "westus2"
}
Make sure you have enabled the guest-level monitoring for Azure virtual machines then try again.
See - https://docs.bmc.com/docs/capacityoptimization/btco115/collecting-additional-metrics-using-guest-os-diagnostics-890312716.html
We have an index with a join datatype and the indexing speed is very slow.
At best we are indexing 100/sec, but mostly around 50/sec, the times is varying depending of the document size. We are using multiple threads with .NET Nest when indexing but both batching and single inserts are pretty slow. We have tested various batch sizes but still not getting any speed to talk about. Even with only small documents containing "metadata" it is slow, but speed will drop radically when the document size is increasing. Document size in this solution can vary from small up to 6 MB
What can we expect using the join datatype and indexing? How much penalty must we expect to get using it? We did of course try to avoid this when designing it, but we did not find any way around it. Any tips or tricks?
We are using a 3-node cluster in Azure, all with 32 GB of RAM and premium SSD disks. The Java Heap size is set to 16GB. Swapping is Disabled. Memory usage on the VM’s is stable about 60% of total, but the CPU is very low < 10 %. We are running Elasticsearch v. 6.2.3.
A short version of the mapping:
"mappings": {
"log": {
"_routing": {
"required": true
},
"properties": {
"body": {
"type": "text"
},
"description": {
"type": "text"
},
"headStepJoinField": {
"type": "join",
"eager_global_ordinals": true,
"relations": {
"head": "step"
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"statusId": {
"type": "keyword"
},
"stepId": {
"type": "keyword"
}
}
}
}
Suddenly, after working for one month with almost no use of cpu (between 1 to 5%). The neo4j server is stuck 100% cpu on garbage collecting.
I have neo4j-entherprise 2.0.3 (not embedded) running on ubuntu 4 processors server .
this is my neo4j configuration:
wrapper:
wrapper.java.additional=-Dorg.neo4j.server.properties=conf/neo4j-server.properties
wrapper.java.additional=-Djava.util.logging.config.file=conf/logging.properties
wrapper.java.additional=-Dlog4j.configuration=file:conf/log4j.properties
#********************************************************************
# JVM Parameters
#********************************************************************
wrapper.java.additional=-XX:+UseConcMarkSweepGC
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled
# Remote JMX monitoring, uncomment and adjust the following lines as needed.
# Also make sure to update the jmx.access and jmx.password files with appropriate permission roles and passwords,
# the shipped configuration contains only a read only role called 'monitor' with password 'Neo4j'.
# For more details, see: http://download.oracle.com/javase/6/docs/technotes/guides/management/agent.html
# On Unix based systems the jmx.password file needs to be owned by the user that will run the server,
# and have permissions set to 0600.
# For details on setting these file permissions on Windows see:
# http://download.oracle.com/javase/1.5.0/docs/guide/management/security-windows.html
wrapper.java.additional=-Dcom.sun.management.jmxremote.port=3637
wrapper.java.additional=-Dcom.sun.management.jmxremote.authenticate=true
wrapper.java.additional=-Dcom.sun.management.jmxremote.ssl=false
wrapper.java.additional=-Dcom.sun.management.jmxremote.password.file=conf/jmx.password
wrapper.java.additional=-Dcom.sun.management.jmxremote.access.file=conf/jmx.access
# Some systems cannot discover host name automatically, and need this line configured:
#wrapper.java.additional=-Djava.rmi.server.hostname=$THE_NEO4J_SERVER_HOSTNAME
# disable UDC (report data to neo4j..)
wrapper.java.additional=-Dneo4j.ext.udc.disable=true
# Uncomment the following lines to enable garbage collection logging
wrapper.java.additional=-Xloggc:data/log/neo4j-gc.log
wrapper.java.additional=-XX:+PrintGCDetails
wrapper.java.additional=-XX:+PrintGCDateStamps
wrapper.java.additional=-XX:+PrintGCApplicationStoppedTime
#wrapper.java.additional=-XX:+PrintPromotionFailure
#wrapper.java.additional=-XX:+PrintTenuringDistribution
# Uncomment the following lines to enable JVM startup diagnostics
#wrapper.java.additional=-XX:+PrintFlagsFinal
#wrapper.java.additional=-XX:+PrintFlagsInitial
# Java Heap Size: by default the Java heap size is dynamically
# calculated based on available system resources.
# Uncomment these lines to set specific initial and maximum
# heap size in MB.
#wrapper.java.initmemory=512
wrapper.java.maxmemory=3072
#********************************************************************
# Wrapper settings
#********************************************************************
# path is relative to the bin dir
wrapper.pidfile=../data/neo4j-server.pid
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.name=neo4j
defaults values:
# Default values for the low-level graph engine
neostore.nodestore.db.mapped_memory=25M
neostore.relationshipstore.db.mapped_memory=120M
neostore.propertystore.db.mapped_memory=90M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=100M
What can I do?
EDIT:
The store file sizes:
[
{
"description": "Information about the sizes of the different parts of the Neo4j graph store",
"name": "org.neo4j:instance=kernel#0,name=Store file sizes",
"attributes": [
{
"description": "The total disk space used by this Neo4j instance, in bytes.",
"name": "TotalStoreSize",
"value": 401188207,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used by the current Neo4j logical log, in bytes.",
"name": "LogicalLogSize",
"value": 24957516,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used to store array properties, in bytes.",
"name": "ArrayStoreSize",
"value": 128,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used to store nodes, in bytes.",
"name": "NodeStoreSize",
"value": 524160,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used to store properties (excluding string values and array values), in bytes.",
"name": "PropertyStoreSize",
"value": 145348280,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used to store relationships, in bytes.",
"name": "RelationshipStoreSize",
"value": 114126903,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
},
{
"description": "The amount of disk space used to store string properties, in bytes.",
"name": "StringStoreSize",
"value": 128,
"isReadable": "true",
"type": "long",
"isWriteable": "false ",
"isIs": "false "
}
],
"url": "org.neo4j/instance%3Dkernel%230%2Cname%3DStore+file+sizes"
}
]
Assuming you have 16 GB of RAM in the machine.
First thing is to set the neostore.xxx.mapped_memory settings to match the size of your store files. I'm assuming their total is 5 GB -> you have 11 GB left. See http://docs.neo4j.org/chunked/2.0.4/configuration-caches.html for more details.
Reserve some RAM for the system: 1GB -> you have 10 GB left.
Assign the remaining RAM to java heap using wrapper.java.initmemory wrapper.java.maxmemory. Set both to the same value.
If hpc is used as cache_type consider tweaking its settings based on cache hit ratio for relationships and nodes. Use JMX to monitor them, http://docs.neo4j.org/chunked/2.0.4/jmx-mxbeans.html#jmx-cache-nodecache.
We also experienced these kind of issues. In addition to configuration changes similar to what #stefan-armbruster mentioned updating Neo4j to 2.1.2 we also configured Neo4j to use G1 garbage collection instead of CMS.
Since making the garbage collection change we have seen far fewer spikes than we did previously.
If you want to give it a shot you can enable G1 GC by adding the following to your conf/neo4j-wrapper.conf file.
wrapper.java.additional=-XX:+UseG1GC
Hopefully with a combination of this and the changes suggested by #stefan-armbruster you'll resolve the issue.