Arangodb freeze when page fault increased - arangodb

I using arango with nodejs and arangojs driver, one of the arango collection has 10,000,000 documents
Sometimes page fault going up (150 or 500) and arango freezed and don't response to query request Also freezed arango web panel.
My server config is:
RAM: 6 GB
CPU: 8 core
(From web panel arango using 4.76 GB (83.90 %) 6 GB of ram)
UPDATE1
This is output of /_api/collection/AdsStatics/figures
{
"id": "191689719157",
"name": "AdsStatics",
"isSystem": false,
"doCompact": true,
"isVolatile": false,
"journalSize": 33554432,
"keyOptions": {
"type": "traditional",
"allowUserKeys": true
},
"waitForSync": false,
"indexBuckets": 8,
"count": 7816780,
"figures": {
"alive": {
"count": 7815806,
"size": 3563838968
},
"dead": {
"count": 306,
"size": 167464,
"deletion": 0
},
"datafiles": {
"count": 104,
"fileSize": 3530743672
},
"journals": {
"count": 1,
"fileSize": 33554432
},
"compactors": {
"count": 0,
"fileSize": 0
},
"shapefiles": {
"count": 0,
"fileSize": 0
},
"shapes": {
"count": 121,
"size": 56520
},
"attributes": {
"count": 24,
"size": 56
},
"indexes": {
"count": 3,
"size": 1660594864
},
"lastTick": "10044860034955",
"uncollectedLogfileEntries": 985,
"documentReferences": 0,
"waitingFor": "-",
"compactionStatus": {
"message": "checked datafiles, but no compaction opportunity found",
"time": "2016-02-24T08:29:27Z"
}
},
"status": 3,
"type": 2,
"error": false,
"code": 200
}
Thanks

It seems that your system is running out of memory. The datafiles for the one collection are 3,530,743,672 bytes in size, the indexes are 1,660,594,864. That is about 5.1 GB for this one collection alone.
arangod will need further memory for its WAL, the V8 contexts and temporary query results in order to operate properly.
Provided the system has 6 GB of total RAM and the OS and other processes need some RAM, too, it looks like you're running out of memory.
I am wondering if you're seeing some sort of swapping activity, which would explain why (all) operations would get extremely slow.

Related

MongoDB slow performance when fetching large array of documents

So I've been at this for a week now and I can't solve it. Queries for large arrays of documents are way too slow.
I'm running a super basic find query followed by a .toArray() and then sending it to our frontend.
const orders = await db.collection('Orders').find({ organization: new ObjectId('5fa28c7ad882490116f8761e') }).toArray()
The query itself takes 25 milliseconds seen below in the executionStats but before the data is retrieved it takes 4.5 seconds.
A couple of hundred documents work as expected, it takes a couple of hundred milliseconds tops. But over 500 documents it quickly slows down and now I'm at a 4.5 seconds fetch for ~25k documents. And this is not even the final query. I'm going to a run an aggregation with a few steps on top of this, but this seem to be the main issue, and it just can't be hardware related. I've tried to bump upp the database to an M60 instead of an M30 mentioned below, almost no difference, and it's way too expensive for our company size.
What can I do? Is MongoDB just unable to return anything more than a few hundred documents?
Here's some stats:
Database: Hosted at MongoDB Atlas with the server in the same country, Cluster size M30 (8 GB RAM, 100 GB storage, 2 vCPUs, 288 IOPS). Up to 10 GBit/s network.
My machine: 64 GB 3600 MHz RAM, 5950x CPU, 970 EVO Plus M.2 SSD. 150 MBit/s network.
Collection stats:
{
"ns" : "[db].Orders",
"size" : 207216190,
"count" : 467771,
"avgObjSize" : 442,
"storageSize" : 59006976,
"freeStorageSize" : 12288,
"capped" : false,
"wiredTiger" : {
"metadata" : {
"formatVersion" : 1
}
}
}
Execution stats:
{
"executionSuccess": true,
"nReturned": 26385,
"executionTimeMillis": 25,
"totalKeysExamined": 26385,
"totalDocsExamined": 26385,
"executionStages": {
"stage": "FETCH",
"nReturned": 26385,
"executionTimeMillisEstimate": 3,
"works": 26386,
"advanced": 26385,
"needTime": 0,
"needYield": 0,
"saveState": 26,
"restoreState": 26,
"isEOF": 1,
"docsExamined": 26385,
"alreadyHasObj": 0,
"inputStage": {
"stage": "IXSCAN",
"nReturned": 26385,
"executionTimeMillisEstimate": 2,
"works": 26386,
"advanced": 26385,
"needTime": 0,
"needYield": 0,
"saveState": 26,
"restoreState": 26,
"isEOF": 1,
"keyPattern": {
"organization": 1,
"state.removed": 1
},
"indexName": "organization_1_state.removed_1",
"isMultiKey": false,
"multiKeyPaths": {
"organization": [],
"state.removed": []
},
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"organization": [
"[ObjectId('5fa28c7ad882490116f8761e'), ObjectId('5fa28c7ad882490116f8761e')]"
],
"state.removed": [
"[MinKey, MaxKey]"
]
},
"keysExamined": 26385,
"seeks": 1,
"dupsTested": 0,
"dupsDropped": 0
}
}
}

Azure Gremlin edge traversal suspiciously high (Out() step) RU cost

I have a weird issue, where doing an out-operation on a few edges causes my RU cost to triple. Hope someone can help me shed light on why + what I can do to mitigate it.
I have a Graph in CosmosDB, where there are two types of vertex labels: "Profile" and "Score". Each profile has 0 or 1 score-vertices via a "ProfileHasAggregatedScore" edge. The partitionKey is the ID of the Profile.
If I make the following queries, the RU currently is:
g.V().hasLabel('Profile').out('ProfileHasAggregatedScore')
>78 RU (8 scores found)
And for reference, the cost of getting all vertices of a type is:
g.V().hasLabel('Profile')
>28 RU (110 profiles found)
g.E().hasLabel('ProfileHasAggregatedScore')
>11 RU (8 edges found)
g.V().hasLabel('AggregatedRating')
>11 RU (8 scores found)
And the cost of a single of the vertices or edges are:
g.V('aProfileId').hasLabel('Profile')
>4 RU (1 found)
g.E('anEdgeId')
> 7RU
G.V('aRatingId')
> 3.5 RU
Can someone please help me as to why, making a traversal with only a few vertices along the way (see traversal at the bottom), is more expensive than searching for everything? And is there something I can do to prevent it? Adding a has-filter with the partitionKey does not seem to help. It seems odd that traversing/finding 16 elements more (8 edges and 8 vertices) after finding 110 vertices triples the cost of the operation?
(NB. With 1000 profiles the cost of doing 1 traversal along an edge to the score node is 2200 RU. This seems high, considering the emphasis their Azure team put on it being scalable?)
Traversal if it can help (It seems most of the time is spent finding the edges with the out() step):
[
{
"gremlin": "g.V().hasLabel('Profile').out('ProfileHasAggregatedScore').executionProfile()",
"totalTime": 46,
"metrics": [
{
"name": "GetVertices",
"time": 13,
"annotations": {
"percentTime": 28.26
},
"counts": {
"resultCount": 110
},
"storeOps": [
{
"fanoutFactor": 1,
"count": 110,
"size": 124649,
"time": 2.47
}
]
},
{
"name": "GetEdges",
"time": 26,
"annotations": {
"percentTime": 56.52
},
"counts": {
"resultCount": 8
},
"storeOps": [
{
"fanoutFactor": 1,
"count": 8,
"size": 5200,
"time": 6.22
},
{
"fanoutFactor": 1,
"count": 0,
"size": 49,
"time": 0.88
}
]
},
{
"name": "GetNeighborVertices",
"time": 7,
"annotations": {
"percentTime": 15.22
},
"counts": {
"resultCount": 8
},
"storeOps": [
{
"fanoutFactor": 1,
"count": 8,
"size": 6303,
"time": 1.18
}
]
},
{
"name": "ProjectOperator",
"time": 0,
"annotations": {
"percentTime": 0
},
"counts": {
"resultCount": 8
}
}
]
}
]
enter code here

Can I use Spark's REST API to get the version of Spark on the Workers

I know I can get the version of Spark v2.2.1 that's running on Spark Master with this command:
http://<spark-master>:4040/api/v1/version
which will return something like
{
"spark" : "2.2.1"
}
However, I also want to check the version of Spark running on each Worker. I know I can get a list of all Workers thusly:
http://<spark-master>:8080/json/
which will return a response similar to
{
"url": "spark://<spark-master>:7077",
"workers": [{
"id": "worker-20180228071440-<ip-address>-7078",
"host": "<ip-address>",
"port": 7078,
"webuiaddress": "http://<ip-address>:8081",
"cores": 8,
"coresused": 8,
"coresfree": 0,
"memory": 40960,
"memoryused": 35875,
"memoryfree": 5085,
"state": "ALIVE",
"lastheartbeat": 1519932580686
}, ...
],
"cores": 32,
"coresused": 32,
"memory": 163840,
"memoryused": 143500,
"activeapps": [{
"starttime": 1519830260440,
"id": "app-20180228070420-0000",
"name": "<spark-app-name>",
"user": "<spark-app-user>",
"memoryperslave": 35875,
"submitdate": "Wed Feb 28 07:04:20 PST 2018",
"state": "RUNNING",
"duration": 102328434
}
],
"completedapps": [],
"activedrivers": [],
"status": "ALIVE"
}
I'd like to use that information to query each Spark Worker's version. Is this possible?

Microsoft.Azure.Search (sdk v3.0.3) don't return all the facets correctly

When I use Microsoft.Azure.Search (v3.0.3)'s "SearchAsync" and "Search" methods to return the indexed items, the sdk doesn't return all the facets.
However; when I try the same thing using Postman, it returns all the facets correctly.
Could this be a bug of the sdk (I believe it is as a direct call to an sdk method doesn't return the all the facets correctly - but couldn't find any records about this possible bug)? If yes, is there a fix for this for the sdk? Any help is appreciated.
UPDATE:
After spending some more time, I have found out that the bug is not .NET SDK Specific.
Both .NET SDK and REST API appear to have this problem and none of them returns all the facets. Can you please tell me is there a known bug for this and what is the fix for it?
Please see the following example;
There must be 2 Coaching facets but only 1 is returning from the Azure Search Service.
The new search query(facet specialisms added)
https://MYPROJECT-search.search.windows.net/indexes/myproject-directory-qa/docs?api-version=2016-09-01&$count=true&facet=specialisms&$filter=listingType eq 'Therapist'
"Coaching:Development coaching", --> This doesn't return as a facet.
"Coaching:Executive coaching", -->This returns fine.
"#search.facets": {
"specialisms#odata.type": "#Collection(Microsoft.Azure.Search.V2016_09_01.QueryResultFacet)",
"specialisms": [
{
"count": 5,
"value": "Anxiety, depression and trauma:Depression"
},
{
"count": 4,
"value": "Addiction, self-harm and eating disorders:Obsessions"
},
{
"count": 4,
"value": "Anxiety, depression and trauma:Post-traumatic stress"
},
{
"count": 4,
"value": "Coaching:Executive coaching"
},
{
"count": 4,
"value": "Identity, culture and spirituality:Self esteem"
},
{
"count": 4,
"value": "Relationships, family and children:Pregnancy related issues"
},
{
"count": 4,
"value": "Stress and work:Redundancy"
},
{
"count": 3,
"value": "Addiction, self-harm and eating disorders:Eating disorders"
},
{
"count": 3,
"value": "Anxiety, depression and trauma:Bereavement"
},
{
"count": 3,
"value": "Anxiety, depression and trauma:Loss"
}
]
},
{
"#search.score": 1,
"contactId": "df394997-6e94-e711-80ed-3863bb34db00",
"location": {
"type": "Point",
"coordinates": [
-2.58586,
51.47873
],
"crs": {
"type": "name",
"properties": {
"name": "EPSG:4326"
}
}
},
"profileImageUrl": "https://myprojectwebqa.blob.core.windows.net/profileimage/3e31457c-5113-4062-b960-30f038ce7bfc.jpg",
"locationText": "Bristol",
"listingType": "Therapist",
"disabledAccess": true,
"flexibleHours": true,
"offersConcessionaryRates": false,
"homeVisits": true,
"howIWillWork": "<p>Some test data</p>",
"specialisms": [
"Health related issues:Asperger syndrome",
"Health related issues:Chronic fatigue syndrome/ME",
"Addiction, self-harm and eating disorders:Addictions",
"Addiction, self-harm and eating disorders:Eating disorders",
"Addiction, self-harm and eating disorders:Obsessions",
"Anxiety, depression and trauma:Bereavement",
"Anxiety, depression and trauma:Depression",
"Anxiety, depression and trauma:Loss",
"Coaching:Development coaching",
"Coaching:Executive coaching",
"Identity, culture and spirituality:Self esteem",
"Identity, culture and spirituality:Sexuality",
"Relationships, family and children:Infertility",
"Relationships, family and children:Relationships",
"Stress and work:Redundancy"
],
"clientele": [
"Adults",
"Children",
"Groups"
],
"approaches": [
"CBT",
"Cognitive",
"Psychoanalytic",
"Psychosynthesis"
],
"sessionTypes": [
"Home visits",
"Long-term face to face work"
],
"hourlyRate": 50,
"fullName": "Test Name",
"id": "ZWUwNGIyNjYtYjQ5Ny1lNzExLTgwZTktMzg2M2JiMzY0MGI4"
}
Please see details below;
For my case I have found out that by default the Azure Search Service returns 10 of the facets. That is why I couldn't see all my facets.
After updating my search query as follows, I have fixed my problem and now I can see all my facets in the search results - please see the facet bit updated to this; facet=specialisms, count:9999.
https://MYPROJECTNAME-search.search.windows.net/indexes/MYPROJECTNAME-directory-qa/docs?api-version=2016-09-01&$count=true&facet=specialisms, count:9999&facet=clientele, count:9999&$filter=listingType eq 'Therapist'
For the Microsoft documentation, please see the following link.
"max # of facet terms; default is 10"
https://learn.microsoft.com/en-us/rest/api/searchservice/search-documents

Node.js memory leak, despite constant Heap + RSS sizes

According to my server monitoring, my memory usage is creeping up over time:
After ~4 weeks of uptime, it ends up causing problems / crashing (which makes sense, given that I'm on EC2 with m1.large instances => 8GB RAM, and RAM seems to be increasing at about 1.5 GB / week).
If I restart my node.js app, the memory usage resets.
Yet... I'm keeping track of my memory usage via process.memoryUsage(), and even after ~1 week, I'm seeing
{"rss":"693 Mb","heapTotal":"120 Mb","heapUsed":"79 Mb"}
What am I missing? Clearly the leak is in node, yet the process seems to not be aware of it...
You can try node-memwatch module, which helps with leak detection and heap diffing in Node.
Heap diff would look similar to:
{
"before": { "nodes": 11625, "size_bytes": 1869904, "size": "1.78 mb" },
"after": { "nodes": 21435, "size_bytes": 2119136, "size": "2.02 mb" },
"change": { "size_bytes": 249232, "size": "243.39 kb", "freed_nodes": 197,
"allocated_nodes": 10007,
"details": [
{ "what": "String",
"size_bytes": -2120, "size": "-2.07 kb", "+": 3, "-": 62
},
{ "what": "Array",
"size_bytes": 66687, "size": "65.13 kb", "+": 4, "-": 78
},
{ "what": "LeakingClass",
"size_bytes": 239952, "size": "234.33 kb", "+": 9998, "-": 0
}
]
}

Resources