How to reliably schedule regularly reoccurring jobs with arangodb? - arangodb

I am evaluating arangodb "Community Edition" v.3.7.6 in Cluster Mode. At the moment, I am trying to figure out, how to reliably schedule regularly reoccurring jobs via queues in my foxx service.
I have read the documentation pages here and here and came up with the following scripts:
setup.js
const queues = require('#arangodb/foxx/queues')
var myQueue = undefined
try {
myQueue = queues.get('myqueue')
} catch (e) {
myQueue = queues.create('myqueue')
}
myQueue.push({
name: "myRegularTask",
mount: module.context.mount
},
{to: 'hello#example.com', body: 'Hello world'},
{
repeatTimes: Infinity,
repeatUntil: Date.now() + (24 * 60 * 60 * 1000),
repeatDelay: 5 * 1000
});
myRegularTask.js
'use strict';
const db = require('#arangodb').db;
const testCollection = db._collection('test')
testCollection.save({text: 'sample text'});
manifest.json
{
"$schema": "http://json.schemastore.org/foxx-manifest",
"main": "index.js",
"engines": {
"arangodb": "^3.0.0"
},
"name": "myapp",
"version": "0.1.0",
"tests": "test/**/*.js",
"scripts": {
"setup": "scripts/setup.js",
"myRegularTask": "scripts/myRegularTask.js"
}
}
So what I think it is supposed to do is that it registers a regular job that is executed every 5 seconds (and inserts a document into my testCollection) once I have installed my foxx service.
It actually does that sometimes. However, most of the time nothing happens, no matter how long I wait.
And that is the question of this post:
How come that the behavior of above code is undeterministic to me? What am I doing wrong to get it working always?
Thanks
Seb

Related

Facing TLS write failed: -2 issue while writing to aerospike

I have aerospike 6.0.0 installed on server and being used as cache. Also I am using Node.js client 5.2.0 to connect to the database.The aerospike nodes are configured to use TLS. The config of the namespace I am using is:
namespace application_cache {
memory-size 5G
replication-factor 2
default-ttl 1h
nsup-period 1h
high-water-memory-pct 80
stop-writes-pct 90
storage-engine memory
}
The config I am passing while connecting to the Db is:
"config": {
"captureStackTraces":true,
"connTimeoutMs": 10000,
"maxConnsPerNode": 1000,
"maxCommandsInQueue" : 300,
"maxCommandsInProcess" : 100,
"user": "test",
"password": "test",
"policies": {
"read": {
"maxRetries": 3,
"totalTimeout": 0,
"socketTimeout": 0
},
"write": {
"maxRetries": 3,
"totalTimeout": 0,
"socketTimeout": 0
},
"info": {
"timeout": 10000
}
},
"tls": {
"enable": true,
"cafile": "./certs/ca.crt",
"keyfile": "./certs/server.key",
"certfile": "./certs/server.crt"
},
"hosts": [
{
"addr": "-----.com",
"port": 4333,
"tlsname": "----"
},
{
"addr": "---.com",
"port": 4333,
"tlsname": "-----"
},
{
"addr": "----.com",
"port": 4333,
"tlsname": "-----"
}
]
}
This is the function I am using to write to a set in this namespace:
async function updateApp(appKey, bins) {
let result = {};
try {
const writePolicy = new Aerospike.WritePolicy({
maxRetries: 3,
socketTimeout: 0,
totalTimeout: 0,
exists: Aerospike.policy.exists.CREATE_OR_UPDATE,
key: Aerospike.policy.key.SEND
});
const key = new Aerospike.Key(namespace, set, appKey);
let meta = {
ttl: 3600
};
let bins = {};
result = await asc.put(key, bins, meta, writePolicy);
} catch (err) {
console.error("Aerospike Error:", JSON.stringify({params: {namespace, set, appKey}, err: {code: err.code, message: err.message}}));
return err;
}
return result;
}
It works most of the time but once in a while I get this error:
Aerospike Error: {"params":{"namespace":"application_cache","set":"apps","packageName":"com.test.application"},"err":{"code":-6,"message":"TLS write failed: -2 34D53CA5C4E0734F 10.57.49.180:4333"}}
Every record in this set has about 12-15 bins and the size of the record varies between 300Kb to 1.5Mb. The bigger the size of the record the higher chance of this error showing up. Has anyone faced this issue before and what could be causing this?
I answered this on the community forum but figured I'd add it here as well.
Looking at the server error codes a code: 6 is AS_ERR_BIN_EXISTS. Your write policy is setup with exists: Aerospike.policy.exists.CREATE_OR_UPDATE but the options for exists are:
Name
Description
IGNORE
Write the record, regardless of existence. (I.e. create or update.)
CREATE
Create a record, ONLY if it doesn't exist.
UPDATE
Update a record, ONLY if it exists.
REPLACE
Completely replace a record, ONLY if it exists.
CREATE_OR_REPLACE
Completely replace a record if it exists, otherwise create it.
My guess is that it's somehow defaulting to a CREATE only causing it to throw that error. Try updating your write policy to use exists: Aerospike.policy.exists.IGNORE and see if you still get that error.

Azure C# (.NET) timer function not running on schedule OR manually

I have an Azure Function with a Timer trigger. The schedule is "*/6 * * * *" (running every six minutes). I cannot run it manually and it does not fire automatically. Below is my function.json:
{
"generatedBy": "Microsoft.NET.Sdk.Functions-1.0.31",
"configurationSource": "attributes",
"bindings": [
{
"type": "timerTrigger",
"schedule": "%TimerTriggerPeriod%",
"useMonitor": true,
"runOnStartup": false,
"name": "myTimer"
}
],
"disabled": false,
"scriptFile": "../bin/AccessChangeMonitoring.dll",
"entryPoint": "Microsoft.IT.Security.AccessChangeMonitoring.AccessChangeMonitoring.InitiateChangeMonitoring"
}
%TimerTriggerPeriod% is defined in my local.settings.json file ("TimerTriggerPeriod": "0 */6 * * * *"). Looking at the Application Function Count metric display on the dashboard, it shows that my function has been executed 0 times:
Below is my host.json:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingExcludedTypes": "Request",
"samplingSettings": {
"isEnabled": true
}
}
},
"functionTimeout": "00:07:00"
}
Below is the function code:
[FunctionName("InitiateChangeMonitoring")]
public static async Task InitiateChangeMonitoring([TimerTrigger("%TimerTriggerPeriod%")] TimerInfo myTimer, ILogger log)
{
log.LogInformation("Change Monitoring started.");
// Reset the listing of app ids we need to retrieve delta role assignments for
oneAuthZAppIds = new List<string>();
await GetOneAuthZAppIdsAsync();
// Create the necessary Cosmos DB infastructure
await CreateDatabaseAsync();
await CreateContainerAsync();
await CreateDeltaAPICallDatabaseAsync();
await CreateDeltaAPICallContainerAsync();
await CreateManagerMappingDatabaseAsync();
await CreateManagerMappingContainerAsync();
// Compute the authentication token needed to access the PAP Service API
log.LogInformation("\nRetrieve PAPServiceAPIToken");
string PAPServiceAPIToken = await GetTokenAsync(Environment.GetEnvironmentVariable("OneAuthZAppUri"), Environment.GetEnvironmentVariable("OneAuthZAppId"),
PAPAuthenticationSecret);
log.LogInformation("PAPServiceAPIToken = " + PAPServiceAPIToken);
string GraphAPIAuthenticationToken = await GetTokenAsync(Environment.GetEnvironmentVariable("GraphAppUri"), Environment.GetEnvironmentVariable("GraphClientId"),
graphKey);
log.LogInformation("graphAPIAuthenticationToken = " + GraphAPIAuthenticationToken);
await runChangeMonitoringSystemAsync(PAPServiceAPIToken);
}
First, I notice you specify you are using local.settings.json to save the environment variable and at the same time you show the metric on azure.
So, I think the first problem why your azure function can not be triggered is because you don't set the environment variable on azure.
You should set in this place instead of local.settings.json (because when a function is deployed to Azure, it will never take environment variable from local.settings.json):
(Don't forget to save the edit.)
Second, as Ivan says, your format of cron is wrong. The format of timetrigger should be in this format:
{second} {minute} {hour} {day} {month} {day of week}

Jest RXjs6 Marbles, how to test existing observable

I am trying to figure out how to structure a jest unit test for the following:
I have a bunch of existing observables and subjects I want to test in a project. So I have started off with what I would assume to be a simple test case using RXjs marbles inside jest:
const testScheduler = new TestScheduler((actual, expected) => {
expect(actual).toEqual(expected);
});
testScheduler.run(({cold, expectObservable}) => {
const inputSubject = new Subject();
const outputSubject = new Subject();
inputSubject.pipe(map(val => val * 2)).subscribe(outputSubject);
cold('a|', {a:1}).subscribe(inputSubject);
expectObservable(outputSubject).toBe('a|', {a:2});
});
}
While this test does run, it fails. I am having trouble understanding the output from the test run. I would like to know if I am doing something wrong, or if I am misunderstanding things in general incorrectly. Or perhaps just some explanation of the output would be helpful.
- Expected
+ Received
## -1,16 +1,7 ##
Array [
Object {
- "frame": 0,
- "notification": Notification {
- "error": undefined,
- "hasValue": true,
- "kind": "N",
- "value": 2,
- },
- },
- Object {
"frame": 1,
"notification": Notification {
"error": undefined,
"hasValue": false,
"kind": "C",
...... the code ......
at TestScheduler.assertDeepEqual (common/chat/newMessages/ratifiedSendMessages/twilioRatifiedSendMessages.unit.test.js:27:24)
at node_modules/rxjs/src/internal/testing/TestScheduler.ts:159:14
at Array.filter (<anonymous>)
at TestScheduler.Object.<anonymous>.TestScheduler.flush (node_modules/rxjs/src/internal/testing/TestScheduler.ts:157:39)
at TestScheduler.run (node_modules/rxjs/src/internal/testing/TestScheduler.ts:392:12)
at Object.<anonymous> (common/chat/newMessages/ratifiedSendMessages/twilioRatifiedSendMessages.unit.test.js:30:21)
NVM. Looks like I was supposed to use 'hot' not 'cold'. As soon as I switched the 'cold' call out for 'hot' instead, it worked fine. Although if anyone else feel like explaining why this is, I am all ears.

Is there a way to create a scheduled query with pubsub notification using googleapiclient using python?

I want to create dynamic scheduled queries using Python and I want to publish a message on PubSub when the query completes. I know I can do that from the UI but that's not what I'm looking for.
Currently I'm doing this, but the "notificationPubsubTopic" field gets ignored in the request
import googleapiclient.http
from googleapiclient import discovery, errors
resource = discovery.build("bigquerydatatransfer", "v1")
body = {
"notificationPubsubTopic": "projects/{my_project}/topics/{my_topic}",
"scheduleOptions": {
"disableAutoScheduling": False
},
"disabled": False,
"displayName": "my_table_name",
"dataSourceId": "scheduled_query",
"destinationDatasetId": "test",
"emailPreferences": {
"enableFailureEmail": False
},
"params": {
"query": "select 1",
"write_disposition": "WRITE_TRUNCATE",
"destination_table_name_template": "table_name_test"
},
"schedule": "every day 09:35"
}
creation_job = res.projects().transferConfigs().create(parent=project, body=body)
creation_job.execute()
Google released a few days ago the new version of the datatransfer library adding the support for notification_pubsub_topic when creating transfer configs.

Highcharts Export Service Gantt Chart not working

Attempting to pilot a server-side render of a gantt chart using highcharts-export-server, but unable to get the series data to render at all.
As per the docs, I've tried running the export via commandline and as a node module, but I always get an empty graph with the title, subtitle and series name displaying, but none of the actual data.
Failed Gantt Render
All the basic line graph example configurations work and I'm able to render the chart using the Highcharts.ganttChart method via the browser, so I believe it's specific to the Gantt chart configuration for the export server, or related to the millisecond date conversion (as you can see in the screenshot, the X Axis is not rendering as dates, but rather plain numbers). As per suggestions on other threads, I ran build.js in the node_modules/highcharts-export-server, globally for the commandline attempt and project-locally for the node module attempt, making sure to enable the both the gantt and moment libraries, but that didn't help either.
Including my options json below. I haven't been able to find a gantt config example specifically for the export server, so this is my best attempt to interpolate what they'd be:
{
"title": {
"text": "Gantt PoC"
},
"subtitle": {
"text": "Timeline"
},
"series": [
{
"name": "Gantt Demo",
"type": "gantt",
"data": [
{
"name": "Demo Task 1",
"id": "demo_task_1",
"start": 1564113600000,
"end": 1564718400000
},
{
"name": "Demo Task 2",
"id": "demo_task_2",
"start": 1564113600000,
"end": 1564718400000
}
]
}
],
"xAxis": {
"min": 1563681600000,
"max": 1571803200000
}
}
Please let me know if there's anything obviously wrong with my config, if I missed any crucial steps to prep the environment, or any ideas you may have for me to troubleshoot. Thanks!
i also tried to generate image using ganttchart, just i tried constructor which you suggested working perfectly, Thanks.
highcharts-export-server --infile gantt.json --outfile gantt.png --type png --constr ganttChart
actually i tried with HTTP Server as well, working perfectly. Thanks #pawel_d
const fs = require("fs");
const chartExporter = require('highcharts-export-server');
chartExporter.initPool();
const chartDetails = {
type: "png",
constr : 'ganttChart',
options: {
series: [{
data: [{
start: 1,
end: 10
}]
}]
}
};
chartExporter.export(chartDetails, (err, res) => {
// Get the image data (base64)
let imageb64 = res.data;
// Filename of the output
let outputFile = "bar.png";
// Save the image to file
fs.writeFileSync(outputFile, imageb64, "base64", function(err) {
if (err) console.log(err);
});
console.log("Saved image!");
chartExporter.killPool();
});

Resources