We are trying to create a simple local UI (standalone desktop app running on the Windows 10 Edge box) for displaying the overall health and diagnostics metrics on our IoT Edge device. The UI must be viewable if/when the device has no connection to the internet (offline). We want to show a list of modules, their status, last connection time, etc. All of this info is available in the Module Twins, but I don't see any way to access the module twins locally, while offline. Is there no local/shadow copy of these twins, or any api to query each module for its reported properties?
We are particularly interested in the edgeHub and edgeAgent modules, as they contain info about all the other modules, but I don't see any API to query those modules, other than the shell commands, iotedge list, iotedge logs edgeAgent, etc.
The only solution seems to be to programatically execute those shell commands and scrape the resulting output from the StandardOutput. Seems like there should be a better way.
If I wanted to access the module twins outside of the IoT Edge process and whilst offline, I would probably mount a volume to the module and save my reported properties in a JSON file. That way your local UI can access the files on the host machine. Of course, that does mean the module's twins are now in a readable unencrypted state on your host machine.
A more complex scenario would be to expose an HTTP endpoint on each module to access the last known values. You would need to specify a port to open on each module for this to work. You can open a port in the deployment json:
"createOptions": {
"ExposedPorts": {
"80/tcp": {}
}
}
Also, consider leveraging the new built-in metrics from system modules coming in the 1.0.10 release. Here is a lab that will help you set this up end-to-end using release candidate bits:
https://labs.iotedge.dev/codelabs/monitor-iotedge
Related
My aim is to create a mobile agent written in Node.js that will perform data collection on each node that is part of the mobile agent platform. If any anomalous events are detected then this is reported back to the control server. The mobile agent will follow a predefined path, saving its state and transferring to another nodes once its task is complete. Are there any frameworks or existing projects that would be suitable for this?
The Java Aglet project is similar to my needs however is obviously only available for Java.
As there was no current solution/package, I have created and now maintain the mobile-agent-platform NPM package.
"A JavaScript Mobile Agent Platform that allows an agent to migrate across a network. A Supervisor node dispatches the Agent which can then autonomously move between Worker nodes, carrying/transferring its state and methods to the new node."
Note: the stack and program counter cannot be transferred due to limitations within Node.js.
Ok, so a bit of background on what we are doing.
We have various weather station and soil monitoring stations across the country that gather up data and then using FTP, upload to a server for processing.
Note: this server is not located in the GCP, but we are migrating all our services over at the moment.
Annoyingly FTP is the only service that these particular stations allow. Newer stations thankfully are using REST APIs instead, so that makes it much simpler.
I have written a small nodejs app that works with ftp-srv. This acts as the FTP server.
I have also written a new FileSystem class that will hook directly into Google Cloud Storage. So instead of getting a local directory, it reads the GCS directory.
This allows for weather stations to upload their dump files direct to GCP for processing.
My question is, what is the best service to use?
First I thought using App Engine, since its just a small nodejs app, I don't really want to have to go and create a VM for it just to run this.
However, I have found that I have been unsuccessful to open up port 21 and any other ports used for passive FTP.
I then thought using Kubernetes Engine. To be honest, I don't know anything at all about this, as of yet. But it seems like its a bit of an overkill just to run the small app.
My last thought would be to use Compute Engine. I have a working copy with PROFTPD installed and working, so I know I can get the ports open and have data flowing, but I feel that it's a bit overkill to run a full VM just for something that is acting as an intermediary between the weather stations and GCS.
Any recommendations would be very appreciated.
Thanks!
Kubernetes just for FTP would be using a crane to lift your fork.
Google Compute Engine and PROFTPD will fit in a micro instance at a whopping cost of about $6.00 per month.
The other Google Compute services do not support FTP. This includes:
App Engine Standard
App Engine Flexible
Cloud Run
Cloud Functions
This leaves you with either Kubernetes or Compute Engine.
I have a question regarding the Node Red dashboard. I've got my dashboard all set up and working. Now, I want to be able to access the dashboard outside of my local network. Right now I do this through a VNC server. What needs to happen next is that clients need to able to access the dashboard, but they are not getting access to my VNC server of course. I have done my fair amount of Google work. I (somewhat) understand that a service like ngrok (ngrok.com) or dataplicity (dataplicity.com) is what I am looking for. What would be the best way of setting this up safely?
Might be useful to clarify: I'm using a raspberry Pi!
Thanks in advance!
If you want to give the outside world access to your dashboard, you can also consider to host your node-red application in the cloud. See links at the bottom-left of page https://nodered.org/docs/getting-started/
Most of those services have a free tier - so it might you cost nothing.
If you cannot deploy your complete node-red in the cloud (e.g. because it is reading local sensors) then you can split your node-red application into 2 node-red applications: one running locally and one (with the dashboard) running in the cloud. Of course then the 2 node-red applications need to exchange messages: for this the cloud services mentioned on that page also provides a secure way to send and receive events from the node-red cloud application that you can use.
I have a java class which manage jobs and execute them via spark(using 1.6).
I am using the API - sparkLauncher. startApplication(SparkAppHandle.Listener... listeners) in order to monitor the state of the job.
The problem is I moved to work in a real cluster environment and this way can’t work when the master and workers are not on the same machine, as the internal implementation is making a use of localhost only (loopback) to open a port for the workers to bind to.
The API sparkLauncher.launch() works but doesn’t let me monitor the status.
What is the best practice for cluster environment using a java code?
I also saw the option of hidden Rest API, is it mature enough? Should I enable it in spark somehow (I am getting access denied, even though the port is open from outside) ?
REST API
In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for both running applications, and in the history server. The endpoints are mounted at /api/v1. Eg., for the history server, they would typically be accessible at http://:18080/api/v1, and for a running application, at http://localhost:4040/api/v1.
More details you can find here.
Every SparkContext launches a web UI, by default on port 4040, that displays useful information about the application. This includes:
A list of scheduler stages and tasks
A summary of RDD sizes and memory usage
Environmental information.
Information about the running executors
You can access this interface by simply opening http://driver-node:4040 in a web browser. If multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc).
More details you can find here.
I've been asked to configure an ELK stack, in order to manage log from several applications of a certain client. I have the given stack hosted and working at a redhat 7 server (followed this cookbook), and a test instance in a virtual machine with Ubuntu 16.04 (followed this other cookbook), but I've hit a roadblock and cannot seem to get through it. Kibana is rather new for me and maybe I don't fully understand the way it works. In addition, the most important application of the client is a JHipster managed application, another tool I am not familiarized.
Up until now, all I've found about jhipster and logstash tells me to install the full ELK stack using Docker (which I haven't, and would rather avoid in orther to keep the configuration I've already made), so that Kibana deployed through that method already has configured a dashboard tunned for displaying the information that the application will send with the native configuration, activated in the application.yml logstash: enabled: true.
So... my questions would be... Can I get that preconfigured jhipster dashboard imported in my preexistent Kibana deploy. Where is the data, logged by the application, stored? can I expect a given humanly readable format? Is there any other way of testing the configuration is working, since I don't have any traffic going through the test instance into the VM?
Since that JHipster app is not the only one I care for, I want other dashboards and inputs to be displayed from other applications, most probably using file beat.
Any reference to useful information is appreciated.
Yes you can. Take a look at this repository: https://github.com/jhipster/jhipster-console/tree/master/jhipster-console
there are the exports (in JSON format) from kibana stored in the repository, along with the load.sh
The scripts adds the configuration by adding them via the API. As you can imply, any recent dashboard is not affected by this, so you can use your existing configuration.