I have a system where I have 2 projects.
Project A contains shell script abc.sh and Project B is a Nodejs project and contains an API. Shell script contains a curl command to PUT. Problem is this curl operation gets executed thrice instead of once. Could it be possible that during API call, if API call takes lot of time to respond, curl automatically retries to make the API call. I can see that calls are made every 2mins. I can see some network logs for 5mins (which is more than usual time for jobs at other times)
Where can I get to see the Curl configuration to know if there is any retry mechanism.
script abc.sh looks like below
echo "making API call"
curl -k -u user -X PUT payload projectb.com/api/submitrequest
echo "api called"
and my Project B API looks like below
router.put(/api/submitrequest)
await dosomejob()
res.send('success')
Ideally the log of shell script should look like below
making API call
///network traffic related information like time elapsed, remaining etc///
'success' (-- this is coming back from API in Project B)
api called
However, sometimes, I see that the API call is made thrice though I see only one call in the script. In this case, I don't even see the response coming back from API. After seeing network traffic related info in the logs for 5mins, I directly see the message 'API called'.
Below is how the logs looks like:
making API call
///network traffic related information like time elapsed, remaining etc///
api called
Related
I have created a NodeJS express post api which process the file stored in azure blob. In the post call to this api, I am just sending filename in body. This call will get the data from blob, create a local file with that data, run command on that file and store it in blob again. But that command is taking 10-15 mins to process one file. That's why the request is getting timed out. I had 2 questions here:
Is there way to respond to the call before processing starts. Like respond to the api call and then start file processing.
If 1 is not possible, which is the best solution for this problem
Thank you in advance.
You must use queue for long running tasks. For that you can choose any library like agendajs, bull, bee-queue, kue etc
Background
We have a PHP App Service and MySQL that is deployed using an Azure Devops Pipeline (YML). The content itself is a PHP site that it packaged up into a single file using Akeeba by an external supplier. The package is a Zip file (which can be deployed as a standard Zip deployment) and inside the Zip file is a huge JPA file. The JPA is essentially the whole web site plus database tables, settings, file renames and a ton of other stuff all rolled into one JPA file. Akeeba essentially unzips the files, copies them to the right places, does all the DB stuff and so on. To kick the process off, we can simply connect to a specific URL (web site + path) and run the PHP which does all the clever unpackaging via a web GUI. But, we want to include this stage in the pipeline instead so that the process is fully automated end to end. Akeeba has a CLI as an alternative to the Web GUI deployment, so it should go like this:
Create web app
Deploy the web site ZIP (zipDeploy)
Use the REST API to access Kudu and run the relevant command (php install.php web.jpa) to unpack the jpa and do the MySQL stuff - this normally takes well over 30 minutes (it is a big site and it has a lot of "stuff" to do - but, it does actually work in the end).
The problem is that the SCM REST API has a hard-coded 230s limit as described here: https://blog.headforcloud.com/2016/11/15/azure-app-service-hard-timeout-limit/
So, the unpack stage keeps throwing "Invoke-RestMethod : 500 - The request timed out" exactly on the 230s mark.
We have tried SCM_COMMAND_IDLE_TIMEOUT and WEBJOBS_IDLE_TIMEOUT but, unsurprisingly, they did not make any difference.
$cmd=#{"command"="php .\site\wwwroot\install.php .\site\wwwroot\web.jpa .\site\wwwroot"}
Invoke-RestMethod -Uri $url -Headers #{"Authorization"="Basic $creds"} -Body (ConvertTo-Json($cmd)) -Method Post -ContentType "application/json" -TimeoutSec 7200
I can think of a few hypothetical ways around it (some quite eccentric):
Find another way to run CLI commands inside the Web App after deployment other than the Kudu REST API. Is there such a thing? I Googled and checked SO but all I found were pointers to the way we do it (or try to do it) now.
Use something like Selenium to click the GUI buttons instead of using the CLI. (I do not know if they would suffer a timeout.)
Instead of running the command via Kudu REST, use the same API to create and deploy a script to the web server, start it and then let the REST API exit whilst the script still runs on the Web App. Essentially, bodge an async call but without the callback and then have the pipeline check in on the site at, say, 5 minute intervals. Clunky.
Extend the 230s limit - but I do not think that Microsoft make this possible.
Make the web site as fast as possible during the deployment in the hope of getting it under the 4-minute mark and then down-scale it. Yuk!
See what the Akeeba JPA unpacking actually does, unpack it pre-deployment and do what the unpackage process does but controlled via the Pipeline. This is potentially a lot of work and would lose the support of the supplier.
Give up on an automated deployment. That would rather defeat much of the purpose of a Devops pipeline.
Try AWS + terraform instead. That's not a approved infrastructure environment, however.
Given that Microsoft understandably do not want long-running API calls hanging around, I understand why the limit exists. However, I would expect therefore there to be a mechanism to interact with an App Service file system via a CLI in another way. Does anyone know how?
The 4 minute idle timeout on the TCP level and this is implemented on the Azure hardware load balancer. This timeout is not configurable and this cannot be changed. One thing I want to mention is that this is idle timeout at the TCP level which means that if the connection is idle only and no data transfer happening, only then this timeout is hit. To provide more info, this will hit if the web application got the request and kept processing the request for > 4minutes without sending any data back.
Resolution
Ideally in a web application, it is not good to keep the underlying HTTP request open and 4 minutes is a decent amount of time. If you have a requirement about background processing within your web application, then the recommended solution is to use Azure WebJobs and have the Azure Webapp interact with the Azure Webjob to notify once the background processing is done (there are many ways that Azure provides like queues triggers etc. and you can choose the method that suits you the best). Azure Webjobs are designed for background processing and you can do as much background processing as you want within them. I am sharing a few articles that talk about webjobs in detail
· http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx
· https://azure.microsoft.com/en-us/documentation/articles/websites-webjobs-resources/
============================================================================
It totally depends on the app. Message Queue comes to mind. There are a lot of potential solutions and it will be up to you to decide.
============================================================================
Option #1)
You can change the code to send some sort of header to continue to the client to keep the session open.
A sample is shown here
This shows the HTTP Headers with the Expect 100-continue header:
https://msdn.microsoft.com/en-us/library/aa287673%28v=vs.71%29.aspx?f=255&MSPPError=-2147217396
This shows how to add a Header to the collection:
https://msdn.microsoft.com/en-us/library/aa287502(v=vs.71).aspx
Option #2) Progress bar
This sample shows how to use a progress bar:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/d84f4c89-ebbf-44d3-bc4e-43525ae1df45/how-to-increase-progressbar-when-i-running-havey-query-in-sql-server-or-oracle-?forum=csharpgeneral
Option #3) A common practice to keep the connection active for a longer period is to use TCP Keep-alive. Packets are sent when no activity is detected on the connection. By keeping on-going network activity, the idle timeout value is never hit and the connection is maintained for a long period
Option #4) You can also try the option of hosting your application as an IaaS VM instead of a APP SERVICE. This may avoid the ARR timeout issue because its architecture is different and I believe that the time-out is configurable.
I have an external bash script that transcodes audio files using FFmpeg and then uploads the files to google cloud storage. I am using the google cloud run platform for this process but the process is stopping in the middle and not getting any clue from the logs. I am using the node js spawn command to execute the bash script
const createHLSVOD = spawn('/bin/bash', [script, file.path, file.destination, contentId, EPPO_MUSIC_HSL_URL, 'Content', speed]);
createHLSVOD.stdout.on('data', d => console.log(`stdout info: ${d}`));
createHLSVOD.stderr.on('data', d => console.log(`stderr error: ${d}`));
createHLSVOD.on('error', d => console.log(`error: ${d}`));
createHLSVOD.on('close', code => console.log(`child process ended with code ${code}`));
on cloud run beginning the process itself taking a lot of time but in my local machine transcoding and uploading is very fast. after some time transcoding logs are being stopped and no new logs appear. I have no clue what is happening
so what is happening here? why it is very slow in the first place and why the process is being stopped in middle without any error
node js script
Transcoding script
Dockerfile
The issue with spawn is that you create an asynchronous process that run in background. It's a problem, because Cloud Run allow CPU to the container only when a request is being processed. And in your case, you have that
A request arrive
A spawn is created
The spawned script run in background
An HTTP answer is sent on the request. Cloud Run throttles the CPU
Your spawned script continue to run
There is 2 consequences
Your script processing is very very long because the throttling limit the CPU under 5%
After a period with activity (i.e. request received by the instance), Cloud Run kill the unused instance to same resource on its side. it's about 15 minutes, but it's subject to change, it's Google Cloud internal sauce
I recommend you to wait the end of the spawned script or to use a synchronous call; such as execSync, instead of async spawn.
I think the best solution for this issue is to change the approach of the architecture. Instead of using the client to upload the file to Cloud Storage through Cloud Run I would suggest using the signed URL. Signed URL provides permission and time to make a request, allowing users without credentials to perform specific action on a resource, for example you can give the user write access to a bucket for a limited time so they can upload a file. From the client side, this is very similar to your current process, changing the Cloud Run URL for the signed URL.
You can check this link to read more about signed URLs, and here is a quick guide of how to generate it.
For example, you can create a signed URL to test with this command:
First create a private key
gcloud iam service-accounts keys create "file where you want to storage the key" \
--iam-account="name of the service account"#projectID.iam.gserviceaccount.com
Now you authenticate the service account
gcloud auth activate-service-account --key-file KEY_FILE_LOCATION/KEY_FILE_NAME
And now create the signed URL
gsutil signurl -m PUT -d 1h -c CONTENT_TYPE -u gs://BUCKET_NAME/OBJECT_NAME
After that, the client uploads the file using the signed URL to the bucket in Cloud Storage. I would use Pub/Sub notification so you can know that the file was uploaded without any problem, and with that notification you can use it to trigger other operations in Cloud Run.
Basically Pub/Sub notifications are sent when an object changes in a specific bucket, you can follow this guide that could help you configure that notification.
For example, to get notified about newly upload objects, you can configure it this way:
gsutil notification create -e OBJECT_FINALIZE -f json gs://<bucket name>
To use this notification to trigger a Cloud Run process you can see this link.
I need kind of snmp v2c proxy (in python) which:
react on snmpset command
read value from command and write it to yaml file
run custom action (prefer in different thread and somehow reply success to snmpset command):
run another snmpset to different machine, or
ssh to user#host and run some command, or
run some local tool
and:
react on snmpget command
check value for requested oid in yaml file
return this value
I'm aware of pysnmp but documentation just confuse me. I can image I need some kind of command responder (I need snmp v2c) and some object to store configuration/values form yaml file. But I'm completely lost.
I think you should be able to implement all that with pysnmp. Technically, that would not be SNMP proxy but SNMP command responder.
You can probably take this script as a prototype and implement your (single) cbFun which (like the one in the prototype) receives the PDU and branches on its type (GET or SET SNMP command in your case). Then you can implement value read from the .yaml file in the GetRequestPDU branch and .yaml file write, along with sending SNMP SET command elsewhere, in the SetRequestPDU branch.
The pysnmp API we are talking here is the low-level one. With it you can't ask pysnmp to route messages between callback functions -- it always calls the same callback for all message types.
However, you can also base your tool on the higher-level SNMP API which was introduces along with the SNMPv3 model. With it you can register your own SNMP applications (effectively, callbacks) based on PDU type they support. But given you only need SNMPv2c support I am not sure the higher-level API would pay off in the end.
Keep in mind that SNMP is generally time sensitive. If running local command or SSH-ing elsewhere is going to take more than a couple of seconds, standard SNMP manager might start retrying and may eventually time out. If you look at how Net-SNMP's snmpd works - it runs external commands and caches the result for tends of seconds. That lets the otherwise timing out SNMP manager eventually get a slightly outdated response.
Alternatively, you may consider writing a custom variation plugin for SNMP simulator which largely can do what you have described.
On cloudControl, I can either run a local task via a worker or I can run a cronjob.
What if I want to perform a local task on a regular basis (I don't want to call a publicly accessible website).
I see possible solutions:
According to the documentation,
"cronjobs on cloudControl are periodical calls to a URL you specify."
So calling the file locally is not possible(?). So I'd have to create a page I can call via URL. And I have to perform checks, if the client is on localhost (=the server) -- I would like to avoid this way.
I make the worker sleep() for the desired amount of time and then make it re-run.
// do some arbitrary action
Foo::doSomeAction();
// e.g. sleep 1 day
sleep(86400);
// restart worker
exit(2);
Which one is recommended?
(Or: Can I simply call a local file via cron?)
The first option is not possible, because the url request is made from a seperate webservice.
You could either use HTTP authentication in the cron task, but the worker solution is also completely valid.
Just keep in mind that the worker can get migrated to a different server (in case of software updates or hardware failure), so do SomeAction() may get executed more often than once per day from time to time.