I am trying to use Logic apps to ping our website every 10 minutes. I would like to know how to get a response time of that call to make sure the website is now slow.
Currently i am doing this
Recurrence (Every 10 minutes)
Get Current Time
Http GET Call
Get Current time 2
Difference of (Current time 2 - Current time)
Condition to see if it is greater than threshold.
This looks like a not clean solution. Wondering if there is a easier way to get the time / latency of that HTTP call in step 3
According to the official doc, with the connector you're using is not possible to get response time. You'd better use Azure Functions for that. More info:
https://learn.microsoft.com/en-us/azure/connectors/connectors-native-http
You can use azure application insights for this kind of situation it's the best and optimal solution.
https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview
Related
Is there a proper way to have a timed process that executes on a day of the month, every month, in Node.js?
I need to pull from an external API into my database if any new records appear for that month, but I don't think it would work just to set setInterval() since the days of the month vary and such; and I think it would be a bad idea to tie it to requests in any way, but I'd still like to expose a management/admin api to set/monitor/change the updates.
Has anyone implemented something like this in Node?
I guess maybe I'd set a setInterval() on a short time and have it pull in/look for changes every cycle? But then that seems inefficient. Can a setInterval() be modified from another scope somehow after it's been set?
With node-schedule you can do exactly that, I have used it for these kind of actions
And also node-cron as someone said.
Both libraries work very similar
I am totally new to Lambda (or AWS) and am still to build knowledge and experience around it.
Now, I was building an app where in it requires to fetch data from twitter Hashtag.
If I got it correctly, Twitter restricts number of API calls we make every minute(?) hence we need to have a backend and needs to have oAuth2 authentication.
In a simple express app, I would have done an API call in the global scope to get the data and use setInterval to hit that API after every x minute so as to not exceed number of limits.
Now based on the very vague understanding, I guess Lambda runs function when we need it, Hence is it right to assume that we can't use lambda for such use cases?
The old-school way of doing this is to run a cron job that fires a particular script every so often. The AWS way of running code periodically is using CloudWatch scheduled events. You can configure how often you want to run a given target, and set the target as a lambda function.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
I want to build a nodejs application to scrape data from a website every 20mins and store it in firebase. Can you please tell me which product of google( compute engine, app engine or cloud functions ) is effective for this requirement as below are the things i am expecting to do,
1. Run Nodejs, cheerio to scrape data from website and store in firebase
2. Schedule it to run 20mins initially later may change it to 30mins or 1hr.
After reading the docs, i know that there are too many ways to implement this, but i am looking for a cost/resource effective way.
Pointers and ideas would be good.
Host the Node.js application within the App Engine[1] as Cloud Functions are event-driven[2]. You can use App Engine standard[3] or App Engine flexible[4] environment. For the scheduling part, Google Cloud Platform has a Cron Service[5] and you can create a cron job for your task hitting App Engine[6]. You can find a sample design here[7].
It depends on how much time your script spends waiting on requests. During that time the script is idle but you're getting charged at a super-high rate.
If you're doing a lot of concurrency then I would say do it with cloud functions.Another pro of doing it that way is your ip won't get blocked because it will be different every time.
Regarding scheduling, I'm not sure if Google lets do that, but I know AWS does.
A cost effective/simple way would be to use cronjob.org and have it send an http request to your cloud functions url to trigger it. If you're worried about other people triggering it, tell your cronjob to send an http header w/ an api key. Check this api key in your cloud function code to verify cronjob.org sent the request. I don't think it gets any more easy/cheap than this.
I am trying to get data from a website into a table in Excel. I am just using the regular button (get data - from web) in Excel (No code) Works fine for two websites but for a different website I am getting the following error:
Details: "The remote server return an HTTP status code '404' when trying to access 'https://smarkets.com/listing/sport/football/premier-league-2017-2018'."
The webpage certainly exists - I am guessing this is a deliberate strategy by the website to prevent data harvesting.
Anyone have any idea how I can get round it either through the get data route or a VBA approach?
Thanks
JL
I inspected traffic with Fiddler and Postman to no avail and in the end contacted the team direct for an answer.
The short answer, from their API team, is no.
Eventually our API, which may be suitable for your needs, will be
available to everyone.
API is in closed alpha stage as I mentioned in comments. More information here: API feed.
API/Odds Feed We're currently working on a new streaming API that is
faster and more scalable. The API is currently in a closed alpha
stage. Unfortunately there is no timeframe on when we'll be able to
release it to the public.
We will prioritise market makers when issuing streaming API accounts.
If you would like to gain alpha access to this service, you can apply
by outlining your proposal here
You can gain access to their XML feed with odds.smarkets.com/oddsfeed.xml .
The feed is updated every few seconds but the information is delayed
by 30 seconds.
I have an app that is hitting the rate limit for an API which is hurting the user experience. I have an idea to solve this but have no idea if this is what should be done ideally to solve this issue. Does this idea makes sense and is it a good way to solve this issue? And how should I go about implementing it? I'm using react-native and nodejs.
Here is the idea:
My app will request the data from a "middleman" API that I make. The middle man API will request data once per minute from the main API that I am having the rate limit problem with (this should solve the rate limit issue then) then store it for the one minute until it updates again. I was thinking the best way to do this is spin a server on AWS that requests from the other API every minute (Is this the easiest way to get a request every minute?) then store it on either a blank middleman webpage (or do I need to store it in a database like MongoDB?). Then my app will call from that middleman webpage/API.
Your idea is good.
Your middleman would be a caching proxy. It would act just as you stated. Hava a look at https://github.com/active-video/caching-proxy it does almost what you want. It creates a server that will receive requests of URLs, fetch and cache those, and serve the cached version from now on.
The only downside is that it does not have a lifetime option for the cache. You could either fork to add the option, or run a daemon that would delete the files that are too old to force a re-fetch.
EDIT:
A very interesting addition to the caching-proxy would be to have a head request to know if the result changed. While this is not provided by all API, this could become useful if yours is displaying such info. Only if HEAD requests do not count toward your API limits...