My company use Amazon QuickSight at the moment, and faced a big problem when using this service. In short, we have a lot of dashboards that, unfortunately, are not updated in real time. We had tried to use real time parameters in QuickSight configuration when we create dashboards, but it did't work properly because QuickSight send request like
"Select * From SomeTable"
constantly and it takes a lot of time.
First because this 'SomeTable' it’s a view that executed several hours and then several hours we send via tcp result to QuickSight. It's all work extremely slow. Can you advice any solution that allow us to create real time updating? We need the following behavior:
QuickSight get all data from us sending query "Select * From SomeTable"
We send from our side any changes to Amazon QuickSight, that happens in our "SomeTable"
Please tell me is it possible to implement this behaviour for Amazon QuickSight? I'II be very grateful. Because we're really just stuck with this problem.
Related
I want to build a nodejs application to scrape data from a website every 20mins and store it in firebase. Can you please tell me which product of google( compute engine, app engine or cloud functions ) is effective for this requirement as below are the things i am expecting to do,
1. Run Nodejs, cheerio to scrape data from website and store in firebase
2. Schedule it to run 20mins initially later may change it to 30mins or 1hr.
After reading the docs, i know that there are too many ways to implement this, but i am looking for a cost/resource effective way.
Pointers and ideas would be good.
Host the Node.js application within the App Engine[1] as Cloud Functions are event-driven[2]. You can use App Engine standard[3] or App Engine flexible[4] environment. For the scheduling part, Google Cloud Platform has a Cron Service[5] and you can create a cron job for your task hitting App Engine[6]. You can find a sample design here[7].
It depends on how much time your script spends waiting on requests. During that time the script is idle but you're getting charged at a super-high rate.
If you're doing a lot of concurrency then I would say do it with cloud functions.Another pro of doing it that way is your ip won't get blocked because it will be different every time.
Regarding scheduling, I'm not sure if Google lets do that, but I know AWS does.
A cost effective/simple way would be to use cronjob.org and have it send an http request to your cloud functions url to trigger it. If you're worried about other people triggering it, tell your cronjob to send an http header w/ an api key. Check this api key in your cloud function code to verify cronjob.org sent the request. I don't think it gets any more easy/cheap than this.
Let me start out by saying I am a SQL Server Database expert, not a coder so making API calls is certainly not an everyday task for me.
Having said that, I am trying to use the Azure Data Factory's data copy tool to import data from Clio to an Azure SQL Server database. I have had some limited success, data is copied over using the API and inserted into the target table but paging really seems to be an issue. I am testing this with the billable_clients call and the first 25 records with the fields I specify are inserted along with the paging record. As I understand, the billable_clients call is eligible for bulk actions which may be the solution, although I've not been able to figure out how it works. The url I am calling is below:
https://app.clio.com/api/v4/billable_clients.json?fields=id,unbilled_hours,name
Using Postman I've tried to make the same call while adding X-BULK true to the header but that returns no results. If there is anyone that can shed some light on how the X-BULK header flag is used when making a call, or if anyone has any experience loading Clio data into a SQL Server database I'd love some feedback on your methods.
If any additional information regarding my attempts or setup would help please let me know.
Thanks!
you need to download the json files with Bulk API and then update them in DB.
It isn't possible to directly insert the data
I am trying to get data from a website into a table in Excel. I am just using the regular button (get data - from web) in Excel (No code) Works fine for two websites but for a different website I am getting the following error:
Details: "The remote server return an HTTP status code '404' when trying to access 'https://smarkets.com/listing/sport/football/premier-league-2017-2018'."
The webpage certainly exists - I am guessing this is a deliberate strategy by the website to prevent data harvesting.
Anyone have any idea how I can get round it either through the get data route or a VBA approach?
Thanks
JL
I inspected traffic with Fiddler and Postman to no avail and in the end contacted the team direct for an answer.
The short answer, from their API team, is no.
Eventually our API, which may be suitable for your needs, will be
available to everyone.
API is in closed alpha stage as I mentioned in comments. More information here: API feed.
API/Odds Feed We're currently working on a new streaming API that is
faster and more scalable. The API is currently in a closed alpha
stage. Unfortunately there is no timeframe on when we'll be able to
release it to the public.
We will prioritise market makers when issuing streaming API accounts.
If you would like to gain alpha access to this service, you can apply
by outlining your proposal here
You can gain access to their XML feed with odds.smarkets.com/oddsfeed.xml .
The feed is updated every few seconds but the information is delayed
by 30 seconds.
We are observing that there are three API calls happening when we execute an offline sync selective pull query
GET domain/tables/Events?$filter=updatedAt%20ge%20datetimeoffset'1969-12-30T22:00:00.000Z'
GET domain/tables/Events?$filter=updatedAt%20ge%20datetimeoffset'2017-06-27T22:00:00.000Z' (current datetime)
GET domain/tables/Events?$filter=updatedAt%20ge%20datetimeoffset'2017-06-27T22:00:00.000Z'&$skip=1
These 3 calls happen every time a pull is done, can anyone explain why this happens? The selective sync query is created in the following format
syncContext
.pull(new WindowsAzure.Query('Events'), 'eventspull')
.then(function() { /* pull complete */ });
We are using latest version of the following javascript offline library. https://zumo.blob.core.windows.net/sdk/azure-mobile-apps-client.js
These 3 calls happen every time a pull is done, can anyone explain why this happens?
This happens because the "pull" function pulls one page from the server table at a time. You can check out the source code here for details.
Let’s say you have thousands of records. If you execute the query without paging, then it is likely you will tie up your client process on the phone for a considerable period of time as you receive and process the data. To alleviate that and allow your mobile application to remain responsive, the client SDK implements paging. By default, 50 records will be requested for each paged operation. In reality, this means that you will see one more request than you expect.
For more info, pelease refer to Understanding offline sync.
I have a website which can have up to 500 concurrent viewers, with data updated every three seconds. Currently each user has an AJAX object which calls a web-page every three seconds which queries a DB and returns with the results.
What I would love to do is have each client get a socket to a node.js object, this node.js would poll the DB every 3 seconds for updated data, if it had updated data it would then be announced (ideally through JSON) and each client would then have the data pushed to it and update the page accordingly.
If this is possible, does anyone have a recommendation as to where I start? I am fairly familiar with JS but node.js seems to confuse me.
Thanks
I myself have quite few experience with node.js.
It is absolutely doable and looks like the perfect use case for node.js.
I recommend starting with an Express Tutorial and later on use socket.io.
I don't know which DBMS you are using, but there probably is a nice package for that as well. Just look through this list.