I have the following problem:
We have a table (partitioned by timestamp) that saves data from IoT devices (a lot of data, the expected ratio for new incoming data is about every 30sec each device).
The problem is that we will serving a query service and users can set a "since" and "until" filters over a "ts" field (the timestamp), but I want to paginate the results by 10,20,... (wherever the user sets "pageSize").
How I can do this? since saving the whole result in a temp table (and using Tabledata.list API) is not an option 'cause I would have a temp table for each different request (different in terms of filters...)
Thanks in advance!!!
For Nodejs the library automatically paginates using the token from previous request for you as described here https://cloud.google.com/bigquery/docs/paging-results#bigquery-paging-nodejs
on the other hand you have some options for manual pagination if you read the implementation code here: https://github.com/googleapis/nodejs-bigquery/blob/master/src/index.ts
Related
I add some data to firestore which contain #serverstamp, but after I add data , I need to get #serverstamp for other progress
If you're using server side generated timestamps you'll need to query the created document to read the actual timestamp value. Nonetheless, if you need the timestamp maybe it would be more practical to generate the timestamp client side, so the document read isn't needed.
I have a query like;
SELECT
*
INTO [documentdb]
FROM
[iothub]
TIMESTAMP BY eventenqueuedutctime
I need to use * because data is dynamic and dont have specific schema. Problem is Iothub system information data is written to documentdb in this query. Is there any way to exclude Iothub system information data?
Thanks.
This is not possible currently but this will be possible in Job Compatibility Level 1.2 in near future. For now, one workaround is that you could create a post create trigger in Cosmos DB to remove this property from the document.
To answer your question, Azure stream analytics service doesn't have an in-built support for excluding columns from dynamic data (iothub information). But, we can achieve this by using UDF. Here is more info on UDF.
UDF can help us in deleting the column from input data and returning us the updated json.
There are two steps basically to achieve this:
Create a JavaScript UDF.
Go to functions from left hand side navigation (below inputs).
Click on Add --> JavaScript UDF.
Give a function alias = removeiothubinfo
keep output type - any.
copy paste following code into function definition.
function main(input) {
delete input['IoTHub'];
return input;
}
Click on Save
Update query
Go to query mode and copy paste the following query :
WITH NewInput AS
(
SELECT
udf.removeiothubinfo(iothub) AS UpdatedJson
FROM
[iothub]
)
SELECT
UpdatedJson.*
INTO
[documentdb]
FROM
NewInput
Click on Save
I suggest you to test your query before running the job by uploading a sample file containing similar structure for json.
Edited
Also, even in job compatibility level 1.2 there has been no additional functionality to achieve this. Check this out for more info.
As #chetangm said in his answer, no such filtering mechanism is supported in ASA so far. Yes, you could use create trigger in Cosmos db, however it need to be triggered in sdk code or REST API. It won't be triggered automatically.
I provide you with another workaround that using Azure Function Cosmos DB Triggered. It could be executed when data is added to or changed in Azure Cosmos DB. You just need to remove the fields you don't want in the function code.
How to get an activity by its id (unique uuid) or by foreign_id + time?
I could not find it in documentation. All information there represents how to get feed in pages. Not a single activity.
If you save the id you get from adding an activity, then you can opt to fetch activities based on the id, using "id_gte" or "id_lte" and only fetch with offset 0 and limit 1. Such as:
$feed->getActivities(0, 1, ['id_gte' => $id]);
This code is based on php, but their sdk should have equal functions for other languages if you require.
The general goal should be to use Stream as a secondary data store. Your proprietary data from your customers should always be accessible in your own primary data store: most likely a RDBMS like PostgreSQL. When new follow relationships get created, or new activities get added, you should store them locally, and replicate the data to GetStream. Then access feeds when a users wants to see a timeline or notification feed, and complement your data from data found in your own DB (for example: comments, likes, author information, ...)
For this reason, there is no getActivity(uuid) method available.
Using the python client you can get an activity by its ID
import stream
client = stream.connect('YOUR_API_KEY', 'API_KEY_SECRET')
client.get_activities(ids=[activity_id])
Or by its foreign_id + time
client.get_activities(foreign_id_times=[
(foreign_id, activity_time),
])
Taken from https://github.com/GetStream/stream-python/blob/main/README.md
HI I'm new to cloudant (and couch and asking questions on stackoverflow so I hope I manage to be vaguely clear about what I'm asking ) and I'm trying to do probably the second most basic geo task but am hitting a dead end.
I've got a database of docs which are geojson objects, I've created an index so I can query for intersections etc but it seems the only options I have in the url is the format=legacy (gives me the ids) and the format=geojson and the include_docs parameter - what I'd like to do is give back a particular view of the result set - I'm not interested in the geometry of the object (which is a big lump of data and it's likely that a number of other properties may be in the document that I'd rather filter out)
is there a correct way to do this in a single api call or do I need to fetch the doc ids (legacy format) and then issue a second query to bring back my chosen 'view' for each document id given in the result of format=legacy response
Thanks
When Importing data to an Index in Azure Search, from SQL (progrematically not through the interface), Is there a way to add Query to filter the data come from the SQL table ?
Looking at the REST API documentation for Create Data Source, as of today it is not possible to define a query to filter the data that populates an index.
However I read somewhere that you can create a View and use that as the data source for populating the index. However when using a view, you will not be able to use SQL Integrated change tracking for change / deletion detection. However, you will still be able to use High Water Mark change detection and Soft Delete Column deletion detection.
Also, please vote for this UserVoice suggestion to request adding support for query parameter.