I am trying to get using the Azure Backend with Sync to work for my app.
It looks like PullAsync is not populating my local table though.
My Table is set up like this:
private IMobileServiceSyncTable<Familie> FamilienTable = App.MobileService.GetSyncTable<Familie>(); // offline sync
Some code first. Initializing the table like this:
if (!App.MobileService.SyncContext.IsInitialized)
{
var store = new MobileServiceSQLiteStore("localGarden.db");
store.DefineTable<Familie>();
await App.MobileService.SyncContext.InitializeAsync(store);
}
I can see that the table is created and it is on my local system later (already tried setting up a new table, too).
And a little later in my code this is called:
try
{
await FamilienTable.PullAsync(null, FamilienTable.CreateQuery());
}
catch (Exception ex)
{
errorString = "Pull failed: " + ex.Message +
"\n\nIf you are still in an offline scenario, " +
"you can try your Pull again when connected with your Mobile Serice.";
}
if (errorString != null)
{
MessageDialog d = new MessageDialog(errorString);
await d.ShowAsync();
}
I set this up originally based on https://azure.microsoft.com/en-us/documentation/articles/mobile-services-xamarin-ios-get-started-offline-data/ and adapted to using a Win10 App. This is using Microsoft.Azure.Mobile.Client.SQLitestore 2.0.1 mainly.
I know that I need to replace the null in the PullAsync with a string for incremental updates in the future.
Using fiddler I discovered that there are 2 calls to my node.js powered API on Azure during the pullAsync:
[removed].azurewebsites.net/tables/Familie?$skip=0&$top=50&__includeDeleted=true (which returns a JSON with 6 rows I have in my table) and immidiatly after that [removed].azurewebsites.net/tables/Familie?$skip=6&$top=50&__includeDeleted=true which returns an empty JSON.
My local table stays empty though.
Can someone tell me, if that behavior is by design, explain why I get 2 calls and give me an idea where to look for the reason why my local table is not populated?
Thank you very much!
Additional Info: Output of LoggingHandler:
CREATE TABLE IF NOT EXISTS [Familie] ([id] INTEGER PRIMARY KEY, [Name] TEXT, [Deleted] BOOLEAN, [Version] DATETIME, [createdAt] DATETIME, [updatedAt] DATETIME)
CREATE TABLE IF NOT EXISTS [__operations] ([id] TEXT PRIMARY KEY, [kind] INTEGER, [state] INTEGER, [tableName] TEXT, [tableKind] INTEGER, [itemId] TEXT, [item] TEXT, [createdAt] DATETIME, [sequence] INTEGER, [version] INTEGER)
CREATE TABLE IF NOT EXISTS [__errors] ([id] TEXT PRIMARY KEY, [httpStatus] INTEGER, [operationVersion] INTEGER, [operationKind] INTEGER, [tableName] TEXT, [tableKind] INTEGER, [item] TEXT, [rawResult] TEXT)
CREATE TABLE IF NOT EXISTS [__config] ([id] TEXT PRIMARY KEY, [value] TEXT)
BEGIN TRANSACTION
INSERT OR IGNORE INTO [__config] ([id]) VALUES (#p0)
The thread 0x2fcc has exited with code 0 (0x0).
UPDATE [__config] SET [value] = #p0 WHERE [id] = #p1
COMMIT TRANSACTION
SELECT * FROM [__operations] ORDER BY [sequence] DESC LIMIT 1
SELECT COUNT(1) AS [count] FROM [__operations]
Pulling changes from remote server
'Gemüsebeetplaner.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Users\Jens\documents\visual studio 14\Projects\Gemüsebeetplaner\Gemüsebeetplaner\bin\x86\Debug\AppX\System.Linq.Queryable.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
SELECT * FROM [__operations] WHERE ([tableName] = #p1) LIMIT 0
SELECT COUNT(1) AS [count] FROM [__operations] WHERE ([tableName] = #p1)
Pulling changes from remote server
SELECT * FROM [__operations] WHERE ([tableName] = #p1) LIMIT 0
SELECT COUNT(1) AS [count] FROM [__operations] WHERE ([tableName] = #p1)
SELECT * FROM [Familie] ORDER BY [Name]
Additonal Info 2:
**Screenshots of the tables **
Screenshot
This is the log with parameters enabled:
Pulling changes from remote server
SELECT * FROM [__operations] WHERE ([tableName] = #p1) LIMIT 0
#p1:Familie
SELECT COUNT(1) AS [count] FROM [__operations] WHERE ([tableName] = #p1)
#p1:Familie
{
"count": 0
}
Pulling changes from remote server
SELECT * FROM [__operations] WHERE ([tableName] = #p1) LIMIT 0
#p1:Familie
SELECT COUNT(1) AS [count] FROM [__operations] WHERE ([tableName] = #p1)
#p1:Familie
{
"count": 0
}
SELECT * FROM [Familie] ORDER BY [Name]
The thread 0x24c8 has exited with code 0 (0x0).
The thread 0x1474 has exited with code 0 (0x0).
The thread 0x1a40 has exited with code 0 (0x0).
Class
public class Familie
{
[JsonProperty(PropertyName = "Id")]
public int Id { get; set; }
[JsonProperty(PropertyName = "Name")]
public string Name { get; set; }
}
There will always be at least 2 calls when doing a Pull operation, because the client has no way of knowing the server batch size. In your example, the server sent 6 records, so the client needs to do another query to find out if there are more records
What is the symptom of your local table not being populated? Do you get empty results when you do a query of FamilienTable?
Does the first call return results? If so, there must be something strange in how the local database is being populated. Try adding logging SQLite store to your app, which will log all local database statements, here's a sample: https://github.com/Azure-Samples/app-service-mobile-dotnet-todo-list-files/blob/master/src/client/MobileAppsFilesSample/Helpers/LoggingHandler.cs
Related
We have a facility in our web app to delete large quantities of data. We do this by paginating through all records found against u_id.
The keys we have are designed for other queries we have in the application - ideally, it would be great to have a primary key for u_id but this would break all our other queries.
The below method works well most of the time, however, after deleting approximately 6-8 million of records, we get:
Dse\Exception\RuntimeException: All connections on all I/O threads are busy
We also sometimes get a slightly different error message:
Dse\Exception\ReadTimeoutException: Operation timed out - received only 0 responses
You'll notice in the below code usleep(2500000) which pauses the script. This has been our workaround but would be good to get this resolved as Cassandra should be able to handle this number of deletes.
$cluster = \Dse::cluster()
->withDefaultTimeout(3600)
->withContactPoints(env('CA_HOST'))
->build();
$session = $cluster->connect(env('CONNECT'));
$options = array('page_size' => 50);
$results = $session->execute("SELECT * FROM datastore WHERE u_id = $u_id;", $options);
$future_deletes = array();
while (true) {
foreach ($results as $result) {
$future_deletes[] = $session->executeAsync("DELETE FROM datastore WHERE record_id = '" . $result['record_id'] . "' AND record_version = " . $result['record_version'] . " AND user_id = " . $result['user_id']);
$future_deletes[] = $session->executeAsync("UPDATE data_count set u_count = u_count - 1 WHERE u_id = " . $u_id);
}
if( !empty($future_deletes) ){
foreach ($future_deletes as $future_delete) {
// we will not wait for each result for more than 5 seconds
$future_delete->get(5);
}
//usleep(2500000); //2.5 seconds
}
$future_deletes = array();
if ($results->isLastPage()) {
break;
}
$results = $results->nextPage();
}
//Disconnect
$session = NULL;
For your reference, here are our tables:
CREATE TABLE datastore (id uuid,
record_id varchar,
record_version int,
user_id int,
u_id int,
column_1 varchar,
column_2 varchar,
column_3 varchar,
column_4 varchar,
column_5 varchar,
PRIMARY KEY((record_id), record_version, user_id)
);
CREATE INDEX u_id ON datastore (u_id);
CREATE TABLE data_count (u_id int PRIMARY KEY, u_count counter);
We are running a server with 8GB RAM.
The version of the DSE driver is 6.0.1.
Thank you in advance!
You need to control, how many "in-flight" requests do you have a the same point of time. There is a limit on number of queries per connection, and number of connections. They are controlled by corresponding functions of the Cluster class (can't find fast enough in PHP docs, but it should be similar to Cluster functions in the C++ driver, because PHP is built on top of C++ driver).
SELECT count(*) FROM device_stats
WHERE orgid = 'XYZ'
AND regionid = 'NY'
AND campusid = 'C1'
AND buildingid = 'C1'
AND floorid = '2'
AND year = 2017;
The above CQL query returns correct result - 32032, in CQL Shell
But when I run the same query using QueryBuilder Java API , I see the count as 0
BuiltStatement summaryQuery = QueryBuilder.select()
.countAll()
.from("device_stats")
.where(eq("orgid", "XYZ"))
.and(eq("regionid", "NY"))
.and(eq("campusid", "C1"))
.and(eq("buildingid", "C1"))
.and(eq("floorid", "2"))
.and(eq("year", "2017"));
try {
ResultSetFuture tagSummaryResults = session.executeAsync(tagSummaryQuery);
tagSummaryResults.getUninterruptibly().all().stream().forEach(result -> {
System.out.println(" totalCount > "+result.getLong(0));
});
I have only 20 partitions and 32032 rows per partition.
What could be the reason QueryBuilder not executing the query correctly ?
Schema :
CREATE TABLE device_stats (
orgid text,
regionid text,
campusid text,
buildingid text,
floorid text,
year int,
endofwindow timestamp,
categoryid timeuuid,
devicestats map<text,bigint>,
PRIMARY KEY ((orgid, regionid, campusid, buildingid, floorid,year),endofwindow,categoryid)
) WITH CLUSTERING ORDER BY (endofwindow DESC,categoryid ASC);
// Using the keys function to index the map keys
CREATE INDEX ON device_stats (keys(devicestats));
I am using cassandra 3.10 and com.datastax.cassandra:cassandra-driver-core:3.1.4
Moving my comment to an answer since that seems to solve the original problem:
Changing .and(eq("year", "2017")) to .and(eq("year", 2017)) solves the issue since year is an int and not a text.
Folks,
I would like to solve the following with one table in Cassandra. Said service tracks when users open an asset. On subsequent events to the same asset, we simply over-write the accessDate.
example record:
{ userId: "string", assetId: "string", accessDate: unixTimestamp }
With this said, we need to fulfill the following access requirements (each requirement has its own bulletpoint for readability):
Be able to return all assets a user has opened, and at what time.
This is easy to achieve, table could look like:
CREATE TABLE user_assets_tracker (
userId uuid,
accessDate timestamp,
assetId uuid,
PRIMARY KEY (userid, accessDate, assetId)
);
This allows us to query for all assets, and when each was last accessed.
SELECT *
FROM user_assets_tracker
WHERE userId = 522b1fe2-2e36-4cef-a667-cd4237d08b89
ORDER BY accessDate DESC;
>
Dandy. Now the harder bits, which I am unsure about, was hoping you folks could chime in:
Show me all the assets user added in the past 30 days.
Naturally the LIMIT here is not what we need. Also, we may need to have 2 tables to achieve this.
SELECT *
FROM user_assets_tracker
WHERE userid = 522b1fe2-2e36-4cef-a667-cd4237d08b89
ORDER BY accessDate DESC;
LIMIT 10; ?????
Show me the last accessed item for the user. I think this one is easier, the LIMIT 1 solves that.
This is probably straight forward, with this schema:
CREATE TABLE user_assets_tracker (
userId uuid,
accessDate timestamp,
assetId uuid,
PRIMARY KEY (userid, accessDate, assetId)
);
SELECT *
FROM user_assets_tracker
WHERE userid = 522b1fe2-2e36-4cef-a667-cd4237d08b89
ORDER BY accessDate DESC;
LIMIT 1;
Retrieve the full record for a particular userId + assetId
Since accessDate comes before assetId in our schema, I am not sure how to do this as well. Another table?
Thanks!!
PS It seems that SASI Index could be the solution
Though you are always selecting assetid orderby accessDate desc.
Define your schema with order by accessDate desc
CREATE TABLE user_assets_tracker (
userid uuid,
accessdate timestamp,
assetid uuid,
PRIMARY KEY (userid, accessdate, assetid)
) WITH CLUSTERING ORDER BY (accessdate DESC, assetid ASC);
Now you don't need to specify order by accessDate desc every time. it will by default order your data by accessDate desc
Show me all the assets user added in the past 30 days.
First get timestamp of 30 day ago.
Let's current timestamp of 30 day ago is : 2017-02-05 12:00:00+0000
Now you can query :
SELECT * FROM user_assets_tracker WHERE userid = 522b1fe2-2e36-4cef-a667-cd4237d08b89 AND accessdate >= '2017-02-05 12:00:00+0000'
Retrieve the full record for a particular userId + assetId
If you are using Cassandra 3.0 or above you can use Materialized Views
CREATE a Materialized View :
CREATE MATERIALIZED VIEW user_assets AS
SELECT *
FROM user_assets_tracker
WHERE userid IS NOT NULL AND assetid IS NOT NULL AND accessdate IS NOT NULL
PRIMARY KEY (userid, assetid, accessdate);
Now if you want to get all data with userid and assetid, here is the query
SELECT * FROM user_assets WHERE userid = 522b1fe2-2e36-4cef-a667-cd4237d08b89 AND assetid = 1d45e6c2-02a1-11e7-aac5-b9ab92bee74c;
Here is another thing, if huge data is inserted into a single user, you should add time bucket with userid as partition key.For more check the answer https://stackoverflow.com/a/41857183/2320144
How do I increment field of table using varchar. Here an example of what I want to have:
Mark 00001
Mark 00002
Mark 00003
Jaques 00001
Jaques 00002
Jaques 00003
Here is my example It can be useful for other people thanks to those who have helped me a lot thank you
The table is
`CREATE TABLE [dbo].[TAG_Sinistre](
[ID] [int] IDENTITY(1,1) NOT NULL,
[ref_ag] [varchar](7) NULL,
[ref_sinistre] [varchar](7) NULL,
)`
The stored procedure is
create PROC [dbo].[sp_Add_AgSinistre]
#ref_ag varchar (7)
AS BEGIN
declare #id int
DECLARE #ref_sin VARCHAR
SET #id = (SELECT ISNULL(MAX(CAST(ID AS INT)), 0) + 1
FROM TAG_Sinistre where ref_ag=#ref_ag
)
select #ref_sin=right('000000'+CAST(#ref_sin AS VARCHAR(6)),6)
BEGIN
INSERT into TAG_Sinistre(ref_ag,ref_sinistre)
VALUES (#ref_ag,#ref_sin)
Assuming the CompanyName field is UNIQUE, you could get the MAX added value, and increment that?
DECLARE #CurrentSequence INT
SET #CurrentSequence = (SELECT MAX(SequenceId)
FROM TableName WHERE CompanyName = #CompanyName)
INSERT INTO TableName
(CompanyName, SequenceId)
VALUES
(#CompanyName, #CurrentSequence+1)
Your StoredProc would pass in the Company Name as #CompanyName. Obviously, this is a naive approach, and I'm not including what would happen if multiple attempts to update the company would happen simultaneously, but there's no mention of that ion the question.
EDIT: Based on comments
DECLARE #maxSeq INT
SET #maxSeq = (SELECT ISNULL(MAX(CAST(SequenceId AS INT)), 0) + 1 FROM
TableName WHERE CompanyName = #CompanyName)
INSERT INTO TableName
(CompanyName, SequenceId)
VALUES
(#CompanyName, right('000000'+CAST(#maxSeq AS VARCHAR(7)),7))
As I said, I would look to use an INT, and get the UI to present the sequence as you want, instead of casting etc in the DB.
EDIT 2: Added Fiddle
SQL Fiddle
Assuming that your columns always look like those specified above.
DB Trigger (MS-SQL) --> Pseudocode/Untested:
CREATE TRIGGER [dbo].[YourTrigger]
ON [dbo].[YourTable]
INSTEAD OF INSERT
AS
BEGIN
DECLARE #maxVal int
-- PSEUDOCODE!!!
SELECT #maxVal = MAX(CAST(dbo.RegexReplace( '.*?(\d+)', myColumn, '$1') AS INT ))
FROM YourTable
INSERT INTO YourTable
SELECT myColumn + ' ' + #maxVal FROM inserted
END
Use insert now like this:
INSERT INTO YourTable values('Jaques')
I have the following table in cassandra;
CREATE TABLE reports (
c_date text,
c_n int,
c_id timeuuid,
report_id bigint,
report text,
PRIMARY KEY ((c_date, c_n), c_id)
)
c_date is for querying reports by date.
c_n is the number of nodes to prevent hotspots(number of nodes to distribute data evenly).
c_id is the inserted timeuuid.
My select query (cql 3) is the following;
select report, dateOf(c_id), report_id
from keyspace.reports
where c_date = '2013-08-02' and
c_n = 1 and
c_id > minTimeuuid('2013-08-02 02:52:10-0400');
I have successfully get the result set;
However, when I use cql_get_rows() function implemented on another example (here),
the timestamp (dateOf(id)) cannot be parsed correctly and bigint fields yield the following warning;
PHP Warning: unpack(): Type N: not enough input, need 4, have 0
in /home/arascan/my-project/tools/vendor/phpcassa/lib/phpcassa/Schema/DataType/LongType.php on line 47
The returned data from cql_get_rows() is the following;
[0] => Array
(
[reportid] => 281474976712782
[report] => some_report
[dateOf(c_id)] => d:1375426331.32100009918212890625;
)
How can I prevent this function throw warning and get the timestamp in date format?
(Please don't suggest # usage)