Redis: How to save (and read) Key-Value pairs at once by namespace/rule? - node.js

I want to utilize Redis for saving and reading a dynamic list of users.
Essentially, Redis is Key-Value pair storage. How can I read all the saved users at once? (for example, creating a namespace "users/user_id")
And since I am a Redis beginner,
Do you think the use of Redis in the above case is proper/efficient?
Thanks.

When using key/values for storing objects you should create a domain specific key by combining the domain name plus the unique id. For example, a user object that might look like this:
// typical user data model
var User = function(params) {
if (!params) params = {};
this.id = params.id;
this.username = params.username;
this.email = params.email;
// other stuff...
};
Then domain key could be created like this:
var createUserDomainKey = function(id) {
return 'User:' + id;
};
If the id was 'e9f6671440e111e49f14-77817cb77f36' the key would be this:
User:e9f6671440e111e49f14-77817cb77f36
Since redis will store string values, you need to serialize, probably with json so to save the user object. Assuming a valid use object would would do something like this:
var client = redis.createClient(),
key = createUserDomainKey( user.id ),
json = JSON.stringify( user ) ;
client.set( key, json, function(err, result) {
if (err) throw err; // do something here...
// result is 'OK'
});
For simple fire-hose queries returning all users, you can do this:
var client = redis.createClient();
client.keys( createUserDomainKey( '*' ), function(err, keys) {
if (err) throw err; // do something here
// keys contains all the keys matching 'User:*'
});
Note that the redis folks discourage the use of 'keys' for production, so a better approach is to build your own index using sorted-set, but if your user list is limited to a few hundred, there is no problem.
And since it returns a list of keys, you need to loop through and get each user then parse the json string to recover the real object. Assuming a populated list of keys, you could do something like this:
var client = redis.getClient(),
users = [];
var loopCallback = function(err, value) {
if (!err && value) {
// parse and add the user model to the list
users.push( JSON.parse( value ) );
}
// pull the next key, if available
var key = keys.pop();
if (key) {
client.get( key, loopCallback );
} else {
// list is complete so return users, probably through a callback;
}
}
// start the loop
loopCallback();
This is a good general purpose solution, but there are others that use sorted sets that are move efficient when you want access to the entire list with each access. This solution gives you the ability to get a single user object when you know the ID.
I hope this helps.
ps: a full implementation with unit tests of this can be found in the AbstractBaseDao module of this project.

Related

How to set push key when pushing to firebase database?

When I write data to firebase database from both frontend(Angular 4) and backend(firebase functions), there is a push key generated by firebase. With this key, I cannot access data in the future because the key is unique. I am wondering is any way I can set the key myself or I can access the data without knowing the key?
Here is my code from frontend:
this.db.list(`${this.basePath}/`).push(upload);
Here is my code from backend:
admin.database().ref('/messages').push({original: original}).then(function (snapshot) {
res.redirect(303, snapshot.ref);});
All data I pushed will be under path/pushID/data
I cannot access data without knowing the pushID.
The best case I want is path/my own pushID/data
Thanks so much for help!!
If you want to loop through all messages:
var ref = firebase.database().ref("messages");
ref.once("value", function(snapshot) {
snapshot.forEach(function(message) {
console.log(message.key+": "+message.val().original);
});
});
If you want to find specific messages, use a query:
var ref = firebase.database().ref("messages");
var query = ref.orderByChild("original").equalTo("aaaa");
query.once("value", function(snapshot) {
snapshot.forEach(function(message) {
console.log(message.key+": "+message.val().original);
});
});
For much more on this, read the Firebase documentation on reading lists of data, sorting and filtering data, and take the Firebase codelab.
The keys should be unique in any way. You can set your own key like this instead of push
admin.database().ref('/messages/'+ yourUniqueId).set({original: original}).then(function (snapshot) {
res.redirect(303, snapshot.ref);});
yourUniqueId can be auth uid or email of user, like something unique.

How to delete on based of subkey using hset in redis in node js app

I am trying to use redis in my nodejs application. I am able to save and retrieve value perfectly. I am using hash which allows me to have single unique key as well sub-keys. Here offer id is my sub key and offer is key.
I am saving offer like below
let offer = {
offerId: req.query.id,
offerName: req.query.offerName,
offerVendor: req.query.offerVendor
}
client.hset('offer', offer.offerId, JSON.stringify(offer), redis.print);
and retrieving value like below
try {
console.log('ping response:', await client.ping());
const offerId = req.query.id;
client.hget('offer', offerId, function(err, reply){
let offer = JSON.parse(reply);
res.send(offer);
});
} catch(err) {
console.log('ping error:', err);
}
I want to delete offer based on id as like saving or getting on based of id. I have gone through documentation https://github.com/NodeRedis/node_redis#usage-example but didn't find there.
What is the way to do delete offer based on id ?
If you already have the offer id, you can delete it from the offer hash using the hdel command. Your code would look something like:
client.hdel('offer', id)
The result from the call will be an integer count of the number of keys deleted. If you don't know the client id, you can look for it using the hscan command.

Documentdb Failed to deserialize stored procedure response or convert it to my defined type

My Stored Procedure: (I created it via Azure Script Explorer)
function GetAllResources() {
var collection = getContext().getCollection();
// Query documents and take 1st item.
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT * FROM MultiLanguage as m',
function (err, docs, options) {
if (err) throw err;
// Check the feed and if empty, set the body to 'no docs found',
// else take 1st element from feed
if (!docs || !docs.length) getContext().getResponse().setBody('no docs found');
else getContext().getResponse().setBody(JSON.stringify(docs));
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
The sproc can be executed successfully from script explorer.
My C# code to call the sproc:
public async Task<IHttpActionResult> GetReources() {
client = new DocumentClient(new Uri(ConfigurationManager.AppSettings["endpoint"]), ConfigurationManager.AppSettings["authKey"]);
var collectionLink = UriFactory.CreateDocumentCollectionUri(DatabaseId, CollectionId);
//var docs = await client.ReadDocumentFeedAsync(collectionLink, new FeedOptions { MaxItemCount = 10 });
//var docs = from d in client.CreateDocumentQuery<Models.Resource>(collectionLink)
// select d;
StoredProcedure storedProcedure = client.CreateStoredProcedureQuery(collectionLink).Where(c => c.Id == "GetAllResources").AsEnumerable().FirstOrDefault();
Models.Resource docs = await client.ExecuteStoredProcedureAsync<Models.Resource>(storedProcedure.SelfLink);
foreach (var d in docs) {
Models.Resource a = new Models.Resource();
a = docs;
//a.id = d.id;
//a.Scenario = d.Scenario;
//a.Translations = d.Translations;
//a.LastModified = d.LastModified;
//a.ModifiedBy = d.ModifiedBy;
//a.LastAccessed = d.LastAccessed;
resources.Add(a);
}
return Ok(resources);
}
First, there is an error for the "foreach..." like said
foreach cannot operate on variables of type Models.Resource because it
doesn't contain a public definition of GetEnumerator.
Then I tried to modify my sproc to only return 1 result and remove the foreach line, then I got error said
Failed to deserialize stored procedure response or convert it to type
'Models.Resource'
I just want to return the result of the stored procedure as my defined class (Models.Resource). How to do this?
It can be simpler to get sproc by name using CreateStoredProcedureUri, like this:
const string endpoint = "https://your.service.azure.com:443/";
const string authKey = "<your magic secret master key>==";
var client = new DocumentClient(new Uri(endpoint), authKey);
Uri sprocUri = UriFactory.CreateStoredProcedureUri("databaseName", "collectionName", "GetAllResources");
var result = await client.ExecuteStoredProcedureAsync<string>(sprocUri);
The stored procedure above serializes results of the query (docs array) to string, if you keep it this way, the result of sproc would be string, which I guess you would need to manually deserialize to objects. You can do this simpler, just return docs from sproc and have result as objects (like Models.Resource[]), serialization would happen automatically.
If you change the sproc to return just one doc (e.g. do __.response.setBody(docs[0]) and Models.Resource represent one item, then the call is correct:
Models.Resource doc = await client.ExecuteStoredProcedureAsync<Models.Resource>(sprocUri);
Also, to // Query documents and take 1st item, I wouldn't recommend to use script as script has overhead of running JavsScript engine. Scripts kick in when you have bulk operations (to optimize for network traffic) or have business logic which makes sense to run on the server. To take 1st item you can do query from client like this: SELECT TOP 1 * FROM c. Typically you would WHERE and ORDER BY clause to that.
There is a number of docdb samples on github, for instance, https://github.com/Azure/azure-documentdb-dotnet/tree/master/samples/code-samples/ServerSideScripts and https://github.com/Azure/azure-documentdb-dotnet/tree/master/samples/code-samples/Queries.
Thanks,
Michael
All right, let's make sure we are on the same page.
I am using the sproc same as above.
I am using client code like this:
class Models
{
// This would have more properties, I am just using id which all docs would have.
public class Resource
{
[JsonProperty("id")]
public string Id { get; set; }
}
}
public async Task<IHttpActionResult> GetResources()
{
const string endpoint = "https://myservice.azure.com:443/";
const string authKey = "my secret key==";
var client = new DocumentClient(new Uri(endpoint), authKey);
Uri sprocUri = UriFactory.CreateStoredProcedureUri("db", "c1", "GetAllResources");
var serializedDocs = await client.ExecuteStoredProcedureAsync<string>(sprocUri);
Models.Resource[] resources = JsonConvert.DeserializeObject<Models.Resource[]>(serializedDocs);
return Ok(resources);
}
It works fine. Is this what you are doing?

How to store documents in an ArangoDb graph using ArangoJs?

I am using latest version of ArangoDb and ArangoJs from a nodejs application. I have got following two vertexes
users
tokens
tokens vertex contain the security tokens issues to one of the user in the users vertex. I have got an edge definition named token_belongs_to connecting tokens to users
How do I store a newly generated token belonging to an existing user using ArangoJs?
I am going to assume you are using ArangoDB 2.7 with the latest version of arangojs (4.1 at the time of this writing) as the API has changed a bit since the 3.x release of the driver.
As you don't mention using the Graph API the easiest way is to simply use the collections directly. Using the Graph API however adds benefits like orphaned edges automatically being deleted when any of their vertices are deleted.
First you need to get a reference to each collection you want to work with:
var users = db.collection('users');
var tokens = db.collection('tokens');
var edges = db.edgeCollection('token_belongs_to');
Or if you are using the Graph API:
var graph = db.graph('my_graph');
var users = graph.vertexCollection('users');
var tokens = graph.vertexCollection('tokens');
var edges = graph.edgeCollection('token_belongs_to');
In order to create a token for an existing user, you need to know the _id of the user. The _id of a document is made up of the collection name (users) and the _key of the document (e.g. 12345678).
If you don't have the _id or _key you can also look up the document by some other unique attribute. For example, if you have a unique attribute email that you know the value of, you could do this:
users.firstExample({email: 'admin#example.com'})
.then(function (doc) {
var userId = doc._id;
// more code goes here
});
Next you'll want to create the token:
tokens.save(tokenData)
.then(function (meta) {
var tokenId = meta._id;
// more code goes here
});
Once you have the userId and tokenId you can create the edge to define the relation between the two:
edges.save(edgeData, userId, tokenId)
.then(function (meta) {
var edgeId = meta._id;
// more code goes here
});
If you don't want to store any data on the edge you can substitute an empty object for edgeData or simply write it as:
edges.save({_from: userId, _to: tokenId})
.then(...);
So the full example would go something like this:
var graph = db.graph('my_graph');
var users = graph.vertexCollection('users');
var tokens = graph.vertexCollection('tokens');
var edges = graph.edgeCollection('token_belongs_to');
Promise.all([
users.firstExample({email: 'admin#example.com'}),
tokens.save(tokenData)
])
.then(function (args) {
var userId = args[0]._id; // result from first promise
var tokenId = args[1]._id; // result from second promise
return edges.save({_from: userId, _to: tokenId});
})
.then(function (meta) {
var edgeId = meta._id;
// Edge has been created
})
.catch(function (err) {
console.error('Something went wrong:', err.stack);
});
Attention - syntax changes:
Edge creation:
const { Database, CollectionType } = require('arangojs');
const db = new Database();
const collection = db.collection("collection_name");
if (!(await collection.exists())
await collection.create({ type: CollectionType.EDGE_COLLECTION });
await collection.save({_from: 'from_id', _to: 'to_id'});
https://arangodb.github.io/arangojs/7.1.0/interfaces/_collection_.edgecollection.html#create

generate random unique token for user id

I want to generate token as user id and store in database , but how to generate unique one?
should I add timestamp var currentUnixTimestamp = (new Date().getTime() / 1000); as salt? how to do with crypto?
var generateToken = function() {
return new Promise(function (fulfill, reject){
crypto.randomBytes(8, function(error, buf) {
if (error) {
reject(error);
} else {
var token = buf.toString('hex');
fulfill(token);
}
});
});
};
Eight random bytes from a properly seeded crypto library has a low chance of a collision, so you don't usually need to concern yourself with duplicates. In fact, increase that to 16 bytes, and your code is on par with UUID version 4. This is considered a standard for UUIDs. The chances of a collision are so remote it is not usually worth considering.
If you are going that far though, consider using a standard format UUID, such as the node package "uuid". There are also database-side uuid functions which you can add as default to schemas e.g. in Postgres. The advantage is a standardised and well-understood format for your ids, and you won't need to spend any time justifying or maintaining your code for this, just point developers to the standard docs.
If you want this token for authentication purposes you should use json web token instead. It will manage for you and its quite efficient.
Only have to include as a middleware .
app.use(expressJWT({
secret: new Buffer("Your-secret-key").toString('base64')
}).unless({
//# pass api without validating
path: unlessRoutes
}));
You could specify which routes you don't want to to skip in jwt middleware by giving an array in unlessRoutes.
var unlessRoutes = [
'/',
/\/login/,
/\/register/,
/\/customers/,
/\/customer$/,
/\/addCustomer/,
/\/just/,
/\/search/,
/\/dynamic/,
/\/favicon.ico/
]
This is what i think we can do for generating the random token using the crypto:
var passwordResetToken = createRandomToken(data.body.email);
exports.createRandomToken = function (string) {
var seed = crypto.randomBytes(20);
return crypto.createHash('abcde').update(seed + string).digest('hex');
};

Resources