I use uuid for just about every ID in my REST backend powered by node and postgres. I also plan to use validate.js to make sure the queries are formatted correctly.
In order to shorten the URLS for my application, I would like to convert all UUIDS used by my backend into URL safe strings when exposed to the REST consumer.
The problem is that, as far as I can tell there is no such setting within node-pg. And node-pg usually returns the query results as JSON objects using either strings or numbers. That makes it hard to autmatically convert them.
I could of course just go through every single rest endpoint and add code that automatically converts all the types where I know a UUID would be. But that would violate DRY and also be a hotbed for bugs.
I also could try to automatically detect strings that look like UUIDs and then just convert them, but that also seams like it may introduce lots of bugs.
One ideal solution would be some sort of custom code injection into node-pg that automatically converts uuids. Or maybe just some pg function I could use to automatically convert the uuids within the pg-queries themselves (although that would be a bit tedious).
Another Ideal solution might be some way to use validate.js to convert the outputs and inputs during the validation. But I don't know how I could do this.
So basically, what would be a good way to autmatically convert uuids in node-pg to url safe (shorter) strings without having to add a bit of code to every single endpoint?
I think this is what I want: https://github.com/brianc/node-pg-types
It lets me set custom converters for each datatype. The input will probably still have to be converted manually
Related
I am using mongodb and what I am trying to do is query a collection, and receive well filtered and ordered and projected result. Sadly the logic I want to implement is complex and even if we assume that it is possible to use db.collection.aggregate it will result in long, complex, hard to read aggregate descriptor, which I believe in most cases is unwanted.
So I was thinking - Mongodb understands javascript, therefor most likely I can pass a javascript function during the query itself, expecting that my mongo server will make the query, run the provided function passing the query result to it, then return the final result to me. Something like:
db.collection.find(myQuery, serverCallback).toArray(function(err, db) { ... });
Sadly it seems this is impossible. Investigating further I reached stored javascript and understood that I can define that serverCallback on the server instead of passing it. Which is good, but seems messy and wrong to me. So basically this is the reason, why i decided to ask here if someone with better mongodb experience can argument this approach.
[My understandings]
I beleive that not every case of filtering, aggregating, etc. can be achieved with db.collection.aggregate, which is pretty normal. For all the cases that need special way of filtering the query result, we have two options - to define stored javascript that we execute on the query result on the mongo server, or to fetch the information from the server and do the processing/filtering/etc. in the client.
If we choose to define stored javascript, it is very likely that we will define some project specific logic into the mongo server. I think that the project specifics should always belong to the project code instead of the database. That way we can version them with git and easily access them if we want to change them.
If we choose to apply the aggregation logic after the query we loose the capability to choose who will make the calculations - the server or the client. Which may be an opinion that we want to have.
[My question]
What is the reasoning behind not allowing serverCallback to be provided during the query? I believe that there must be reasons that I do not understand here.
[Edits]
First I want to say that I have resolve my problem and as it was way too complex to explain it easily. I will prefer to stick to something easier to explain and understand. I believe this example of MongoDB stored javascript provides great example so lets use it. Basically what i tried to ask above was is there a way to pass this sum function during db.collection.find (and why there isn't any). Something like this:
function sum(queryResultAsArray) {
//Do whatever we want with queryResultAsArray
//For the example we filter result rows with x + y == 6;
return queryResultAsArray.filter(function(row) {
return row.x + row.y == 6
});
}
db.test.find({}, queryResultAsArray);
And this to be equal to the examples:
db.test.find({$where: "sum(this.x, this.y) == 6"});
For reasoning on why would one can prefer passing function rather than stored javascript see the original post.
I work with a tool that contains everything within XML inside the database.
Some reports that are stored in the database use a third party tool to load, and store the main data to configure the 'report' definition in what is not a human-readable format.
I'd post it here, but it's some 130,000 bytes.
I have attempted to decode it using popular methods that I assumed it would have been encoded in, such as base64, base 32, etc, but none have been able to decode the string.
Is there a way to identify what encoding a given string has, using a tool available online?
I don't have the benefit of access to the developer that built this functionality, the source code generating this string, or any documentation on it.
To give some context around what I'm trying to do - I need to reverse-engineer how a specific definition in a system is generated, so that it can be modified slightly (manually) in a text editor to support an operation that would otherwise require manually re-creating the report.
I apologize is if this may be the wrong exchange site for this question - I realize it's not specific to a 'programming' issue and I haven't tried to solve it using a programming language. If so - please redirect me to the appropriate place and I'll be happy to ask there instead.
Update: The text consists of strictly A-Z, 0-9 characters.
You can check amongst known encoding formats with this tool only if you are sure data is not encrypted
I use MEAN stack to develop an application.
I'm trying to develop a restful API to get users by first name or lastname
Should I write one get function to get the users for both firstname and lastname?
What is the best practice to write the URL to be handled by the backend?
Should I use the following?
To get user by firstname: localhost:3000/users?firstname=Joe
To get user by name:localhost:3000/users?firstname=Terry
And then check what is the parameter in my code and proceed.
In other words,What is the best practice if I want to pass one of multiple parameters to restful API and search by only one parameter?
Should I use content-location header?
There is no single best practice. There are lots of different ways to design a REST interface. You can use a scheme that is primarily path based such as:
http://myserver.com/query/users?firstname=Joe
Or primarily query parameter based:
http://myserver.com/query?type=users&firstname=Joe
Or, even entirely path based:
http://myserver.com/query/users/firstname/Joe
Only the last scheme dictates that only one search criteria can be passed, but this is likely also a limiting aspect of this scheme because if you, at some time in the future, want to be able to search on more than one parameter, you'd probably need to redesign.
In general, you want to take into account these considerations:
Make a list of all the things you think your REST API will want to do now and possibly in the future.
Design a scheme that anticipates all the things in your above list and feels extensible (you could easily add more things on to it without having to redesign anything).
Design a scheme that feels consistent for all of the different things a client will do with it. For example, there should be a consistent use of path and query parameters. You don't want some parts of your API using exclusively path segments and another part looking like a completely different design that uses only query parameters. An appropriate mix of the two is often the cleanest design.
Pick a design that "makes sense" to people who don't know your functionality. It should read logically and with a good REST API, the URL is often fairly self describing.
So, we can't really make a concrete recommendation on your one URL because it really needs to be considered in the totality of your whole API.
Of the three examples above, without knowing anything more about the rest of what you're trying to do, I like the first one because it puts what feels to me like the action into the path /query/users and then puts the parameters to that action into the query string and is easily extensible to add more arguments to the query. And, it reads very clearly.
There are clearly many different ways to successfully design and structure a REST API so there is no single best practice.
I need to save a User model, something like:
{ "nickname": "alan",
"email": ...,
"password":...,
...} // and a couple of other fields
Today, I use a Set: users
In this Set, I have a member like user:alan
In this member I have the hash above
This is working fine but I was just wondering if instead of the above approach that could make sense to use the following one:
Still use users Set (to easily get the users (members) list)
In this set only use a key / value storage like:
key: alan
value : the stringify version of the above user hash
Retrieving a record would then be easier (I will then have to Parse it with JSON).
I'm very new to redis and I am not sure what could be the best. What do you think ?
You can use Redis hashes data structure to store your JSON object fields and values. For example your "users" set can still be used as a list which stores all users and your individual JSON object can be stored into hash like this:
db.hmset("user:id", JSON.stringify(jsonObj));
Now you can get by key all users or only specific one (from which you get/set only specified fields/values). Also these two questions are probably related to your scenario.
EDIT: (sorry I didn't realize that we talked about this earlier)
Retrieving a record would then be easier (I will then have to Parse it with JSON).
This is true, but with hash data structure you can get/set only the field/value which you need to work with. Retrieving entire JSON object can result in decrease of performance (depends on how often you do it) if you only want to change part of the object (other thing is that you will need to stringify/parse the object everytime).
One additional merit for JSON over hashes is maintaining type. 123.3 becomes the string "123.3" and depending on library Null/None can accidentally be casted to "null".
Both are a bit tedious as that will require writing a transformer for extracting the strings and converting them back to their expected types.
For space/memory consumption considerations, I've started leaning towards storing just the values as a JSON list ["my_type_version", 123.5, null , ... ] so I didn't have overhead of N * ( sum(len(concat(JSON key names))) which in my case was +60% of Redis's used memory footprint.
bear in mind: Hashes cannot store nested objects, JSON can do it.
Truthfully, either way works fine. The way you store it is a design decision you will need to make. It depends on how you want to retrieve the user information, etc.
In terms of performance, storing the JSON encoded version of the user object will use less memory and take less time for storage/retrieval. That is, JSON parsing is probably faster than retrieving each field from Redis. And, even if not, it is probably more memory efficient. The difference in performance is probably minimal anyway.
I looked at tinyurl, tinypic, imgur and youtube! I thought they would use a text safe representation of a index and use it as a primary ID in their DB. However trying to put the keys into Convert.FromBase64String("key") yields no results and throw an exception. So these sites dont use a base64 array. What are they using? What might i want to use if i were to do a youtube like site or tinyurl?
im guessing they have developed their own encoding which is simply an alphanumeric equivalent of the ids in their database. im sure they dont generate random strings simply because this will cause catastrophic overflows at a certain point
I don't know about TinyURL, Tinypic, etc. but shorl.com uses something called koremutake.
If I were to develop such a system, I guess some sort of short hash or plain random strings could be possible choices.
My guess is that they simply generate a random string and use that as the primary key. I don't really see a reason to do anything else.