I'm using the node aws-sdk package and I need to send a custom user agent in the S3 requests in order to identify the process in the console log.
I've seen a method to do this in the Java SDK but I can't see any similar in the node package.
Is there any way to do this easily?
After browsing in the source code I found an undocumented option to set the user agent: customUserAgent
const options = { customUserAgent: 'my-process-name' };
const client = new AWS.S3(options);
You can define an agent in the httpoptions field of the options you send to the constructor as per here:
httpOptions (map) — A set of options to pass to the low-level HTTP request.
Currently supported options are:
proxy [String] — the URL to proxy requests through
agent [http.Agent, https.Agent] — the Agent object to perform HTTP requests with. Used for connection pooling. Defaults to the global agent (http.globalAgent) for non-SSL connections. Note that for SSL connections, a special Agent object is used in order to enable peer certificate verification. This feature is only available in the Node.js environment.
connectTimeout [Integer] — Sets the socket to timeout after failing to establish a connection with the server after connectTimeout milliseconds. This timeout has no effect once a socket connection has been established.
timeout [Integer] — Sets the socket to timeout after timeout milliseconds of inactivity on the socket. Defaults to two minutes (120000).
xhrAsync [Boolean] — Whether the SDK will send asynchronous HTTP requests. Used in the browser environment only. Set to false to send requests synchronously. Defaults to true (async on).
Is that what you're looking for?
Related
In the Node https module docs,
regarding https.request, an example is shown:
const options = {
hostname: 'encrypted.google.com',
port: 443,
path: '/',
method: 'GET',
key: fs.readFileSync('test/fixtures/keys/agent2-key.pem'),
cert: fs.readFileSync('test/fixtures/keys/agent2-cert.pem')
};
options.agent = new https.Agent(options);
const req = https.request(options, (res) => {
// ...
});
This example is slightly ambiguous in my opinion, and I've asked an SO question regarding this ambiguity and following a comment reaffirming the strange wording, opened an issue for this.
Regardless, I am still trying to understand the part that the Agent plays in this scenario, seeing as the https.Agent module does accept TLS connection options:
interface AgentOptions extends http.AgentOptions, tls.ConnectionOptions
The definition of the https.Agent object is:
An Agent object for HTTPS similar to http.Agent.
And the definition for the http.Agent object is:
An Agent is responsible for managing connection persistence and reuse for HTTP clients.
From this I understand that an Agent is 'in charge' of managing a connection - and clearly, the fact that https.Agent exists on top of the 'plain' http.Agent exists would imply that it is 'in charge' of managing an HTTPS connection - hence the TLS configuration options it may receive.
My question is this - does this mean that the Agent in this case has an added responsibility of configuring the network security of the requests? this is a strange API if this is true - I would have expected to see the network connection config on a separate key for the https.request (as is shown in the example after the snippet above). Why overload the same object for another responsibility? Really, why have an https.Agent at all? The http.Agent should control connection pooling and keeping connections alive, while another layer should control configuring the actual requests. The https.Agent object doesn't seem well-defined to me.
Actually the HTTPS doc points to some good resources, it also contains a link to HTTPS module source code, which reveals a lot. But to answer your questions:
does this mean that the Agent in this case has an added responsibility of configuring the network security of the requests
Yes sort of, your HTTPS agent can perform custom security feature, or even use external HTTPS implementations, but for the built-in https.Agent, the native TLS module is used, and the options you passed to constructor is ultimately passed to tls.connect, allowing you to configure custom TLS options.
Why overload the same object for another responsibility? Really, why have an https.Agent at all?
Because HTTPS runs on TLS. Actually https.Agent internally calls and constructs with http.Agent. The extra bits are mainly the use of SSL "session" over TCP "socket", if you search getName in https.Agent source and compare that to http.Agent's, you will find the SSL session cache are based on many more fields, like DHparams, client cert, etc. things that does not exist for HTTP at all.
I followed the guide of Pusher docs to create a simple example, and the code is copy from the Code examples.
This example works well when I'm at home, but it's never working when I'm in office.
In office, the HTML page can alert the message when I use the Debug Console provided by Pusher to create new Event. However, the HTML page won't alert any messages when I trigger the event from node JS.
I guess the problem is related to the network, so I want to know how to jump over the network limitation while using pusher js from node js.
var pusher = new Pusher({ // you can set other options in any of the 3 ways described above
proxy: 'HTTP_PROXY_URL', // optional, URL to proxy the requests through
timeout: TIMEOUT, // optional, timeout for all requests in milliseconds
keepAlive: KEEP_ALIVE // optional, enables keep-alive, defaults to false
});
The issue has been solved, it indeed is the network matters, and just set the proxy, then it works.
In Node.js there is a default timeout for a server (for an incoming HTTP request) at 120000ms (2 minutes) (see HTTP's server.timeout documentation).
But if I want to do an HTTP request in Node.js (using http.request), looking at the documentation, I only find a function request.setTimeout() to set the timeout manually.
Anyone know if there is a default timeout for HTTP requests in Node.js? Or does Node.js try to send the HTTP request with no end?
You want to set the server.timeout property (it defaults to 120,000, as you've found).
Update: Node.js 13 has removed the default timeout:
https://nodejs.org/api/http.html#servertimeout
https://github.com/nodejs/node/pull/27558
I was also interested in this. By reading the code, Node.js uses Socket under the hood of http request (naturally). (The source link below is referring v8.8.0 at the point when I'm writing this)
https://github.com/nodejs/node/blob/master/lib/_http_outgoing.js
And Socket does not have the timeout by default by this document
https://nodejs.org/dist/latest-v6.x/docs/api/net.html#net_socket_settimeout_timeout_callback
And the source tells the same.
https://github.com/nodejs/node/blob/master/lib/net.js
No. There is no default timeout.
Node.js uses Socket under the hood of http request and socket does not have the timeout by default.
Use the {timeout: XX} parameter of http.request to configure a proper request timeout.
I'm using Redis To Go in combination with the https://github.com/mranney/node_redis library. Redis gives me a url that looks like redis://me:978287c0b670694673d045f08b2e0371#icefish.redistogo.com:9393 but I don't know how to use it as createClient() only takes the host and the port.
I believe that the scheme for the URL you have is:
redis://username:password#host:port.
I don't believe username is used. node_redis provides two methods that you'll use to log in: createClient and auth. There are details in the readme, but for reference here is the relevant portion:
redis.createClient(port, host, options)
Create a new client connection. port defaults to 6379 and host
defaults to 127.0.0.1. If you have redis-server running on the
same computer as node, then the defaults for port and host are
probably fine. options in an object with the following possible
properties:
parser: which Redis protocol reply parser to use. Defaults to
hiredis if that module is installed. This may also be set to
javascript.
return_buffers: defaults to false. If set to true, then bulk
data replies will be returned as node Buffer objects instead of
JavaScript Strings.
createClient() returns a RedisClient object that is named client
in all of the examples here.
client.auth(password, callback)
When connecting to Redis servers that require authentication, the
AUTH command must be sent as the first command after connecting.
This can be tricky to coordinate with reconnections, the ready check,
etc. To make this easier, client.auth() stashes password and will
send it after each connection, including reconnections. callback is
invoked only once, after the response to the very first AUTH command
sent.
I also had to add the parameter no_ready_check: true to the call to redis.createClient().
client = redis.createClient(settings.redis.port,
settings.redis.host,
{no_ready_check: true});
if (settings.redis.password) {
client.auth(settings.redis.password, function() {
console.log('Redis client connected');
});
}
I'm using Redis To Go in combination with the https://github.com/mranney/node_redis library. Redis gives me a url that looks like redis://me:978287c0b670694673d045f08b2e0371#icefish.redistogo.com:9393 but I don't know how to use it as createClient() only takes the host and the port.
I believe that the scheme for the URL you have is:
redis://username:password#host:port.
I don't believe username is used. node_redis provides two methods that you'll use to log in: createClient and auth. There are details in the readme, but for reference here is the relevant portion:
redis.createClient(port, host, options)
Create a new client connection. port defaults to 6379 and host
defaults to 127.0.0.1. If you have redis-server running on the
same computer as node, then the defaults for port and host are
probably fine. options in an object with the following possible
properties:
parser: which Redis protocol reply parser to use. Defaults to
hiredis if that module is installed. This may also be set to
javascript.
return_buffers: defaults to false. If set to true, then bulk
data replies will be returned as node Buffer objects instead of
JavaScript Strings.
createClient() returns a RedisClient object that is named client
in all of the examples here.
client.auth(password, callback)
When connecting to Redis servers that require authentication, the
AUTH command must be sent as the first command after connecting.
This can be tricky to coordinate with reconnections, the ready check,
etc. To make this easier, client.auth() stashes password and will
send it after each connection, including reconnections. callback is
invoked only once, after the response to the very first AUTH command
sent.
I also had to add the parameter no_ready_check: true to the call to redis.createClient().
client = redis.createClient(settings.redis.port,
settings.redis.host,
{no_ready_check: true});
if (settings.redis.password) {
client.auth(settings.redis.password, function() {
console.log('Redis client connected');
});
}