How can i get response size of headers REST API - node.js

I know in headers we can get content length, which indeed gives Content body length But I need Response size of headers
For example:
API Response =
{
"1": 1
}
If I print console.log(res.getHeader('content-length')); it gives 7 which is content length of body.
But I need response size of header which is 377Bytes(header + body) as shown in postman

One of the possible way to get the bytes of file is to download it , by using curl command in the following mannner:
curl -so /dev http://www.yourip.org/http-your-file/ -w '%{size_download}'
where -w/--write-out
defines what to display after a completed and successful operation

There is a NPM module to get object sizeof, you can install it with npm install object-sizeof
var sizeof = require('object-sizeof');
// 2B per character, 6 chars total => 12B
console.log(sizeof({abc: 'def'}));
// 8B for Number => 8B
console.log(sizeof(12345));
var param = {
'a': 1,
'b': 2,
'c': {
'd': 4
}
};
// 4 one two-bytes char strings and 3 eighth-bytes numbers => 32B
console.log(sizeof(param));
You can use it however you want in your code.
Example ::
console.log(sizeof(response.body));

Related

Push aggregated metrics to the Prometheus Pushgateway from clustered Node JS

I've ran my node apps in a cluster with Prometheus client implemented. And I've mutated the metrics (for testing purpose) so I've got these result:
# HELP otp_generator_generate_failures A counter counts the number of generate OTP fails.
# TYPE otp_generator_generate_failures counter
otp_generator_generate_failures{priority="high"} 794
# HELP otp_generator_verify_failures A counter counts the number of verify OTP fails.
# TYPE otp_generator_verify_failures counter
otp_generator_verify_failures{priority="high"} 802
# HELP smsblast_failures A counter counts the number of SMS delivery fails.
# TYPE smsblast_failures counter
smsblast_failures{priority="high"} 831
# HELP send_success_calls A counter counts the number of send API success calls.
# TYPE send_success_calls counter
send_success_calls{priority="low"} 847
# HELP send_failure_calls A counter counts the number of send API failure calls.
# TYPE send_failure_calls counter
send_failure_calls{priority="high"} 884
# HELP verify_success_calls A counter counts the number of verify API success calls.
# TYPE verify_success_calls counter
verify_success_calls{priority="low"} 839
# HELP verify_failure_calls A counter counts the number of verify API failure calls.
# TYPE verify_failure_calls counter
verify_failure_calls{priority="high"} 840
FYI: I have adopted the given example from: https://github.com/siimon/prom-client/blob/master/example/cluster.js
Then, I pushed the metrics and everything work fine at the process. But, when I checked the gateway portal, it gives different result from those metrics above.
The one that becomes my opinion is the pushed metrics came from single instance instead of aggregated metrics from few instances running. Isn't it?
So, has anyone ever solved this problem?
Oh, this is my push code:
const { Pushgateway, register } = require('prom-client')
const promPg = new Pushgateway(
config.get('prometheus.pushgateway'),
{
timeout: 5000
},
register
)
promPg.push({ jobName: 'sms-otp-middleware' }, (err, resp, body) => {
console.log(`Error: ${err}`)
console.log(`Body: ${body}`)
console.log(`Response status: ${resp.statusCode}`)
res.status(200).send('metrics_pushed')
})

How to read binary data in pyspark

I'm reading binary file http://snap.stanford.edu/data/amazon/productGraph/image_features/image_features.b
using pyspark.
import array
from io import StringIO
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 4106)
def mapper(features):
a = array.array('f')
a.frombytes(features)
return a.tolist()
def byte_mapper(bytes):
return str(bytes)
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
When just product_id is selected from the rdd using
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
The output for product_id is
["b'1582480311'", "b'\\x00\\x00\\x00\\x00\\x88c-?\\xeb\\xe2'", "b'7#\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\xec/\\x0b?\\x00\\x00\\x00\\x00K\\xea'", "b'\\x00\\x00c\\x7f\\xd9?\\x00\\x00\\x00\\x00'", "b'L\\xa6\\n>\\x00\\x00\\x00\\x00\\xfe\\xd4'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\xe5\\xd0\\xa2='", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"]
The file is hosted on s3.
The file in each row has first 10 bytes for product_id next 4096 bytes as image_features
I'm able to extract all the 4096 image features but facing issue when reading the first 10 bytes and converting it into proper readable format.
EDIT:
Finally, the problem comes from the recordLength. It's not 4096 + 10 but 4096*4 + 10. Chaging to :
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 16394)
Should work.
Actually you can find this in the provided code from the web site you downloaded the binary file:
for i in range(4096):
feature.append(struct.unpack('f', f.read(4))) # <-- so 4096 * 4
Old answer:
I think the issue comes from your byte_mapper function.
That's not the correct way to convert bytes to string. You should be using decode:
bytes = b'1582480311'
print(str(bytes))
# output: "b'1582480311'"
print(bytes.decode("utf-8"))
# output: '1582480311'
If you're getting the error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 4: invalid start byte
That means product_id string contains non-utf8 characters. If you don't know the input encoding, it's difficult to convert into strings.
However, you may want to ignore those characters by adding option ignore to decode function:
bytes.decode("utf-8", "ignore")

DocuSign document blank after request and save

After requesting a document via the DocuSign api and writing it to the file system it appears blank after opening it. The docs say it returns a "PDF File" and the response body is returned as below.
const doc =
await rp.get(`${apiBaseUrl}/${BASE_URI_SUFFIX}/accounts/${accountId}/envelopes/${envelopeId}/documents/${document.documentId}`,
{auth: { bearer: token }}
);
fs.writeFile(document.name, new Buffer(doc, "binary"), function(err) {
if (err) throw err;
console.log('Saved!');
});
Response body:
{
"documents": [
{
"name": "Name of doc.docx",
"content": "%PDF-1.5\n%\ufffd\ufffd\ufffd\ufffd\n%Writing objects...\n4 0 obj\n<<\n/Type /Page\n/Resources 5 0 R\n/Parent 3 0 R\n/MediaBox [0 0 612 792 ]\n/Contents [6 0 R 7 0 R 8 0 R 9 0 R 10 0 R ]\n/Group <<\n/Type /Group\n/S /Transparency\n/CS /DeviceRGB\n>>\n/Tabs /S\n/StructParents 0\n>>\nendobj\n5 0 obj\n<<\n/Font <<\n/F1 11 0 R\n/F2 12 0 R\n/F3 13 0 R\n>>\n/ExtGState <<\n/GS7 14 0 R\n/GS8 15 0 R\n>>\n/ProcSet [/PDF /Text ...
}
]}
Screenshot of document:
The EnvelopeDocuments::get API method returns the PDF itself, not an object as you are showing.
For a working example of the method, see example 7, part of the Node.js set of examples.
Added
Also, the fs.writeFile call supports writing from a string source. I'd try:
fs.writeFile(document.name, doc, {encoding: "binary"},
function(err) {
if (err) throw err;
console.log('Saved!');
});
Incorrect encoding
Your question shows the pdf's content as a string with the control characters encoded as unicode strings:
"%PDF-1.5\n%\ufffd\ufffd\ufffd\ufffd\n%Writing objects...
but this is not correct. The beginning of a PDF file includes binary characters that are not displayable except in a hex editor. This is what you should see at the top of a PDF:
Note the 10th character. It is hex c4. In your string, the equivalent character has been encoded as \ufffd (it is ok that they aren't the same character, they are two different PDFs). The fact that the character has been encoded is your problem.
Solutions
Convince the requests library and the fs.WriteFile methods to not encode the data. Or to decode it as needed. See this solution for the requests library.
Or use the DocuSign Node.js SDK as I show in the example code referenced above.

How to send commands to smart card reader (and not to the smart card) while no card present?

Preface:
I have a dual interface smart card reader that has some extended capabilities (other than sending APDU commands to card and receiving APDU responses).
For example in its document it is mentioned that you can get firmware version of your reader using following command:
GET_FIRMWARE_VERSION: FF 69 44 42 05 68 92 00 05 00
In its tool, there is a button for this function and it works fine:
I even sniffed USB port to see what exactly exchanged in the connection between my PC and my reader for this function:
Command:
Response:
Problem:
I want to get my reader version (and maybe send other extended commands) using other tools or via code, but I must insert a card in the card reader to be able sending commands, otherwise I receive No Card Present exception, while I don't want to send commands to card! (The reader tool answers successfully to GET_FIRMWARE_VERSION without any card available in the reader's slots)
What I did so far:
1.I tried some tools, including OpenSCTool , PyAPDUTool, and another reader's tool.
2.I wrote following python script to send extended commands.
#--- Importing required modules.
import sys
import time
sys.path.append("D:\\PythonX\\Lib\\site-packages")
from smartcard.scard import *
import smartcard.util
from smartcard.System import readers
#---This is the list of commands that we want to send device
cmds =[[,0xFF,0x69,0x44,0x42,0x05,0x68,0x92,0x00,0x04,0x00],]
#--- Let's to make a connection to the card reader
r=readers()
print "Available Readers :",r
print
target_reader = input("--- Select Reader (0, 1 , ...): ")
print
while(True):
try:
print "Using :",r[target_reader]
reader = r[target_reader]
connection=reader.createConnection()
connection.connect()
break
except:
print "--- Exception occured! (Wrong reader or No card present)"
ans = raw_input("--- Try again? (0:Exit/1:Again/2:Change Reader)")
if int(ans)==0:
exit()
elif int(ans)==2:
target_reader = input("Select Reader (0, 1 , ...): ")
#--- An struct for APDU responses consist of Data, SW1 and SW2
class stru:
def __init__(self):
self.data = list()
self.sw1 = 0
self.sw2 = 0
resp = stru()
def send(cmds):
for cmd in cmds:
#--- Following 5 line added to have a good format of command in the output.
temp = stru() ;
temp.data[:]=cmd[:]
temp.sw1=12
temp.sw2=32
modifyFormat(temp)
print "req: ", temp.data
resp.data,resp.sw1,resp.sw2 = connection.transmit(cmd)
modifyFormat(resp)
printResponse(resp)
def modifyFormat(resp):
resp.sw1=hex(resp.sw1)
resp.sw2=hex(resp.sw2)
if (len(resp.sw2)<4):
resp.sw2=resp.sw2[0:2]+'0'+resp.sw2[2]
for i in range(0,len(resp.data)):
resp.data[i]=hex(resp.data[i])
if (len(resp.data[i])<4):
resp.data[i]=resp.data[i][0:2]+'0'+resp.data[i][2]
def printResponse(resp):
print "res: ", resp.data,resp.sw1,resp.sw2
send(cmds)
connection.disconnect()
Output:
>>> ================================ RESTART ================================
Available Readers : ['CREATOR CRT-603 (CZ1) CCR RF 0', 'CREATOR CRT-603 (CZ1) CCR SAM 0']
--- Select Reader (0, 1 , ...): 0
Using : CREATOR CRT-603 (CZ1) CCR RF 0
--- Exception occured! (Wrong reader or No card present)
--- Try again? (0:Exit/1:Again/2:Change Reader)
>>> ================================ RESTART ================================
Available Readers : ['CREATOR CRT-603 (CZ1) CCR RF 0', 'CREATOR CRT-603 (CZ1) CCR SAM 0']
--- Select Reader (0, 1 , ...): 1
Using : CREATOR CRT-603 (CZ1) CCR SAM 0
--- Exception occured! (Wrong reader or No card present)
--- Try again? (0:Exit/1:Again/2:Change Reader)
But both have the mentioned problem!
Questions:
1- How to send extended commands to reader while there is no card available?
2- Why I can't see command header in the sniffed data? (Note that, as Header is a pre-designated fixed value for all extended commands, I think the reader tool doesn't send the header with GET_FIRMWARE_VERSION command and it send only the data! but how does it works?)
Update:
Using trial and error I found something useful.
Assumptions:
Pseudo-APDUs fixed header = FF 69 44 42
Pseudo-APDU data field for GET_READER_FIRMWARE_VERSION = 68 92 00 04 00
Pseudo-APDU data field for CHANGE_SAM_SLOT = 68 92 01 00 03 XX 00 00
(My reader has two SAM slots, so XX can be 01 or 02)
SELECT APDU command = 00 A4 04 00 00
Ok, I wrote the following Java Program:
import java.util.List;
import java.util.Scanner;
import javax.smartcardio.Card;
import javax.smartcardio.CardChannel;
import javax.smartcardio.CardException;
import javax.smartcardio.CardTerminal;
import javax.smartcardio.CommandAPDU;
import javax.smartcardio.ResponseAPDU;
import javax.smartcardio.TerminalFactory;
import javax.xml.bind.DatatypeConverter;
public class TestPCSC {
public static void main(String[] args) throws CardException {
TerminalFactory tf = TerminalFactory.getDefault();
List< CardTerminal> terminals = tf.terminals().list();
System.out.println("Available Readers:");
System.out.println(terminals + "\n");
Scanner scanner = new Scanner(System.in);
System.out.print("Which reader do you want to send your commands to? (0 or 1 or ...): ");
String input = scanner.nextLine();
int readerNum = Integer.parseInt(input);
CardTerminal cardTerminal = (CardTerminal) terminals.get(readerNum);
Card connection = cardTerminal.connect("DIRECT");
CardChannel cardChannel = connection.getBasicChannel();
System.out.println("Write your commands in Hex form, without '0x' or Space charaters.");
System.out.println("\n---------------------------------------------------");
System.out.println("Pseudo-APDU Mode:");
System.out.println("---------------------------------------------------");
while (true) {
System.out.println("Pseudo-APDU command: (Enter 0 to send APDU command)");
String cmd = scanner.nextLine();
if (cmd.equals("0")) {
break;
}
System.out.println("Command : " + cmd);
byte[] cmdArray = hexStringToByteArray(cmd);
byte[] resp = connection.transmitControlCommand(CONTROL_CODE(), cmdArray);
String hex = DatatypeConverter.printHexBinary(resp);
System.out.println("Response : " + hex + "\n");
}
System.out.println("\n---------------------------------------------------");
System.out.println("APDU Mode:");
System.out.println("---------------------------------------------------");
while (true) {
System.out.println("APDU command: (Enter 0 to exit)");
String cmd = scanner.nextLine();
if (cmd.equals("0")) {
break;
}
System.out.println("Command : " + cmd);
byte[] cmdArray = hexStringToByteArray(cmd);
ResponseAPDU resp = cardChannel.transmit(new CommandAPDU(cmdArray));
byte[] respB = resp.getBytes();
String hex = DatatypeConverter.printHexBinary(respB);
System.out.println("Response : " + hex + "\n");
}
connection.disconnect(true);
}
public static int CONTROL_CODE() {
String osName = System.getProperty("os.name").toLowerCase();
if (osName.indexOf("windows") > -1) {
/* Value used by both MS' CCID driver and SpringCard's CCID driver */
return (0x31 << 16 | 3500 << 2);
} else {
/* Value used by PCSC-Lite */
return 0x42000000 + 1;
}
}
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4)
+ Character.digit(s.charAt(i + 1), 16));
}
return data;
}
}
In the above program, I can send commands to my reader using both connection.transmitControlCommand and cardChannel.transmit() methods. The point is that, all the commands that send to the reader using first method, are assumed as Pseudo-APDU command and I should not use the Psedo-APDU header for them! And all the commands that send to the reader using second method, are assumed as regular APDU commands, so if I need to send Pseudo-APDU commands via second method, I must add the Pseudo-APDU header to it.
Let see output for the contact-less reader:
run:
Available Readers:
[PC/SC terminal ACS ACR122 0,
PC/SC terminal CREATOR CRT-603 (CZ1) CCR RF 0,
PC/SC terminal CREATOR CRT-603 (CZ1) CCR SAM 0]
Which reader do you want to send your commands to? (0 or 1 or ...): 1
Write your commands in Hex form, without '0x' or Space charaters.
---------------------------------------------------
Pseudo-APDU Mode:
---------------------------------------------------
Pseudo-APDU command: (Enter 0 to send APDU command)
00A4040000
Command : 00A4040000
Response : 6800
//Based on reader's documents, 0x6800 means "Class byte is not correct"
//As I have a regular java card in the RF field of my reader, I conclude that
//this response is Reader's response (and not card response)
Pseudo-APDU command: (Enter 0 to send APDU command)
6892000400
Command : 6892000400
Response : 433630335F435A375F425F31353038323100039000
Pseudo-APDU command: (Enter 0 to send APDU command)
FF694442056892000400
Command : FF694442056892000400
Response : 6800
//Pseudo-APDU commands doesn't work in Pseudo-APDU mode if I add the Pseudo-APDU header to them.
Pseudo-APDU command: (Enter 0 to send APDU command)
00A4040000
Command : 00A4040000
Response : 6800
Pseudo-APDU command: (Enter 0 to send APDU command)
0
---------------------------------------------------
APDU Mode:
---------------------------------------------------
APDU command: (Enter 0 to exit)
00A4040000
Command : 00A4040000
Response : 6F198408A000000018434D00A50D9F6E061291921101009F6501FF9000
APDU command: (Enter 0 to exit)
6892000400
Command : 6892000400
Response : 6E00
//This is the response of my card. I can't receive Firmware version in APDU mode using this command without Pseudo-APDU header.
APDU command: (Enter 0 to exit)
FF694442056892000400
Command : FF694442056892000400
Response : 433630335F435A375F425F31353038323100099000
//I successfully received Firmware version in APDU mode using the fixed Pseudo-APDU header.
APDU command: (Enter 0 to exit)
00A4040000
Command : 00A4040000
Response : 6F198408A000000018434D00A50D9F6E061291921101009F6501FF9000
APDU command: (Enter 0 to exit)
0
BUILD SUCCESSFUL (total time: 1 minute 36 seconds)
Is there any problem still?
Yes, two problems!:
1-the above program works fine only for its first run. I mean, if I stop running and rerun it, the second method throw an exception:
run:
Available Readers:
[PC/SC terminal ACS ACR122 0, PC/SC terminal CREATOR CRT-603 (CZ1) CCR RF 0, PC/SC terminal CREATOR CRT-603 (CZ1) CCR SAM 0]
Which reader do you want to send your commands to? (0 or 1 or ...): 1
Write your commands in Hex form, without '0x' or Space charaters.
---------------------------------------------------
Pseudo-APDU Mode:
---------------------------------------------------
Pseudo-APDU command: (Enter 0 to send APDU command)
00A4040000
Command : 00A4040000
Response : 6800
Pseudo-APDU command: (Enter 0 to send APDU command)
FF694442056892000400
Command : FF694442056892000400
Response : 6800
Pseudo-APDU command: (Enter 0 to send APDU command)
6892000400
Command : 6892000400
Response : 433630335F435A375F425F31353038323100049000
Pseudo-APDU command: (Enter 0 to send APDU command)
00A4040000
Command : 00A4040000
Response : 6800
Pseudo-APDU command: (Enter 0 to send APDU command)
0
---------------------------------------------------
APDU Mode:
---------------------------------------------------
APDU command: (Enter 0 to exit)
00A4040000
Command : 00A4040000
Exception in thread "main" javax.smartcardio.CardException: sun.security.smartcardio.PCSCException: Unknown error 0x16
at sun.security.smartcardio.ChannelImpl.doTransmit(ChannelImpl.java:219)
at sun.security.smartcardio.ChannelImpl.transmit(ChannelImpl.java:90)
at TestPCSC.main(TestPCSC.java:58)
Caused by: sun.security.smartcardio.PCSCException: Unknown error 0x16
at sun.security.smartcardio.PCSC.SCardTransmit(Native Method)
at sun.security.smartcardio.ChannelImpl.doTransmit(ChannelImpl.java:188)
... 2 more
Java Result: 1
BUILD SUCCESSFUL (total time: 39 seconds)
As you see above, I can't use second method anymore and I need to power-off the reader and power on it again to make it work fine again.
2-The contact interface (I mean the SAM reader) throw previous exception always! I mean the second method doesn`t work at all (neither first run, nor second and third .... )
Note that I tried different readers, it seems that this is not limited only ti this reader. Some ACS readers also have a similar or exactly the same problem with rerun
Does anyone have any idea?
And as a side question, does Python have any equal methods for sending Pseudo-APDU like Java?
And finally, from the view of Reader, what's the difference between connection.transmitControlCommand and cardChannel.transmit() methods?
When you stop "Smart Card" service, does the tool still return a firmware version? If yes, then the tool might use raw IOCTL commands (DeviceIoControl) to communicate with the driver.
Also take a look at this question. The author says you have to set SCARD_PROTOCOL_UNDEFINED as protocol parameter:
SCardConnect(hSC,
readerState.szReader,
SCARD_SHARE_DIRECT,
SCARD_PROTOCOL_UNDEFINED,
&hCH,
&dwAP
);
I just tried it and it seems to work for Windows 10 at least. Communication was possible with no card inserted. I did not test for other Windows versions though.
readers() is array index of available readers
reader = r[target_reader]
convert
reader = r[int(target_reader)]
output
Available Readers : ['JAVACOS Virtual Contact Reader 0', 'JAVACOS Virtual Contactless Reader 1', 'OMNIKEY CardMan 3x21 0']
--- Select Reader (0, 1 , ...): 2
Using : OMNIKEY CardMan 3x21 0
ATR : 3B 9E 94 80 1F 47 80 31 A0 73 BE 21 13 66 86 88 02 14 4B 10 19

How do I insert a row with a TimeUUIDType column in Cassandra?

In Cassandra, I have the following Column Family:
<ColumnFamily CompareWith="TimeUUIDType" Name="Posts"/>
I'm trying to insert a record into it as follows using a C++ generated function generated by Thrift:
ColumnPath new_col;
new_col.__isset.column = true; /* this is required! */
new_col.column_family.assign("Posts");
new_col.super_column.assign("");
new_col.column.assign("1968ec4a-2a73-11df-9aca-00012e27a270");
client.insert("Keyspace1", "somekey", new_col, "Random Value", 1234, ONE);
However, I'm getting the following error: "UUIDs must be exactly 16 bytes"
I've even tried the Cassandra CLI with the following command:
set Keyspace1.Posts['somekey']['1968ec4a-2a73-11df-9aca-00012e27a270'] = 'Random Value'
but I still get the following error:
Exception null
InvalidRequestException(why:UUIDs must be exactly 16 bytes)
at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:11994)
at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:659)
at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:632)
at org.apache.cassandra.cli.CliClient.executeSet(CliClient.java:420)
at org.apache.cassandra.cli.CliClient.executeCLIStmt(CliClient.java:80)
at org.apache.cassandra.cli.CliMain.processCLIStmt(CliMain.java:132)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:173)
Thrift is a binary protocol; 16 bytes means 16 bytes. "1968ec4a-2a73-11df-9aca-00012e27a270" is 36 bytes. You need to get your library to give you the raw, 16 bytes form.
I don't use C++ myself, but "version 1 uuid" is the magic string you want to google for when looking for a library that can do this. http://www.google.com/search?q=C%2B%2B+version+1+uuid

Resources