I have been trying to convert this b-string to a png image.
Here is the bytes string for a barcode received from an api. It is called Cloudmersive 1D barcode generator api.
I have tried to use base64.b64decode() and then write binary to an image file but it does not work. I also tried using BytesIO but that does not work either.
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00h\x00\x00\x00d\x08\x02\x00\x00\x00\xe5\xbc\xe2\x8d\x00\x00\x00\x01sRGB\x00\xae\xce\x1c\xe9\x00\x00\x00\x04gAMA\x00\x00\xb1\x8f\x0b\xfca\x05\x00\x00\x00\tpHYs\x00\x00\x0e\xc3\x00\x00\x0e\xc3\x01\xc7o\xa8d\x00\x00\x0c\x8aIDATx^\xed\x95\xd1\x95d\xc5\x0e\x041\x0f\x830\x07_p\xe5y\xc2SU\xce\xc4\x84\xa4Z\x18\xf6{\xe3C'\x94\xca{\xbb\xa7\x81\xc3o\x7f\x8b\xdf~;+\xd3R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083M\xdbG\xd5R\x9c\x86\xf2\xb8'\xdc\xe3\xbb\t;\xc4}\x1a\x85\x11Z\xf0"\xbe\xc3\xac\x84\x968\x12\x083\xcd\xdc\x7f\xf1M~\xfdp?\xc9\xaf\x1f\xee'\xf9\xc6\x0f\xf7\xd7\x1f\xf7\xbf\xf7\xe2\xf7?\xff\xf7\x91}R7\x87\xff\xfb\xf3\xf7\x1fU{s\xbf\xf3+\xf9\xe4\x8f\xbf\xfa\xe1c?<?\x87b\xfb\xec\xef>\xfe\x95\x1dT\xfe\x01\xff\xfa\xc3\x9d\x0f\xcek\xee\xab\xfd\xc6|\xa7\xaf\xcf>{\xb6S\xf5\xd7\xef\xcds\xfd|\xcf\xd7\xeb\x85\x1e\xaf{L\xc5\x97\xea\xcb\xf9\xc3\xbf\xfb\xf8\xd5\x0f\xfb\x1e\xff\xf6\xc3\xe9\xd5w\xf9\xf8F\xf7{\xfe\xf6\xfb\x1f\x7f\xe8\x07\xfa\xba\x9aW\xd3\xec\xa7\x9e\x7fx\xe2\xfbM\xda\x13\x9f\xcb3\xfc\xfe\xe3\xe7\xfc\x95~\x87\xff\xf8o\x1c_\xe3\xaf\xbf>>n\x7f\xb3\xce\xa3)N<\x9e\xfa\xfa\xc0\xf1\xce\xf6\x17\x7f\xf2\x19\xb6\xe3\xe7+\xbe\xff\xf8\x95\xdf\xcf?\xe2\xf3Oy\xd5\x1e|\xe7\x7f\x0e\xe7{\xbc_\xe8\x9f\xe3~\x87\xbf\xce\x0fq\xe8?\xc6\xeb\x87\xbb\xbf\xd9zi\xef\xfd\xdb_~\xbeY\xb2\xfb\xb6\x8f\xea\xfd\xbe\xdf\xf9\xe1\xfa\xe3\x1fW\xbf\xe9\x1f\xf8\xd6\xbfq\x1f\xaf<\xda\xdf\xa8\xcf\xcb\xd7\xd5\x87\x7f\xe6a'\x97\xf9-G\xed\xac?\xfc\xcb\xef\xb3J\xee\x178\xfc\xf1\xe7\xc7c\xff\xe9\xf1/\xf6\xdf\xf9\xe0\xdf~\xb8\xf9\xd9\xfd\x8d\xe7\xfa\xf9\xd9\xed\xd8\x1e;\xb8\xd9\xe8\x7f\xcf\xf8\xeb\xe6\xa7\x7f\xdd\xce\xe5\xfd\xc2\xa2\x8e\xf7\xf4\xf3\x8f\xf7/\xff\xe2?\xfdp\xeb\x8d'\xf8\xfc\xf8\xef7\x1b\xed\xef\xa9\xa5?\xa6\xab^y\xf4\x87\x7f\xb6\x9b?\xf5x\xffJ?\xe2_\xffS=\x9f\xf2\xf1\x9a\xf5g\xf9X|}\xe0?7u\xed/8[\x7f\xec\xab\xfb\xf5P\x7f\xe6\x03\x85\xfe\xf0\xef>\xaeg\x8e\xae\xfb\xe6\xbf\xfc\xcf\xe1\x1f\xbf\xee\xe5\xec\xe1\xf3\x9b\x7f\xd2\x9b_\xbdV\xd4\xd7\xff\x82O\xd7\x1f\xd6\xc9\xe1\xeb\x9d\xedk\xfe\xf7\xc7\xf7Wx\xf1\x9d\x1f\xee\x17\x0f~\xfdp?\xc9\xaf\x1f\xee\xa7\xf8\xfb\xef\xff\x03\x11\xda\xa3\xaefM\x89\xbf\x00\x00\x00\x00IEND\xaeB`\x82'
There's no need to use b64decode or any other operation on that byte string, it's ready to write to the file as is.
with open(r'c:\temp\temp.png', 'wb') as f:
f.write(b_str)
It produces this:
Related
I am struggling to read raw RGB image stream from buffer, converting to openCV (numpy) array and encoding back to .jpg Bytes without using PIL. I can achieve this using PIL, but can't figure out how to achieve the same just using numpy/opencv:
Reading in RBG raw stream with PIL (works):
PILFr = Image.frombuffer('RGB',(768,576),buf,"raw",'RGB',0,1)
opencvPILFr = np.array(PILFr)
Attempting a similar read with openCV, I have:
FlatNp=np.frombuffer(buf,dtype=np.uint8, count=-1)
opencvPILFr=np.resize(FlatNp,(576,768))
re-encoding to buffer (self.frame) as JPEG bytes stream with PIL (works):
PILFr=Image.fromarray(cv2.cvtColor(opencvPILFr, cv2.COLOR_BGR2RGB))
tmp = io.BytesIO()
PILFr.save(tmp,format='jpeg', quality=100)
self.frame = tmp.getvalue()
Attempting to re-encode with openCV (does not work!):
tmp = io.BytesIO()
tmp=cv2.imencode(".jpg",opencvPILFr) #this returns a tuple? not numpy array?
self.frame=tmp.tobytes()
Help appreciated -I can see some reasons why the above WOULDN'T work...but can't see how to fix it.
I figured this out. Further improvements welcome. Still not entirely sure I understand the shape and contents of the imencode returned tuple.
See below my working numpy/opencv alternative to the PIL approach above:
Read in RBG 24bit image from buffer and convert to numpy array (opencvFr) for openCV:
FlatNp=np.frombuffer(buf,dtype=np.uint8, count=-1)
opencvFr=np.resize(FlatNp,(576,768,3))
Encoding the edited numpy image array (opencvFr) as JPEG bytes:
tmp=cv2.imencode('.jpg',opencvFr) #this returns a tuple.
self.frame=tmp[1].tobytes() #bytes containing compressed JPEG image
(I am experimenting with this for a streaming camera application)
I am trying to access csv file from aws s3 bucket and getting error 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte code is below I am using python 3.7 version
from io import BytesIO
import boto3
import pandas as pd
import gzip
s3 = boto3.client('s3', aws_access_key_id='######',
aws_secret_access_key='#######')
response = s3.get_object(Bucket='#####', Key='raw.csv')
# print(response)
s3_data = StringIO(response.get('Body').read().decode('utf-8')
data = pd.read_csv(s3_data)
print(data.head())
kindly help me out here how i can resolve this issue
using gzip worked for me
client = boto3.client('s3', aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
csv_obj = client.get_object(Bucket=####, Key=###)
body = csv_obj['Body']
with gzip.open(body, 'rt') as gf:
csv_file = pd.read_csv(gf)
The error you're getting means the CSV file you're getting from this S3 bucket is not encoded using UTF-8.
Unfortunately the CSV file format is quite under-specified and doesn't really carry information about the character encoding used inside the file... So either you need to know the encoding, or you can guess it, or you can try to detect it.
If you'd like to guess, popular encodings are ISO-8859-1 (also known as Latin-1) and Windows-1252 (which is roughly a superset of Latin-1). ISO-8859-1 doesn't have a character defined for 0x8b (so that's not the right encoding), but Windows-1252 uses that code to represent a left single angle quote (‹).
So maybe try .decode('windows-1252')?
If you'd like to detect it, look into the chardet Python module which, given a file or BytesIO or similar, will try to detect the encoding of the file, giving you what it thinks the correct encoding is and the degree of confidence it has in its detection of the encoding.
Finally, I suggest that, instead of using an explicit decode() and using a StringIO object for the contents of the file, store the raw bytes in an io.BytesIO and have pd.read_csv() decode the CSV by passing it an encoding argument.
import io
s3_data = io.BytesIO(response.get('Body').read())
data = pd.read_csv(s3_data, encoding='windows-1252')
As a general practice, you want to delay decoding as much as you can. In this particular case, having access to the raw bytes can be quite useful, since you can use that to write a copy of them to a local file (that you can then inspect with a text editor, or on Excel.)
Also, if you want to do detection of the encoding (using chardet, for example), you need to do so before you decode it, so again in that case you need the raw bytes, so that's yet another advantage to using the BytesIO here.
I have audio as bytes in the form of:
b'ID3\x04\x00\x00\x00\x00\x00#TSSE\x00\x00\x00\x0f\x00\x00\x03Lavf57.71.100\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\...
That I got from Amazon web services:
import boto3
client = boto3.client('polly')
response = client.synthesize_speech(
Engine='neural',
LanguageCode='en-GB',
OutputFormat='mp3',
SampleRate='8000',
Text='hey whats up this is a test',
VoiceId='Brian'
)
And I want to input it into moviepy audiofile using
AudioFileClip()
AudioFileClip takes filename or an array representing a sound. I know I can save the audio as a file and read it, but I would like to have AudioFileClip take the bytes output I showed above.
I tried:
AudioFileClip(response['AudioStream'].read())
But this gives the error:
TypeError: endswith first arg must be bytes or a tuple of bytes, not
str
What can I do?
You need to convert the stream of audio to a different type. (Thats why its called TypeError). You are putting it as a string and it wants a byte format.
You can convert a str to a byte by using the bytearrayfunction!
https://docs.python.org/3/library/functions.html#func-bytearray
You can also look at this question:
Best way to convert string to bytes in Python 3?
For more help just comment on this anwser, and Ill try to help you as soon as possible.
Hope this can help you on your project,
PythonMasterLua
I want to encode an image in my directory "x.png" to a String or Array{UInt8, 1}.
I am writing a code in Julia to serialize an image using protobufs. It requires the image to be in encoded
String format.
In Python, it is done as follows. I am looking for similar functionality in Julia.
from PIL import Image
img = Image.load('x.png')
import io
output = io.BytesIO()
img.save(output, 'PNG')
img_string_data = output.getvalue()
output.close()
The output may be a String object or an Array{UInt8, 1}
In Julia you can achieve by writing:
img_string_data = read("x.png")
img_string_data now is Vector{UInt8}. You could also write read("x.png", String) to get a String (which is not that useful though as it will probably mostly contain invalid characters).
There is one difference between Julia solution and your Python solution. Julia approach will store in img_string_data the contents identical to what "x.png" holds on binary level while your Python solution will store an identical image, but possibly different on binary level (i.e. PIL might change some bytes in your file).
I am trying to take the contents of a BytesIO or StringIO object and use base64.standard_b64encode() to encode it. I have tried both. This works fine in python 2.7, however in python 3.5 I get the following error.
TypeError: Can't convert 'bytes' object to str implicitly
This is the portion of code having the problem.
output = BytesIO()
img.save(output, format="PNG")
output.seek(0)
data = "data:image/png;base64," + base64.standard_b64encode(output.read())
html = "<html><body><img src='DATA'></body></html>"
I have seen references to fixing this error for strings using b"sting" but I don't know how that would apply to reading from a file.
Thanks
Turns out I the problem was not with the base64 encoding, but rather the string I was trying to append it to. I had to do the following so that python did not see it as a byte encoding anymore.
base64.b64encode(output.read()).decode()