I am transmitting data with Python-can using isotp.stack().
Successfully able to do the same for 8 bytes of frame size but not able to transmit for 64 bytes of frame.
rxid and txid are enabled for 64 bytes of frame size as tested using capl script.
I even tried to add param 'tx_data_min_length': 64 in the stack, but did not work.
Below is my code that I am using.
bus = VirtualBus(net_channel=channel)
print(bus.bus.channel_info)
address = isotp.Address(addressing_mode=isotp.AddressingMode.Normal_29bits, rxid=rxid, txid=txid)
stack = isotp.CanStack(bus=bus.bus,
address=address,
error_handler=print,
params={'tx_padding': 0xCC,
'can_fd': True,
'tx_data_length': 64,
'rx_consecutive_frame_timeout': 100000})
stack.send(data)
while stack.transmitting():
stack.process()
time.sleep(stack.sleep_time())
ERROR:
E RuntimeError: (c_ubyte_Array_8) <class 'IndexError'>: invalid index
Is there any other params I am missing out?
Document that I have read https://can-isotp.readthedocs.io/en/latest/isotp/implementation.html
Related
I'm working on multiple well-formed xml files, whose sizes range from 100 MB to 4 GB. My goal is to read them as strings and then import them as ElementTree objects using .fromstring() method (from xml.etree.ElementTree module).
However, as the process goes through and the string size increases, two exceptions occured related to memory restriction :
xml.etree.ElementTree.ParseError: out of memory: line 1, column 0
OverflowError: size does not fit in an int
It looks like .fromstring() method enforces a string size limit to the input, around 1GB... ?
To debug this, I wrote a short script using a for loop:
xmlFiles_list = [path1, path2, ...]
for fp in xmlFiles_list:
xml_fo = open(fp, mode='r', encoding="utf-8")
xml_asStr = xml_fo.read()
xml_fo.close()
print(len(xml_asStr.encode("utf-8")) / 10**9) # display string size in GB
try:
etree = cElementTree.fromstring(xml_asStr)
print(".fromstring() success!\n")
except Exception as e:
print(f"Error :{type(e)} {str(e)}\n")
continue
The ouput is as following :
0.895206753
.fromstring() success!
1.220224531
Error :<class 'xml.etree.ElementTree.ParseError'> out of memory: line 1, column 0
1.328233473
Erreur :<class 'xml.etree.ElementTree.ParseError'> out of memory: line 1, column 0
2.567867904
Error :<class 'OverflowError'> size does not fit in an int
4.080672538
Error :<class 'OverflowError'> size does not fit in an int
I found multiple workarounds to avoid this issue : .parse() method or lxml module for bette performance. I just hope someone could shed some light on this :
Is there a specific string size limit in xml.etree.ET module and .fromstring() method ?
Why do I end up with two different exceptions as the string size increases ? Are they related to the same memory-allocation restriction ?
Python version/system: 3.9 (64 bits)
RAM : 32go
Hope my topic is clear enough, I'm new on stackoverflow
I’m trying to get feature vectors from the encoder model using pre-trained ALBERT v2 weights. i have a nvidia 1650ti gpu (4 GB) , and sufficient RAM(8GB) but for some reason I’m getting Runtime error saying -
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:75] data.
DefaultCPUAllocator: not enough memory: you tried to allocate
491520000 bytes. Buy new RAM!
I’m really new to pytorch and deep learning in general. Can anyone please tell me what is wrong?
My entire code -
encoded_test_data = tokenized_test_values[‘input_ids’]
encoded_test_masks = tokenized_test_values[‘attention_mask’]
encoded_train_data = torch.from_numpy(encoded_train_data).to(device)
encoded_masks = torch.from_numpy(encoded_masks).to(device)
encoded_test_data = torch.from_numpy(encoded_test_data).to(device)
encoded_test_masks = torch.from_numpy(encoded_test_masks).to(device)
config = EncoderDecoderConfig.from_encoder_decoder_configs(BertConfig(),BertConfig())
EnD_model = EncoderDecoderModel.from_pretrained(‘albert-base-v2’,config=config)
feature_extractor = EnD_model.get_encoder()
feature_vector = feature_extractor.forward(input_ids=encoded_train_data,attention_mask = encoded_masks)
feature_test_vector = feature_extractor.forward(input_ids = encoded_test_data, attention_mask = encoded_test_masks)
Also 491520000 bytes is about 490 MB which should not be a problem.
I tried reducing the number of training examples and also the length of the maximum padded input . The OOM error still exists even though the required space now is 153 MB , which should easily be managable.
I also have maxed out the RAM limit of the heap of pycharm software to 2048 MB. I really dont know what to do now…
I'm reading binary file http://snap.stanford.edu/data/amazon/productGraph/image_features/image_features.b
using pyspark.
import array
from io import StringIO
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 4106)
def mapper(features):
a = array.array('f')
a.frombytes(features)
return a.tolist()
def byte_mapper(bytes):
return str(bytes)
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
When just product_id is selected from the rdd using
decoded_embeddings = img_embedding_file.map(lambda x: [byte_mapper(x[:10]), mapper(x[10:])])
The output for product_id is
["b'1582480311'", "b'\\x00\\x00\\x00\\x00\\x88c-?\\xeb\\xe2'", "b'7#\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\xec/\\x0b?\\x00\\x00\\x00\\x00K\\xea'", "b'\\x00\\x00c\\x7f\\xd9?\\x00\\x00\\x00\\x00'", "b'L\\xa6\\n>\\x00\\x00\\x00\\x00\\xfe\\xd4'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\xe5\\xd0\\xa2='", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"]
The file is hosted on s3.
The file in each row has first 10 bytes for product_id next 4096 bytes as image_features
I'm able to extract all the 4096 image features but facing issue when reading the first 10 bytes and converting it into proper readable format.
EDIT:
Finally, the problem comes from the recordLength. It's not 4096 + 10 but 4096*4 + 10. Chaging to :
img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b", 16394)
Should work.
Actually you can find this in the provided code from the web site you downloaded the binary file:
for i in range(4096):
feature.append(struct.unpack('f', f.read(4))) # <-- so 4096 * 4
Old answer:
I think the issue comes from your byte_mapper function.
That's not the correct way to convert bytes to string. You should be using decode:
bytes = b'1582480311'
print(str(bytes))
# output: "b'1582480311'"
print(bytes.decode("utf-8"))
# output: '1582480311'
If you're getting the error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 4: invalid start byte
That means product_id string contains non-utf8 characters. If you don't know the input encoding, it's difficult to convert into strings.
However, you may want to ignore those characters by adding option ignore to decode function:
bytes.decode("utf-8", "ignore")
I have been trying to put my keras model .tflite file into the google's Tflitecamera demo. But i am getting an allocation error(Cannot convert between a TensorFlowLite buffer with 12288 bytes and a ByteBuffer with 1072812 bytes. at ).
I assume it is because of wrong bytebuffer allocation.
ByteBuffer.allocate(
DIM_BATCH_SIZE
* getImageSizeX()
* getImageSizeY()
* DIM_PIXEL_SIZE
* getNumBytesPerChannel());
Could anyone shed some light on this. I am a newbie to TensorFlow.
Following is the log
08-10 11:56:28.905 28066-28066/android.example.com.tflitecamerademo E/MultiWindowProxy: getServiceInstance failed!
08-10 11:56:35.675 28066-28092/android.example.com.tflitecamerademo E/AndroidRuntime: FATAL EXCEPTION: CameraBackground
Process: android.example.com.tflitecamerademo, PID: 28066
java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite buffer with 12288 bytes and a ByteBuffer with 1072812 bytes.
at org.tensorflow.lite.Tensor.throwExceptionIfTypeIsIncompatible(Tensor.java:175)
at org.tensorflow.lite.Tensor.setTo(Tensor.java:65)
at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:126)
at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:168)
at org.tensorflow.lite.Interpreter.run(Interpreter.java:145)
at com.example.android.tflitecamerademo.ImageClassifierFloatInception.runInference(ImageClassifierFloatInception.java:103)
at com.example.android.tflitecamerademo.ImageClassifier.classifyFrame(ImageClassifier.java:136)
at com.example.android.tflitecamerademo.Camera2BasicFragment.classifyFrame(Camera2BasicFragment.java:702)
at com.example.android.tflitecamerademo.Camera2BasicFragment.-wrap0(Camera2BasicFragment.java)
at com.example.android.tflitecamerademo.Camera2BasicFragment$4.run(Camera2BasicFragment.java:597)
at android.os.Handler.handleCallback(Handler.java:822)
at android.os.Handler.dispatchMessage(Handler.java:104)
at android.os.Looper.loop(Looper.java:207)
at android.os.HandlerThread.run(HandlerThread.java:61)
You can find the reason of this error in the following link :
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/java/src/main/java/org/tensorflow/lite/Tensor.java
As you see in line 170 to 181, if Capacity of the output buffer is not equal to the number of Bytes then this error will happen.
In Cassandra, I have the following Column Family:
<ColumnFamily CompareWith="TimeUUIDType" Name="Posts"/>
I'm trying to insert a record into it as follows using a C++ generated function generated by Thrift:
ColumnPath new_col;
new_col.__isset.column = true; /* this is required! */
new_col.column_family.assign("Posts");
new_col.super_column.assign("");
new_col.column.assign("1968ec4a-2a73-11df-9aca-00012e27a270");
client.insert("Keyspace1", "somekey", new_col, "Random Value", 1234, ONE);
However, I'm getting the following error: "UUIDs must be exactly 16 bytes"
I've even tried the Cassandra CLI with the following command:
set Keyspace1.Posts['somekey']['1968ec4a-2a73-11df-9aca-00012e27a270'] = 'Random Value'
but I still get the following error:
Exception null
InvalidRequestException(why:UUIDs must be exactly 16 bytes)
at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:11994)
at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:659)
at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:632)
at org.apache.cassandra.cli.CliClient.executeSet(CliClient.java:420)
at org.apache.cassandra.cli.CliClient.executeCLIStmt(CliClient.java:80)
at org.apache.cassandra.cli.CliMain.processCLIStmt(CliMain.java:132)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:173)
Thrift is a binary protocol; 16 bytes means 16 bytes. "1968ec4a-2a73-11df-9aca-00012e27a270" is 36 bytes. You need to get your library to give you the raw, 16 bytes form.
I don't use C++ myself, but "version 1 uuid" is the magic string you want to google for when looking for a library that can do this. http://www.google.com/search?q=C%2B%2B+version+1+uuid