I am learning SMSC smc91cx driver code, and I learned how to program test code for smc91c111 nic by the instructions of Application Note 9-6. I cannot understand the following instructions under "Transmitting A Packet":
Write the destination address (three successive writes: bytes 10, bytes 32, bytes 54)
Write 0xFFFF, 0xFFFF, 0xFFFF
Write the source address (three successive writes: bytes 10, bytes32, bytes 54)
Write 0x0000, 0x0000, 0x0000
I cannot make sense of these instructions. Should I write 10 bytes size of 0xFF plus 32 bytes size plus 54 bytes size to the buffer, or just write 0xFF in 10th byte postion, 32th, 54th byte postion?
But if so, why would you write 0x0000 to the same position?
Rather than allocating several different registers to write to, that chip has you write to the same DATA register serially until you set all the info. The DATA register is 2 bytes wide, but a MAC address is 6 bytes, numbered 0-5. So you have to write it 2 bytes at a time: bytes number 1 and 0 first, followed by bytes number 3 and 2, then bytes number 5 and 4. Then write 0xFFFF 3 times to the DATA register, then repeat for the source address and the 0x0000s.
Related
From the doc
buffering is an optional integer used to set the buffering policy.
Pass 0 to switch buffering off (only allowed in binary mode), 1 to
select line buffering (only usable in text mode), and an integer > 1
to indicate the size in bytes of a fixed-size chunk buffer. When no
buffering argument is given, the default buffering policy works as
follows:
Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying
device’s “block size” and falling back on io.DEFAULT_BUFFER_SIZE. On
many systems, the buffer will typically be 4096 or 8192 bytes long.
“Interactive” text files (files for which isatty() returns True) use line buffering. Other text files use the policy described above
for binary files.
I open a file named test.log with text mode, and set the buffering to 16. So I think the chunk size is 16, and when I write 32 bytes string to the file. It will call write(syscall) twice. But acutally, it only call once.(test in Python 3.7.2 GCC 8.2.1 20181127 on Linux)
import os
try:
os.unlink('test.log')
except Exception:
pass
with open('test.log', 'a', buffering=16) as f:
for _ in range(10):
f.write('a' * 32)
Using strace -e write python3 test.py to trace syscall, and get following
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 320) = 320
What does the buffering means?
This answer is valid for CPython 3.7 other implementations of Python can differ.
The open() function in text mode returns _io.TextIOWrapper(). The _io.TextIOWrapper() has internal 'buffer' called pending_bytes with size of 8192 bytes (it is hard coded) and it also have handle on _io.BufferedWriter() for text mode w or _io.BufferedRandom() for text mode a. The size of _io.BufferedWriter()/_io.BufferedRandom() is specified by the argument buffering in the open() function.
When you call into _io.TextIOWrapper().write("some text") it will add the text into internal pending_bytes buffer. After some writes you will fill the pending_bytes buffer and then it will be written into buffer inside _io.BufferedWriter(). When you fill up also the buffer inside _io.BufferedWriter() then it will be written into target file.
When you open file in binary mode you will get directly the _io.BufferedWriter()/_io.BufferedRandom() object initialized with buffer size from buffering parametr.
Let's look at some examples. I will start with simpler one using binary mode.
# Case 1
with open('test.log', 'wb', buffering=16) as f:
for _ in range(5):
f.write(b'a'*15)
strace output:
write(3, "aaaaaaaaaaaaaaa", 15) = 15
write(3, "aaaaaaaaaaaaaaa", 15) = 15
write(3, "aaaaaaaaaaaaaaa", 15) = 15
write(3, "aaaaaaaaaaaaaaa", 15) = 15
write(3, "aaaaaaaaaaaaaaa", 15) = 15
In the first iteration it fill buffer with 15 bytes. In the second iteration it discovers that adding another 15 bytes would overflow the buffer so it first flush it (calls system write) and then save those new 15 bytes. In next iteration the same happens again. After last iteration in the buffer is 15 B which are written on close of the file (leaving the with context).
The second case, I will try write into buffer more data than the buffer's size:
# Case 2
with open('test.log', 'wb', buffering=16) as f:
for _ in range(5):
f.write(b'a'*17)
strace output:
write(3, "aaaaaaaaaaaaaaaaa", 17) = 17
write(3, "aaaaaaaaaaaaaaaaa", 17) = 17
write(3, "aaaaaaaaaaaaaaaaa", 17) = 17
write(3, "aaaaaaaaaaaaaaaaa", 17) = 17
write(3, "aaaaaaaaaaaaaaaaa", 17) = 17
What happens here is that in the first iteration it will try write into buffer 17 B but it cannot fit there so it is directly written into the file and buffer stays empty. This applies for every iteration.
Now let's look at the text mode.
# Case 3
with open('test.log', 'w', buffering=16) as f:
for _ in range(5):
f.write('a'*8192)
strace output:
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384) = 16384
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384) = 16384
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 8192) = 8192
First recall that pending_bytes has size 8192 B. In the first iteration it writes 8192 bytes (from code: 'a'*8192) into pending_bytes buffer. In the second iteration it adds to the pending_buffer another 8192 bytes and discovers it is more than 8192 (size of pending_bytes buffer) and writes it into underlying _io.BufferedWriter(). The buffer in _io.BufferedWriter() has size 16 B (buffering parameter) so it will immediately writes into file (same as case 2). Now the pending_buffer is empty and in the third iteration it's again filled with 8192 B. In the fourth iteration it adds another 8192 B pending_bytes buffer overflows and it again written directly into file as in the second iteration. In the last iteration it adds 8192 B into pending_bytes buffer which is flushed when the files is closed.
Last example contains buffering bigger than 8192 B. Also for better explanation I added 2 more iterations.
# Case 4
with open('test.log', 'w', buffering=30000) as f:
for _ in range(7):
f.write('a'*8192)
strace output:
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384) = 16384
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384) = 16384
write(3, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 24576) = 24576
Iterations:
Add 8192 B into pending_bytes.
Add 8192 B into pending_bytes but it is more than maximal size so it is written into underlying _io.BufferedWritter() and it stays there (pending_bytes is empty now).
Add 8192 B into pending_bytes.
Add 8192 B into pending_bytes but it is more than maximal size so it tries to write into into underlying _io.BufferedWritter(). But it would exceed maximal capacity of the underlying buffer cause 16384 + 16384 > 30000 (first 16384 B are still there from iteration 2) so it first writes the old 16384 B into file and then puts those new 16384 B (from pending_bytes) into buffer. (Now again the pending_bytes buffer is empty)
Same as 3
Same as 4
Currently pending_buffer is empty and _io.BufferedWritter() contains 16384 B. In this iteration it fills pending_buffer with 8192 B. And that's it.
When the program leave with section it close the file. The process of closing follows:
Writes 8192 B from pending_buffer into _io.BufferedWriter() (it is possible cause 8192 + 16384 < 30000)
Writes (8192 + 16384=) 24576 B into file.
Close the file descriptor.
Btw currently I have no idea why is there that pending_buffer when it can use for buffering the underlying buffer from _io.BufferedWritter(). My best guess is it's there because it improve performance with files working in text mode.
I have a weird problem! I made a client / server Python code with Bluetooth in series, to send and receive byte frames (for example: [0x73, 0x87, 0x02 ....] )
Everything works, the send reception works very well !
The problem is the display of my frames, I noticed that the bytes from 0 to 127 are displayed, but from 128, it displays the byte but it adds a C2 (194) behind, for example: [0x73, 0x7F, 0x87, 0x02, 0x80 ....] == [115, 127, 135, 2, 128 ....] in hex display I would have 73 7F C2 87 2 C2 80 .. , we will notice that he adds a byte C2 from nowhere!
I think that since it is from 128! that it is due to a problem of signed (-128 to 127) / unsigned (0 to 255).
Anyone have any indication of this problem?
Thank you
0xc2 and 0xc3 are byte values that appear when encoding character values between U+0080 and U+00FF as UTF-8. Something on the transmission side is trying to send text instead of bytes, and something in the middle is (properly) converting the text to UTF-8 bytes before sending. The fix is to send bytes instead of text in the first place.
I trying to use this package in Github for string matching. My dictionary is 4 MB. When creating the Trie, I got fatal error: runtime: out of memory. I am using Ubuntu 14.04 with 8 GB of RAM and Golang version 1.4.2.
It seems the error come from the line 99 (now) here : m.trie = make([]node, max)
The program stops at this line.
This is the error:
fatal error: runtime: out of memory
runtime stack:
runtime.SysMap(0xc209cd0000, 0x3b1bc0000, 0x570a00, 0x5783f8)
/usr/local/go/src/runtime/mem_linux.c:149 +0x98
runtime.MHeap_SysAlloc(0x57dae0, 0x3b1bc0000, 0x4296f2)
/usr/local/go/src/runtime/malloc.c:284 +0x124
runtime.MHeap_Alloc(0x57dae0, 0x1d8dda, 0x10100000000, 0x8)
/usr/local/go/src/runtime/mheap.c:240 +0x66
goroutine 1 [running]:
runtime.switchtoM()
/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc208518a60 sp=0xc208518a58
runtime.mallocgc(0x3b1bb25f0, 0x4d7fc0, 0x0, 0xc20803c0d0)
/usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xc208518b10 sp=0xc208518a60
runtime.newarray(0x4d7fc0, 0x3a164e, 0x1)
/usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc208518b48 sp=0xc208518b10
runtime.makeslice(0x4a52a0, 0x3a164e, 0x3a164e, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc208518b90 sp=0xc208518b48
github.com/mf/ahocorasick.(*Matcher).buildTrie(0xc2083c7e60, 0xc209860000, 0x26afb, 0x2f555)
/home/go/ahocorasick/ahocorasick.go:104 +0x28b fp=0xc208518d90 sp=0xc208518b90
github.com/mf/ahocorasick.NewStringMatcher(0xc208bd0000, 0x26afb, 0x2d600, 0x8)
/home/go/ahocorasick/ahocorasick.go:222 +0x34b fp=0xc208518ec0 sp=0xc208518d90
main.main()
/home/go/seme/substrings.go:66 +0x257 fp=0xc208518f98 sp=0xc208518ec0
runtime.main()
/usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xc208518fe0 sp=0xc208518f98
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208518fe8 sp=0xc208518fe0
exit status 2
This is the content of the main function (taken from the same repo: test file)
var dictionary = InitDictionary()
var bytes = []byte(""Partial invoice (€100,000, so roughly 40%) for the consignment C27655 we shipped on 15th August to London from the Make Believe Town depot. INV2345 is for the balance.. Customer contact (Sigourney) says they will pay this on the usual credit terms (30 days).")
var precomputed = ahocorasick.NewStringMatcher(dictionary)// line 66 here
fmt.Println(precomputed.Match(bytes))
Your structure is awfully inefficient in terms of memory, let's look at the internals. But before that, a quick reminder of the space required for some go types:
bool: 1 byte
int: 4 bytes
uintptr: 4 bytes
[N]type: N*sizeof(type)
[]type: 12 + len(slice)*sizeof(type)
Now, let's have a look at your structure:
type node struct {
root bool // 1 byte
b []byte // 12 + len(slice)*1
output bool // 1 byte
index int // 4 bytes
counter int // 4 bytes
child [256]*node // 256*4 = 1024 bytes
fails [256]*node // 256*4 = 1024 bytes
suffix *node // 4 bytes
fail *node // 4 bytes
}
Ok, you should have a guess of what happens here: each node weighs more than 2KB, this is huge ! Finally, we'll look at the code that you use to initialize your trie:
func (m *Matcher) buildTrie(dictionary [][]byte) {
max := 1
for _, blice := range dictionary {
max += len(blice)
}
m.trie = make([]node, max)
// ...
}
You said your dictionary is 4 MB. If it is 4MB in total, then it means that at the end of the for loop, max = 4MB. It it holds 4 MB different words, then max = 4MB*avg(word_length).
We'll take the first scenario, the nicest one. You are initializing a slice of 4M of nodes, each of which uses 2KB. Yup, that makes a nice 8GB necessary.
You should review how you build your trie. From the wikipedia page related to the Aho-Corasick algorithm, each node contains one character, so there is at most 256 characters that go from the root, not 4MB.
Some material to make it right: https://web.archive.org/web/20160315124629/http://www.cs.uku.fi/~kilpelai/BSA05/lectures/slides04.pdf
The node type has a memory size of 2084 bytes.
I wrote a litte program to demonstrate the memory usage: https://play.golang.org/p/szm7AirsDB
As you can see, the three strings (11(+1) bytes in size) dictionary := []string{"fizz", "buzz", "123"} require 24 MB of memory.
If your dictionary has a length of 4 MB you would need about 4000 * 2084 = 8.1 GB of memory.
So you should try to decrease the size of your dictionary.
Set resource limit to unlimited worked for me
if ulimit -a return 0 run ulimit -c unlimited
Maybe set a real size limit to be more secure
purpose :i want to get information like iostat command can get .
I have already known that if open /proc/diskstats or /sys/block/sdX/stat there are information that :sectors read and sectors write. So if i want to get read/write bytes/s ,the following formula is right ?
read/write bytes per second:
(sectors read/write(now)-sectors read/write(last))*512 bytes/time interval
read /write operations per second :
(read/write IOs(now)+read/write merges(now)-read/write IOs(last)-read/write merges(last ))/time interval
So if i have a timer that every second control software read the information from those two files ,and then using the above formula to calculate the value .Can i get the correct answer ?
TLDR Sector is 512 bytes (octets; 1 sector is 512 bytes; each bytes is 8 bits; every bit is either 0 or 1, but not superposition of them).
"The standard sector size of 512 bytes for magnetic disks was established ....[dubious – discuss] " (c) wiki https://en.wikipedia.org/wiki/Disk_sector
How to check sector size for io statistics (in /proc) in linux:
Check how iostat tool works (it shows kilobyte per second when started as iostat 1) - it is part of sysstat package:
https://github.com/sysstat/sysstat/blob/master/iostat.c
* Read stats from /proc/diskstats.
void read_diskstats_stat(int curr)
...
/* major minor name rio rmerge rsect ruse wio wmerge wsect wuse running use aveq */
i = sscanf(line, "%u %u %s %lu %lu %lu %lu %lu %lu %lu %u %u %u %u",
&major, &minor, dev_name,
&rd_ios, &rd_merges_or_rd_sec, &rd_sec_or_wr_ios, &rd_ticks_or_wr_sec,
&wr_ios, &wr_merges, &wr_sec, &wr_ticks, &ios_pgr, &tot_ticks, &rq_ticks);
if (i == 14) {
....
sdev.rd_sectors = rd_sec_or_wr_ios;
....
sdev.wr_sectors = wr_sec;
....
* #fctr Conversion factor.
...
if (DISPLAY_KILOBYTES(flags)) {
printf(" kB_read/s kB_wrtn/s kB_read kB_wrtn\n");
*fctr = 2;
}
...
/* rrq/s wrq/s r/s w/s rsec wsec rqsz qusz await r_await w_await svctm %util */
... 4 columns skipped
cprintf_f(4, 8, 2,
S_VALUE(ioj->rd_sectors, ioi->rd_sectors, itv) / fctr,
S_VALUE(ioj->wr_sectors, ioi->wr_sectors, itv) / fctr,
So, read sector count and divide by two to get kilobyte/s (seems like 1 sector read is 0.5 kb read; 2 sector read is 1 kb read and so on). We can conclude that the sector is always 512 bytes. Same is stated in the doc, isn't it?:
internet search for "/proc/diskstats" ->
https://www.kernel.org/doc/Documentation/ABI/testing/procfs-diskstats ->
https://www.kernel.org/doc/Documentation/iostats.txt "I/O statistics fields" by ricklind from usa's ibm
Field 3 -- # of sectors read
This is the total number of sectors read successfully.
Field 7 -- # of sectors written
This is the total number of sectors written successfully.
No info about sector size here (why?). Is the source code being the best documentation (it may be)? The writer of /proc/diskstats is in kernel sources in file block/genhd.c, function diskstats_show:
http://lxr.free-electrons.com/source/block/genhd.c?v=4.4#L1149
1170 seq_printf(seqf, "%4d %7d %s %lu %lu %lu "
1171 "%u %lu %lu %lu %u %u %u %u\n",
...
1176 part_stat_read(hd, sectors[READ]),
...
1180 part_stat_read(hd, sectors[WRITE]),
Structure sectors is defined in http://lxr.free-electrons.com/source/include/linux/genhd.h?v=4.4#L82
82 struct disk_stats {
83 unsigned long sectors[2]; /* READs and WRITEs */
It is read with part_stat_read and written with __part_stat_add
http://lxr.free-electrons.com/source/include/linux/genhd.h?v=4.4#L307
Adding to the sectors counter ... is... at http://lxr.free-electrons.com/source/block/blk-core.c?v=4.4#L2264
2264 void blk_account_io_completion(struct request *req, unsigned int bytes)
2265 {
2266 if (blk_do_io_stat(req)) {
2267 const int rw = rq_data_dir(req);
2268 struct hd_struct *part;
2269 int cpu;
2270
2271 cpu = part_stat_lock();
2272 part = req->part;
2273 part_stat_add(cpu, part, sectors[rw], bytes >> 9);
2274 part_stat_unlock();
2275 }
2276 }
It uses hard-coded "bytes >> 9" to compute sector size from request size in bytes (why round down??) or for human, not no-floating-point compiler, it is the same as bytes / 512.
There is also blk_rq_sectors function (unused here...) to get sector count from request, which does the same >>9 from bytes to sectors
http://lxr.free-electrons.com/source/include/linux/blkdev.h?v=4.4#L853
841 static inline unsigned int blk_rq_bytes(const struct request *rq)
842 {
843 return rq->__data_len;
844 }
853 static inline unsigned int blk_rq_sectors(const struct request *rq)
854 {
855 return blk_rq_bytes(rq) >> 9;
856 }
Authors of FS/VFS subsystem in Linux says in reply to https://lkml.org/lkml/2015/8/17/234 "Why is SECTOR_SIZE = 512 inside kernel ?" (2015):
#define SECTOR_SHIFT 9
Message https://lkml.org/lkml/2015/8/17/269 by Theodore Ts'o:
It's cast in stone. There are too many places all over the kernel,
especially in a huge number of file systems, which assume that the
sector size is 512 bytes. So above the block layer, the sector size
is always going to be 512.
This is actually better for user space programs using
/proc/diskstats, since they don't need to know whether a particular
underlying hardware is using 512, 4k, (or if the HDD manufacturers
fantasies become true 32k or 64k) sector sizes.
For similar reason, st_blocks in struct size is always in units of 512
bytes. We don't want to force userspace to have to figure out whether
the underlying file system is using 1k, 2k, or 4k. For that reason
the units of st_blocks is always going to be 512 bytes, and this is
hard-coded in the POSIX standard.
Below is an example of a question given on my last test in a Computer Engineering course. Anyone mind explaining to me how to get the start/end addresses of each? I have listed the correct answers at the bottom...
The MSP430F2410 device has an address space of 64 KB (the basic MSP430 architecture). Fill in the table below if we know the following. The first 16 bytes of the address space (starting at the address 0x0000) is reserved for special function registers (IE1, IE2, IFG1, IFG2, etc.), the next 240 bytes is reserved for 8-bit peripheral devices, and the next 256 bytes is reserved for 16-bit peripheral devices. The RAM memory capacity is 2 Kbytes and it starts at the address 0x1100. At the top of the address space is 56KB of flash memory reserved for code and interrupt vector table.
What Start Address End Address
Special Function Registers (16 bytes) 0x0000 0x000F
8-bit peripheral devices (240 bytes) 0x0010 0x00FF
16-bit peripheral devices (256 bytes) 0x0100 0x01FF
RAM memory (2 Kbytes) 0x1100 0x18FF
Flash Memory (56 Kbytes) 0x2000 0xFFFF
For starters, don't get thrown off by what's stored in each segment - that's only going to confuse you. The problem is just asking you to figure out the hex numbering, and that's not too difficult. Here are the requirements:
64 KB total memory
The first 16 bytes of the address space (starting at the address 0x0000) is reserved for special function registers (IE1, IE2, IFG1, IFG2, etc.)
The next 240 bytes is reserved for 8-bit peripheral devices
The next 256 bytes is reserved for 16-bit peripheral devices
The RAM memory capacity is 2 Kbytes and it starts at the address 0x1100
At the top of the address space is 56KB of flash memory reserved for code and interrupt vector table.
Since each hex digit in your memory address can handle 16 values (0-F), you'll need 4 digits to display 64KB of memory (16 ^ 4 = 65536, or 64K).
You start with 16 bytes, and that covers 0x0000 - 0x000F (one full digit of your address). That means that the next segment, which starts immediately after it (8-bit devices), begins at 0x0010 (the next byte), and since it's 240 bytes long, it ends at byte 256 (240 + 16), or 0x00FF.
The next segment (16-bit devices) starts at the next byte, which is 0x0100, and is 256 bytes long - that puts the end at 0x01FF.
Then comes 2KB (2048 bytes) of RAM, but it starts at 0x1100, as the description states, instead of immediately after the previous segment, so that's your starting address. Add 2048 to that, and you get 0x18FF.
The last segment covers the upper section of the memory, so you'll have to work backwards, You know it ends at 0xFFFF (the end of the available memory), and it's 56KB long. If you convert the 56KB to hex, it's 0xDFFF. If you imagine that this segment starts at 0, That leaves 2000 unused (0xE000-0xEFFF and 0xF000-0xFFFF), so you know that this segment has to start at 0x2000 to end at the upper end of the memory space.
I hope that's more clear, though when I read over it, I don't know that it's any help at all :( Maybe that's why I'll leave teaching that concept to somebody more qualified...
#define NUM_SIZES 5
uint16_t sizes[5] = {16, 240, 256, 2 * 1024, 56 * 1024};
uint16_t address = 0;
printf("Start End\n");
for (int i = 0; i < NUM_SIZES; i++)
{
printf("0x%04X 0x%04X\n", address, address + sizes[i] - 1);
address += sizes[i];
}