Zero copy in using vmsplice/splice in Linux - linux

I am trying to get zero copy semantics working in linux using
vmsplice()/splice() but I don't see any performance improvement. This
is on linux 3.10, tried on 3.0.0 and 2.6.32. The following code tries
to do file writes, I have tried network socket writes() also, couldn't
see any improvement.
Can somebody tell what am I doing wrong ?
Has anyone gotten improvement using vmsplice()/splice() in production ?
#include <assert.h>
#include <fcntl.h>
#include <iostream>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <unistd.h>
#include <vector>
const char *filename = "Test-File";
const int block_size = 4 * 1024;
const int file_size = 4 * 1024 * 1024;
using namespace std;
int pipes[2];
vector<char *> file_data;
static int NowUsecs() {
struct timeval tv;
const int err = gettimeofday(&tv, NULL);
assert(err >= 0);
return tv.tv_sec * 1000000LL + tv.tv_usec;
}
void CreateData() {
for (int xx = 0; xx < file_size / block_size; ++xx) {
// The data buffer to fill.
char *data = NULL;
assert(posix_memalign(reinterpret_cast<void **>(&data), 4096, block_size) == 0);
file_data.emplace_back(data);
}
}
int SpliceWrite(int fd, char *buf, int buf_len) {
int len = buf_len;
struct iovec iov;
iov.iov_base = buf;
iov.iov_len = len;
while (len) {
int ret = vmsplice(pipes[1], &iov, 1, SPLICE_F_GIFT);
assert(ret >= 0);
if (!ret)
break;
len -= ret;
if (len) {
auto ptr = static_cast<char *>(iov.iov_base);
ptr += ret;
iov.iov_base = ptr;
iov.iov_len -= ret;
}
}
len = buf_len;
while (len) {
int ret = splice(pipes[0], NULL, fd, NULL, len, SPLICE_F_MOVE);
assert(ret >= 0);
if (!ret)
break;
len -= ret;
}
return 1;
}
int WriteToFile(const char *filename, bool use_splice) {
// Open and write to the file.
mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH;
int fd = open(filename, O_CREAT | O_RDWR, mode);
assert(fd >= 0);
const int start = NowUsecs();
for (int xx = 0; xx < file_size / block_size; ++xx) {
if (use_splice) {
SpliceWrite(fd, file_data[xx], block_size);
} else {
assert(write(fd, file_data[xx], block_size) == block_size);
}
}
const int time = NowUsecs() - start;
// Close file.
assert(close(fd) == 0);
return time;
}
void ValidateData() {
// Open and read from file.
const int fd = open(filename, O_RDWR);
assert(fd >= 0);
char *read_buf = (char *)malloc(block_size);
for (int xx = 0; xx < file_size / block_size; ++xx) {
assert(read(fd, read_buf, block_size) == block_size);
assert(memcmp(read_buf, file_data[xx], block_size) == 0);
}
// Close file.
assert(close(fd) == 0);
assert(unlink(filename) == 0);
}
int main(int argc, char **argv) {
auto res = pipe(pipes);
assert(res == 0);
CreateData();
const int without_splice = WriteToFile(filename, false /* use splice */);
ValidateData();
const int with_splice = WriteToFile(filename, true /* use splice */);
ValidateData();
cout << "TIME WITH SPLICE: " << with_splice << endl;
cout << "TIME WITHOUT SPLICE: " << without_splice << endl;
return 0;
}

I did a proof-of-concept some years ago where I got as 4x speedup using an optimized, specially tailored, vmsplice() code. This was measured against a generic socket/write() based solution. This blog post from natsys-lab echoes my findings. But I believe you need to have the exact right use case to get near this number.
So what are you doing wrong? Primarily I think you are measuring the wrong thing. When writing directly to a file you have 1 system call, which is write(). And you are not actually copying data (except to the kernel). When you have a buffer with data that you want to write to disk, it's not gonna get faster than that.
In you vmsplice/splice setup you are still copying you data into the kernel, but you have a total of 2 system calls vmsplice()+splice() to get it to disk. The speed being identical to write() is probably just a testament to Linux system call speed :-)
A more "fair" setup would be to write one program that read() from stdin and write() the same data to stdout. Write an identical program that simply splice() stdin into a file (or point stdout to a file when you run it). Although this setup might be too simple to really show anything.
Aside: an (undocumented?) feature of vmsplice() is that you can also use to to read data from a pipe. I used this in my old POC. It was basically just an IPC layer based on the idea of passing memory pages around using vmsplice().
Note: NowUsecs() probably overflows the int

Related

Audio Recording and Playback in C : problem with audio gain

The question essentially is how to correctly apply gain to an audio sample?
I'm programming on FreeBSD and OSS, but manipulate volume in audio sample is probably the same for other OS and applications.
I'm studying others' applications internals like ecasound (in C++) and SoX (in C) but I don't know whats wrong when I read a sample and apply gain to it : it becomes distorted and noisy. My point is to understand why it is not working to turn the volume down (gain lesser than 1).
I'm working with stereo 16 bit LE samples. Without applying gain, it works perfectly (recording and playback).
I thought that I should convert an integer sample to float; multiply by a gain factor and restore it to integer. But it is not working. And it seems to be the exact same approach for SoX in src/vol.c in function static int flow.
Below is my code (no additional libs used). The function playback is where I'm applying gain.
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include "/usr/include/sys/soundcard.h"
#include <sys/ioctl.h>
#include <sys/time.h>
#include <sys/stat.h> //man 2 chmod
#include <signal.h>
#define DEBUG 1
#define log(msg) if (DEBUG) printf("[LOG] %s\n",msg)
#define err(msg) {printf("[ERR] %s\n",msg); exit(1); }
const char *device = "/dev/dsp3.1"; //Audio device
char *rawFile = "/tmp/raw-file.wav"; //Raw file to record and playback
int fragmentSize = 256;
int b_continue = 1;
void signalHandler(int sigNum){
log("Signal captured");
b_continue = 0;
}
void configDevice(int fdDsp){
int ossCapabilities = 0;
if(fdDsp == -1)
err("can't open device");
if( ioctl(fdDsp, SNDCTL_DSP_GETCAPS, &ossCapabilities) == -1)
err("unsupported: SNDCTL_DSP_GETCAPS");
/*
* http://www.opensound.com/pguide/audio2.html
*/
if(ossCapabilities & DSP_CAP_TRIGGER != DSP_CAP_TRIGGER){
err("Triggering of recording/playback is not possible with this OSS device.");
}
if(ossCapabilities & DSP_CAP_REALTIME != DSP_CAP_REALTIME){
err("No DSP_CAP_REALTIME.");
}
if(ioctl(fdDsp, SNDCTL_DSP_SETDUPLEX, &ossCapabilities) == -1)
err("can't SNDCTL_DSP_SETDUPLEX");
if(ossCapabilities & DSP_CAP_DUPLEX != DSP_CAP_DUPLEX)
err("can't DSP_CAP_DUPLEX");
int format = AFMT_S16_LE; //set format
if(ioctl(fdDsp, SNDCTL_DSP_SETFMT, &format ) == -1){
err("Error setting format.");
}
int channels = 1; //mono=0 stereo=1
if(ioctl(fdDsp, SNDCTL_DSP_STEREO, &channels ) == -1){
err("Error setting channels." );
}
// FREQUENCY RATE
int speed = 44100;
if(ioctl(fdDsp, SNDCTL_DSP_SPEED, &speed ) == -1){
err("Error setting speed.");
}
// FRAGMENT SIZE
if(ioctl(fdDsp, SNDCTL_DSP_SETBLKSIZE, &fragmentSize) == -1){ //normalmente 2048 bits
err("Cannot SNDCTL_DSP_SETBLKSIZE.");
}
}
void record(){
int fdDsp = open(device, O_RDONLY);
configDevice(fdDsp);
//create file for writing
const int fdOutput = open(rawFile, O_WRONLY | O_CREAT, S_IWUSR | S_IRUSR);
if(fdOutput ==-1)
err("can't open file to write");
log("Recording...");
do{
// Triggers recording
int enableBits = PCM_ENABLE_INPUT;
if(ioctl(fdDsp, SNDCTL_DSP_SETTRIGGER, &enableBits) == -1)
err("Can't record: SNDCTL_DSP_SETTRIGGER");
int *buf[fragmentSize];
read(fdDsp, buf, fragmentSize);
write(fdOutput, buf, fragmentSize);
} while(b_continue == 1);
close(fdOutput);
close(fdDsp);
}
void playback(){
log("Opening file:");
log(rawFile);
log("On device:");
log(device);
int fdDsp = open(device, O_WRONLY);
configDevice(fdDsp);
const int fdInput = open(rawFile, O_RDONLY);
if(fdInput ==-1)
err("can't open file");
log("Playing...");
int eof = 0;
do{
// TRIGGERs PLAYBACK
int enableBits = PCM_ENABLE_OUTPUT;
if(ioctl(fdDsp, SNDCTL_DSP_SETTRIGGER, &enableBits) == -1){
err("Cannot SNDCTL_DSP_SETTRIGGER.");
}
int buf[fragmentSize];
eof = read(fdInput, buf, fragmentSize); //bytes read or -1 if EOF
// audio processing:
for(int i=0;i<fragmentSize;i++){
// learning how to get left and right channels from buffer
int l = (buf)[i] & 0xffff;
int r = ((buf)[i] >> 16) & 0xffff ;
// FIXME: it is causing distortion:
float fl = l;
float fr = r;
fl *= 1.0;
fr *= 0.3; //if different than 1, sounds distorted and noisy
l = fl;
r = fr;
// OK: unite Left and Right channels again
int lr = (l ) | (r << 16);
// OK: other options to mix these two channels:
int lleft = l; //Just the left channel
int rright = (r << 16); //Just the right channel
int lmono = (l << 16) | l; //Left ch. on both channels
int rmono = (r << 16) | r; //Right ch. on both channels
// the output:
(buf)[i] = lr;
}
write(fdDsp, buf, fragmentSize);
if(b_continue == 0) break;
} while(eof > 0);
close(fdInput);
close(fdDsp);
}
int main(int argc, char *argv[])
{
signal(SIGINT, signalHandler);
log("Ctrl^C to stop recording/playback");
record();
b_continue = 1; playback();
log("Stopped.");
return 0;
}
UPDATE:
As pointed out by CL, I was using the wrong type and the last parameter of read()/write() is greater than the size of the buffer.
So, in FreeBSD I changed the buffer type to int16_t (short) defined in #include <stdint.h> .
Now I can correctly apply a gain as desired:
float fl = l;
float fr = r;
fl *= 1.0f;
fr *= 1.5f;
l = fl;
r = fr;
I'll accept CL's answer.
Now the audio processing loop is working with one sample per time (left and right interleaved).
Updated code:
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include "/usr/include/sys/soundcard.h"
#include <sys/ioctl.h>
#include <sys/time.h>
#include <sys/stat.h> //man 2 chmod
#include <signal.h>
#include <stdint.h> //has type int16_t (short)
#define DEBUG 1
#define log(msg) if (DEBUG) printf("[LOG] %s\n",msg)
#define err(msg) {printf("[ERR] %s\n",msg); exit(1); }
const char *device = "/dev/dsp3.1"; //Audio device
char *rawFile = "/tmp/stereo.wav"; //Raw file to record and playback
int fragmentSize = 256;
int b_continue = 1;
void signalHandler(int sigNum){
log("Signal captured");
b_continue = 0;
}
void configDevice(int fdDsp){
int ossCapabilities = 0;
if(fdDsp == -1)
err("can't open device");
if( ioctl(fdDsp, SNDCTL_DSP_GETCAPS, &ossCapabilities) == -1)
err("unsupported: SNDCTL_DSP_GETCAPS");
/*
* http://www.opensound.com/pguide/audio2.html
*/
if(ossCapabilities & DSP_CAP_TRIGGER != DSP_CAP_TRIGGER){
err("Triggering of recording/playback is not possible with this OSS device.");
}
if(ossCapabilities & DSP_CAP_REALTIME != DSP_CAP_REALTIME){
err("No DSP_CAP_REALTIME.");
}
if(ioctl(fdDsp, SNDCTL_DSP_SETDUPLEX, &ossCapabilities) == -1)
err("can't SNDCTL_DSP_SETDUPLEX");
if(ossCapabilities & DSP_CAP_DUPLEX != DSP_CAP_DUPLEX)
err("can't DSP_CAP_DUPLEX");
int format = AFMT_S16_LE; //set format
if(ioctl(fdDsp, SNDCTL_DSP_SETFMT, &format ) == -1){
err("Error setting format.");
}
int channels = 1; //mono=0 stereo=1
if(ioctl(fdDsp, SNDCTL_DSP_STEREO, &channels ) == -1){
err("Error setting channels." );
}
// FREQUENCY RATE
int speed = 44100;
if(ioctl(fdDsp, SNDCTL_DSP_SPEED, &speed ) == -1){
err("Error setting speed.");
}
// FRAGMENT SIZE
if(ioctl(fdDsp, SNDCTL_DSP_SETBLKSIZE, &fragmentSize) == -1){ //normalmente 2048 bits
err("Cannot SNDCTL_DSP_SETBLKSIZE.");
}
}
void record(){
int fdDsp = open(device, O_RDONLY);
configDevice(fdDsp);
//create file for writing
const int fdOutput = open(rawFile, O_WRONLY | O_CREAT, S_IWUSR | S_IRUSR);
if(fdOutput ==-1)
err("can't open file to write");
log("Recording...");
do{
// Triggers recording
int enableBits = PCM_ENABLE_INPUT;
if(ioctl(fdDsp, SNDCTL_DSP_SETTRIGGER, &enableBits) == -1)
err("Can't record: SNDCTL_DSP_SETTRIGGER");
// Wrong:
// int *buf[fragmentSize];
// read(fdDsp, buf, fragmentSize);
// write(fdOutput, buf, fragmentSize);
int16_t *buf[fragmentSize/sizeof (int16_t)];
read(fdDsp, buf, fragmentSize/sizeof (int16_t));
write(fdOutput, buf, fragmentSize/sizeof (int16_t));
} while(b_continue == 1);
close(fdOutput);
close(fdDsp);
}
void playback(){
log("Opening file:");
log(rawFile);
log("On device:");
log(device);
int fdDsp = open(device, O_WRONLY);
configDevice(fdDsp);
const int fdInput = open(rawFile, O_RDONLY);
if(fdInput ==-1)
err("can't open file");
log("Playing...");
int eof = 0;
do{
// TRIGGERs PLAYBACK
int enableBits = PCM_ENABLE_OUTPUT;
if(ioctl(fdDsp, SNDCTL_DSP_SETTRIGGER, &enableBits) == -1){
err("Cannot SNDCTL_DSP_SETTRIGGER.");
}
//Wrong buffer type (too large) and wrong last parameter for read():
// int buf[fragmentSize];
// eof = read(fdInput, buf, fragmentSize);
int16_t buf[fragmentSize/sizeof (int16_t)];
eof = read(fdInput, buf, fragmentSize/sizeof (int16_t));
// audio processing:
for(int i=0;i<fragmentSize/sizeof (int16_t);i++){
int16_t l = buf[i];
int16_t r = buf[i+1];
// Using int16_t (short) buffer, gain works but stereo is inverted with factor >= 1.4f
float fl = l;
float fr = r;
fl *= 2.0f;
fr *= 3.0f;
l = fl;
r = fr;
// the output:
(buf)[i] = l;
i++;
(buf)[i] = r;
}
// write(fdDsp, buf, fragmentSize); //wrong
write(fdDsp, buf, fragmentSize/sizeof (int16_t));
if(b_continue == 0) break;
} while(eof > 0);
close(fdInput);
close(fdDsp);
}
int main(int argc, char *argv[])
{
signal(SIGINT, signalHandler);
log("Ctrl^C to stop recording/playback");
record();
b_continue = 1; playback();
log("Stopped.");
return 0;
}
Thanks,
The last parameter of read()/write() is the number of bytes, but an entry in buf[] has more than one byte.
In the two's complement representation of binary numbers, negative values are (or must be) sign extended, i.e., the most significant bits are ones. In this code, neither extracting L/R channels nor combining them works correctly for negative samples.
The easiest way of handling negative samples would be to use one array entry per sample, i.e., short int.

Named Pipe, Communication between 2 children

I have a problem with my code. I want to make communication between 2 children process. One of them is a server, which opens a file and sends each letter to the second process. The second process is counting letters and it should make a new file and save results. I have problems with the last step because the first process gonna finish faster than the second, what causes the end of the program. I have no idea how fix it. Looking for some tips :).
Here you got result.
My code:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <string.h>
//stale
#define FIFO "my_fifo"
#define SIZE 26
//zmienne globalne
int desk; //deskryptor pliku
int tab[SIZE];
//prototypy funkcji
void parentKillAll();
void server(FILE * file);
void client();
void cleanUp(FILE * file);
int checkEntryData(int argc, char *argv);
void replaceTabWithZero(int * tab);
void countLetters(int * tab, char ch);
void saveResults(int * tab, char *title);
void showTab(int * tab);
int main(int argc, char *argv[]) {
if (!checkEntryData(argc, argv[1]))
return 1;
replaceTabWithZero(tab);
FILE *file = fopen(argv[1], "r");
umask(0);
mkfifo(FIFO, 0666);
if (file) {
if (fork() == 0) {
server(file);
exit(0);
} else if (fork() == 0) {
client();
saveResults(tab, strcat(argv[1], "Result"));
showTab(tab);
exit(0);
} else {
cleanUp(file);
parentKillAll();
}
} else {
perror("Error");
}
return 0;
}
void parentKillAll() {
sleep(1);
kill(0, SIGKILL);
exit(0);
}
void server(FILE * file) {
char ch;
while ((ch = fgetc(file)) != EOF) {
desk = open(FIFO, O_WRONLY);
write(desk, &ch, 1);
}
}
void client() {
char ch;
while (1) {
desk = open(FIFO, O_RDONLY);
read(desk, &ch, 1);
countLetters(tab, ch);
printf("%c", ch);
}
}
void cleanUp(FILE *file) {
wait(0);
fclose(file);
close(desk);
}
int checkEntryData(int argc, char *argv) {
if (argc < 2) {
fprintf(stderr, "Nie poprawna ilosc argumentow\n");
return 0;
}
if (access(argv, F_OK)) {
fprintf(stderr, "Podany plik \'%s\' nie istnieje\n", argv);
return 0;
}
if (access(argv, R_OK)) {
fprintf(stderr, "Brak uprawnien do odczytu pliku \'%s\'\n", argv);
return 0;
}
return 1;
}
void replaceTabWithZero(int * tab) {
for (int i = 0; i < SIZE; i++)
tab[i] = 0;
}
void countLetters(int *tab, char ch) {
int chVal = ch;
if (chVal > 92)
chVal -= 32;
if (chVal > 64 && chVal < 91)
tab[chVal-65] += 1;
}
void saveResults(int *tab, char * title) {
FILE *plik = fopen(title, "w");
if (plik) {
for (int i = 0; i < SIZE; i++)
fprintf(plik, "%c - %d\n", (i+97), tab[i]);
} else {
perror("Error");
}
fclose(plik);
}
void showTab(int * tab) {
for (int i = 0; i < SIZE; i++)
printf("\n%d", tab[i]);
}
The real problem is that the client process can never finish, because it runs an infinite while(1) loop without any exit conditions.
You should rewrite it so that it exits after reading all available data:
void client() {
char ch;
// Open the fifo only once, instead of once per character
desk = open(FIFO, O_RDONLY);
// Loop until there is no more data to read
while(read(desk, &ch, 1) > 0) {
countLetters(tab, ch);
printf("%c", ch);
}
}
This is technically sufficient to make it work, but you should also look into a series of other issues:
You should have two wait(0) calls so that you wait for both processes, and you shouldn't try to kill anything.
The server process should only be opening the fifo once, not once per character.
You should be comparing fgetc output to EOF before forcing the value into a char. Since you do it after, running your program on a ISO-8859-1 terminal will cause it to confuse EOF and the letter ΓΏ
You are using strcat on argv[1], even though you don't know how much space that array has. You should use your own buffer of a known length.
You should check the return value of all your system calls to ensure they succeed. Checking with access and then assuming it'll be fine is not as good since calls can fail for other reasons.
Canonical Unix behavior is to exit with 0 for success, and >= 1 for error.
It's good practice to use a larger buffer (e.g. 65536 bytes instead of 1) when using read/write directly. stdio functions like fgetc already uses a larger buffer behind the scenes.
Using a named pipe obviously works, but since you spawn both processes it would be more natural to use an unnamed one.

How to write to multiple files on different disks simultaneously in one thread with DMA?

I use aio to write multiple files on different disk in one thread. When I use buffered writing, IO processing is concurrent. But cpu loads is very high. When I open files with DIRECT flag, IO processing isn't concurrent.
How to write to multiple files on different disks simultaneously in one thread with DMA?
#include <malloc.h>
#include <stdio.h>
#include <string.h>
#include <iostream>
#include <sstream>
#include <inttypes.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/syscall.h>
#include <linux/aio_abi.h>
using namespace std;
long double timeDiff(timespec start, timespec end) {
const long double s = start.tv_sec + start.tv_nsec * 1.0e-9;
const long double e = end.tv_sec + end.tv_nsec * 1.0e-9;
return e - s;
}
// nr: maximum number of requests that can simultaneously reside in the context.
inline int io_setup(unsigned nr, aio_context_t *ctxp) {
return syscall(__NR_io_setup, nr, ctxp);
}
inline int io_destroy(aio_context_t ctx) {
return syscall(__NR_io_destroy, ctx);
}
// Every I/O request that is submitted to
inline int io_submit(aio_context_t ctx, long nr, struct iocb **iocbpp) {
return syscall(__NR_io_submit, ctx, nr, iocbpp);
}
// For every completed I/O request kernel creates an io_event structure.
// minimal number of events one wants to get.
// maximum number of events one wants to get.
inline int io_getevents(aio_context_t ctx, long min_nr, long max_nr,
struct io_event *events, struct timespec *timeout) {
return syscall(__NR_io_getevents, ctx, min_nr, max_nr, events, timeout);
}
int main(int argc, char *argv[]) {
// prepare data
const unsigned int kAlignment = 4096;
const long data_size = 1600 * 1024 * 12 / 8;
//const long data_size = 2448 * 1344 * 12 / 8;
void * data = memalign(kAlignment, data_size);
memset(data, 0, data_size);
//for (int i = 0; i < data_size; ++i)
// data[i] = 'A';
// prepare fd
//const int file_num = 3;
const int file_num = 2;
int fd_arr[file_num];
for (int i = 0; i < file_num; ++i) {
ostringstream filename;
if (i == 0) {
//filename << "/data/test";
filename << "/test";
} else {
filename << "/data" << i << "/test";
}
//filename << "/data/test" << i;
int fd = open(filename.str().c_str(), O_WRONLY | O_NONBLOCK | O_CREAT | O_DIRECT | O_APPEND, 0644);
//int fd = open(filename.str().c_str(), O_WRONLY | O_NONBLOCK | O_CREAT | O_DIRECT, 0644);
//int fd = open(filename.str().c_str(), O_WRONLY | O_NONBLOCK | O_CREAT, 0644);
if (fd < 0) {
perror("open");
return -1;
}
fd_arr[i] = fd;
}
aio_context_t ctx;
struct io_event events[file_num];
int ret;
ctx = 0;
ret = io_setup(1000, &ctx);
if (ret < 0) {
perror("io_setup");
return -1;
}
struct iocb cbs[file_num];
for (int i = 0; i < file_num; ++i) {
memset(&cbs[i], 0, sizeof(cbs[i]));
}
struct iocb * cbs_pointer[file_num];
for (int i = 0; i < file_num; ++i) {
/* setup I/O control block */
cbs_pointer[i] = &cbs[i];
cbs[i].aio_fildes = fd_arr[i];
cbs[i].aio_lio_opcode = IOCB_CMD_PWRITE; // IOCV_CMD
cbs[i].aio_nbytes = data_size;
}
timespec tStart, tCurr;
clock_gettime(CLOCK_REALTIME, &tStart);
const int frame_num = 10000;
for (int k = 0; k < frame_num; ++k) {
for (int i = 0; i < file_num; ++i) {
/* setup I/O control block */
cbs[i].aio_buf = (uint64_t)data;
//cbs[i].aio_offset = k * data_size;
}
ret = io_submit(ctx, file_num, cbs_pointer);
if (ret < 0) {
perror("io_submit");
return -1;
}
/* get reply */
ret = io_getevents(ctx, file_num, file_num, events, NULL);
//printf("events: %d, k: %d\n", ret, k);
}
clock_gettime(CLOCK_REALTIME, &tCurr);
cout << "frame: " << frame_num << " time: " << timeDiff(tStart, tCurr) << endl;
ret = io_destroy(ctx);
if (ret < 0) {
perror("io_destroy");
return -1;
}
// close fd
for (int i = 0; i < file_num; ++i) {
fsync(fd_arr[i]);
close(fd_arr[i]);
}
return 0;
}
Linux can make writes actually async if and only if the physical extents being written are allocated on the disc already. Otherwise it has to take a mutex and do the allocation first, thus everything becomes synchronous.
Note that truncating the file to a new length usually doesn't actually allocate the underlying extents. You need to prewrite the contents first. Thereafter, rewriting the same extents will now be done async and thus become concurrent.
As you might be gathering, async file i/o on Linux is not great, though it keeps on getting better over time. Windows or FreeBSD have far superior implementations. Even OS X is not terrible. Use any of those instead.

Decrease in Random read IOPs on NVME SSD if requests issued over small region

(TL;DR) On NVME SSDs (Intel p3600 as well as Avant), I am seeing decrease in the IOPS if I issue random reads over a small subset of the disk instead of the entire disk.
While reading the same offset over and over, the IOPS are about 36-40K for 4k blocksize. The IOPS gradually increase as I grow the region over which random reads are being issued. The program (seen below) uses asynchronous IO on Linux to submit the read requests.
Disk Range(in 4k blocks), IOPS
0, 38833
1, 68596
10, 76100
30, 80381
40, 113647
50, 148205
100, 170374
200, 239798
400, 270197
800, 334767
OS : Linux 4.2.0-35-generic
SSD : Intel P3600 NVME Flash
What could be causing this problem ?
The program can be run as follows
$ for i in 0 1 10 30 40 50 100 200 400 800
do
<program_name> /dev/nvme0n1 10 $i
done
and validate if you also see the increasing pattern of IOPS seen above
/**
* $ g++ <progname.cpp> -o progname -std=c++11 -lpthread -laio -O3
* $ progname /dev/nvme0n1 10 100
*/
#include <random>
#include <libaio.h>
#include <stdlib.h>//malloc, exit
#include <future> //async
#include <unistd.h> //usleep
#include <iostream>
#include <sys/time.h> // gettimeofday
#include <vector>
#include <fcntl.h> // open
#include <errno.h>
#include <sys/types.h> // open
#include <sys/stat.h> // open
#include <cassert>
#include <semaphore.h>
io_context_t ioctx;
std::vector<char*> buffers;
int fd = -1;
sem_t sem;
constexpr int numPerRound = 20;
constexpr int numRounds = 100000;
constexpr int MAXEVENT = 10;
constexpr size_t BLKSIZE = 4096;
constexpr int QDEPTH = 200;
off_t startBlock = 0;
off_t numBlocks = 100;
const int numSubmitted = numRounds * numPerRound;
void DoGet()
{
io_event eventsArray[MAXEVENT];
int numCompleted = 0;
while (numCompleted != numSubmitted)
{
bzero(eventsArray, MAXEVENT * sizeof(io_event));
int numEvents;
do {
numEvents = io_getevents(ioctx, 1, MAXEVENT, eventsArray, nullptr);
} while (numEvents == -EINTR);
for (int i = 0; i < numEvents; i++)
{
io_event* ev = &eventsArray[i];
iocb* cb = (iocb*)(ev->data);
assert(ev->res2 == 0);
assert(ev->res == BLKSIZE);
sem_post(&sem); // free ioctx
}
numCompleted += numEvents;
}
std::cout << "completed=" << numCompleted << std::endl;
}
int main(int argc, char* argv[])
{
if (argc == 1) {
std::cout << "usage <nvme_device_name> <start_4k_block> <num_4k_blocks>" << std::endl;
exit(1);
}
char* deviceName = argv[1];
startBlock = atoll(argv[2]);
numBlocks = atoll(argv[3]);
int ret = 0;
ret = io_queue_init(QDEPTH, &ioctx);
assert(ret == 0);
ret = sem_init(&sem, 0, QDEPTH);
assert(ret == 0);
auto DoGetFut = std::async(std::launch::async, DoGet);
// preallocate buffers
for (int i = 0; i < QDEPTH; i++)
{
char* buf ;
ret = posix_memalign((void**)&buf, 4096, BLKSIZE);
assert(ret == 0);
buffers.push_back(buf);
}
fd = open("/dev/nvme0n1", O_DIRECT | O_RDONLY);
assert(fd >= 0);
off_t offset = 0;
struct timeval start;
gettimeofday(&start, 0);
std::mt19937 generator (getpid());
// generate random offsets within [startBlock, startBlock + numBlocks]
std::uniform_int_distribution<off_t> offsetgen(startBlock, startBlock + numBlocks);
for (int j = 0; j < numRounds; j++)
{
iocb mycb[numPerRound];
iocb* posted[numPerRound];
bzero(mycb, sizeof(iocb) * numPerRound);
for (int i = 0; i < numPerRound; i++)
{
// same buffer may get used in 2 different async read
// thats ok - not validating content in this program
char* iobuf = buffers[i];
iocb* cb = &mycb[i];
offset = offsetgen(generator) * BLKSIZE;
io_prep_pread(cb, fd, iobuf, BLKSIZE, offset);
cb->data = iobuf;
posted[i] = cb;
sem_wait(&sem); // wait for ioctx to be free
}
int ret = 0;
do {
ret = io_submit(ioctx, numPerRound, posted);
} while (ret == -EINTR);
assert(ret == numPerRound);
}
DoGetFut.wait();
struct timeval end;
gettimeofday(&end, 0);
uint64_t diff = ((end.tv_sec - start.tv_sec) * 1000000) + (end.tv_usec - start.tv_usec);
io_queue_release(ioctx);
std::cout
<< "ops=" << numRounds * numPerRound
<< " iops=" << (numRounds * numPerRound *(uint64_t)1000000)/diff
<< " region-size=" << (numBlocks * BLKSIZE)
<< std::endl;
}
Surely it is to do with the structure of the memory. Internally this drive is built from many memory chips and may have multiple memory buses internally. If you do requests across a small range all the requests will resolve to a single or few chips and will have to be queued. If you access across the whole device then the multiple request are across many internal chips and buses and can be run asynchronously so will provide more throughput.

I2C_SLAVE ioctl purpose

I am writing code for implementing a simple i2c read/write function using the general linux i2c driver linux/i2c-dev.h
I am confused about the ioctl : I2C_SLAVE
The kernel documentation states as follows :
You can do plain i2c transactions by using read(2) and write(2) calls.
You do not need to pass the address byte; instead, set it through
ioctl I2C_SLAVE before you try to access the device
However I am using the ioctl I2C_RDWR where I again set the slave address using i2c_msg.addr.
The kernel documentation also mentions the following :
Some ioctl() calls are for administrative tasks and are handled by
i2c-dev directly. Examples include I2C_SLAVE
So is it must to use the ioctl I2C_SLAVE? If so do I need to set it just once or every time I perform a read and write?
If I had an i2c device I could have just tested the code on the device and would not have bothered you guys but unfortunately I don't have one right now.
Thanks for the help.
There are three major methods of communicating with i2c devices from userspace.
1. IOCTL I2C_RDWR
This method allows for simultaneous read/write and sending an uninterrupted sequence of message. Not all i2c devices support this method.
Before performing i/o with this method, you should check whether the device supports this method using an ioctl I2C_FUNCS operation.
Using this method, you do not need to perform an ioctl I2C_SLAVE operation -- it is done behind the scenes using the information embedded in the messages.
2. IOCTL SMBUS
This method of i/o is more powerful but the resulting code is more verbose. This method can be used if the device does not support the I2C_RDWR method.
Using this method, you do need to perform an ioctl I2C_SLAVE operation (or, if the device is busy, an I2C_SLAVE_FORCE operation).
3. SYSFS I/O
This method uses the basic file i/o system calls read() and write(). Uninterrupted sequential operations are not possible using this method. This method can be used if the device does not support the I2C_RDWR method.
Using this method, you do need to perform an ioctl I2C_SLAVE operation (or, if the device is busy, an I2C_SLAVE_FORCE operation).
I can't think of any situation when this method would be preferable to others, unless you need the chip to be treated like a file.
Full IOCTL Example
I haven't tested this example, but it shows the conceptual flow of writing to an i2c device.-- automatically detecting whether to use the ioctl I2C_RDWR or smbus technique.
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <errno.h>
#include <string.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <linux/i2c.h>
#include <linux/i2c-dev.h>
#include <sys/ioctl.h>
#define I2C_ADAPTER "/dev/i2c-0"
#define I2C_DEVICE 0x00
int i2c_ioctl_write (int fd, uint8_t dev, uint8_t regaddr, uint16_t *data, size_t size)
{
int i, j = 0;
int ret;
uint8_t *buf;
// the extra byte is for the regaddr
size_t buff_size = 1 + size;
buf = malloc(buff_size);
if (buf == NULL) {
return -ENOMEM;
}
buf[j ++] = regaddr;
for (i = 0; i < size / sizeof(uint16_t); i ++) {
buf[j ++] = (data[i] & 0xff00) >> 8;
buf[j ++] = data[i] & 0xff;
}
struct i2c_msg messages[] = {
{
.addr = dev,
.buf = buf,
.len = buff_size,
},
};
struct i2c_rdwr_ioctl_data payload = {
.msgs = messages,
.nmsgs = sizeof(messages) / sizeof(messages[0]),
};
ret = ioctl(fd, I2C_RDWR, &payload);
if (ret < 0) {
ret = -errno;
}
free (buf);
return ret;
}
int i2c_ioctl_smbus_write (int fd, uint8_t dev, uint8_t regaddr, uint16_t *data, size_t size)
{
int i, j = 0;
int ret;
uint8_t *buf;
buf = malloc(size);
if (buf == NULL) {
return -ENOMEM;
}
for (i = 0; i < size / sizeof(uint16_t); i ++) {
buf[j ++] = (data[i] & 0xff00) >> 8;
buf[j ++] = data[i] & 0xff;
}
struct i2c_smbus_ioctl_data payload = {
.read_write = I2C_SMBUS_WRITE,
.size = I2C_SMBUS_WORD_DATA,
.command = regaddr,
.data = (void *) buf,
};
ret = ioctl (fd, I2C_SLAVE_FORCE, dev);
if (ret < 0)
{
ret = -errno;
goto exit;
}
ret = ioctl (fd, I2C_SMBUS, &payload);
if (ret < 0)
{
ret = -errno;
goto exit;
}
exit:
free(buf);
return ret;
}
int i2c_write (int fd, uint8_t dev, uint8_t regaddr, uint16_t *data, size_t size)
{
unsigned long funcs;
if (ioctl(fd, I2C_FUNCS, &funcs) < 0) {
return -errno;
}
if (funcs & I2C_FUNC_I2C) {
return i2c_ioctl_write (fd, dev, regaddr, data, size);
} else if (funcs & I2C_FUNC_SMBUS_WORD_DATA) {
return i2c_ioctl_smbus_write (fd, dev, regaddr, data, size);
} else {
return -ENOSYS;
}
}
int parse_args (uint8_t *regaddr, uint16_t *data, size_t size, char *argv[])
{
char *endptr;
int i;
*regaddr = (uint8_t) strtol(argv[1], &endptr, 0);
if (errno || endptr == argv[1]) {
return -1;
}
for (i = 0; i < size / sizeof(uint16_t); i ++) {
data[i] = (uint16_t) strtol(argv[i + 2], &endptr, 0);
if (errno || endptr == argv[i + 2]) {
return -1;
}
}
return 0;
}
void usage (int argc, char *argv[])
{
fprintf(stderr, "Usage: %s regaddr data [data]*\n", argv[0]);
fprintf(stderr, " regaddr The 8-bit register address to write to.\n");
fprintf(stderr, " data The 16-bit data to be written.\n");
exit(-1);
}
int main (int argc, char *argv[])
{
uint8_t regaddr;
uint16_t *data;
size_t size;
int fd;
int ret = 0;
if (argc < 3) {
usage(argc, argv);
}
size = (argc - 2) * sizeof(uint16_t);
data = malloc(size);
if (data == NULL) {
fprintf (stderr, "%s.\n", strerror(ENOMEM));
return -ENOMEM;
}
if (parse_args(&regaddr, data, size, argv) != 0) {
free(data);
usage(argc, argv);
}
fd = open(I2C_ADAPTER, O_RDWR | O_NONBLOCK);
ret = i2c_write(fd, I2C_DEVICE, regaddr, data);
close(fd);
if (ret) {
fprintf (stderr, "%s.\n", strerror(-ret));
}
free(data);
return ret;
}
If you use the read() and write() methods, calling ioctl with I2C_SLAVE once is enough. You can also use I2C_SLAVE_FORCE if the device is already in use.
However I haven't yet found a consistent way to read specific registers for every device using the read()/write() methods.
I'm not too sure if this helps because I don't use ioctl I2C_RDWR but I've been using the following code with success:
int fd;
fd = open("/dev/i2c-5", O_RDWR);
ioctl(fd, I2C_SLAVE_FORCE, 0x20);
i2c_smbus_write_word_data(fd, ___, ___);
i2c_smbus_read_word_data(fd, ___);
All I do is set I2C_SLAVE_FORCE once at the beginning and I can read and write as much as I want to after that.
PS - This is just a code sample and obviously you should check the returns of all of these functions. I'm using this code to communicate with a digital I/O chip. The two i2c_* functions are just wrappers that call ioctl(fd, I2C_SMBUS, &args); where args is a struct i2c_smbus_ioctl_data type.
For the interested, SLAVE_FORCE is used when the device in question is already being managed by a kernel driver. (i2cdetect will show UU for that address)

Resources