man 7 mq_overview says that the POSIX "...message queues on the system can be viewed and manipulated using the commands usually used for files (e.g., ls(1) and rm(1))."
For example I was able to read using a mqd_t as a file descriptor as follows:
#include <iostream>
#include <fcntl.h>
#include <mqueue.h>
#include <unistd.h>
#include <stdlib.h>
int main(int argc, char **argv) {
if (argc != 2) {
std::cout << "Usage: mqgetinfo </mq_name>\n";
exit(1);
}
mqd_t mqd = mq_open(argv[1], O_RDONLY);
struct mq_attr attr;
mq_getattr(mqd, &attr);
std::cout << argv[1] << " attributes:"
<< "\nflag: " << attr.mq_flags
<< "\nMax # of msgs: " << attr.mq_maxmsg
<< "\nMax msg size: " << attr.mq_msgsize
<< "\nmsgs now in queue: " << attr.mq_curmsgs << '\n';
// Get the queue size in bytes, and any notification info:
char buf[1024];
int n = read(mqd, buf, 1023);
buf[n] = '\0';
std::cout << "\nFile /dev/mqueue" << argv[1] << ":\n"
<< buf << '\n';
mq_close(mqd);
}
Running this on msg queue /myq when it contains 5 msgs, 549 bytes gives:
$ g++ mqgetinfo.cc -o mqgetinfo -lrt
$ ./mqgetinfo /myq
/myq attributes:
flag: 0
Max # of msgs: 10
Max msg size: 8192
msgs now in queue: 5
File /dev/mqueue/myq:
QSIZE:549 NOTIFY:0 SIGNO:0 NOTIFY_PID:0
$
Also:
$ !cat
cat /dev/mqueue/myq
QSIZE:549 NOTIFY:0 SIGNO:0 NOTIFY_PID:0
So the file /dev/mqueue/myq has some info associated with the msg queue.
My question is: Where is the queue itself, i.e., where are the 549 bytes? I'm guessing they are in some list-type data structure internal to the kernel, but I don't see this mentioned in the man pages etc, and wondering how to find out about that..
Since the internal handling of message queues are implementation specific (not part of the standard, as it only specifies the programming interface and behaviour), I recommend you to have a look into the linux kernel source file ipc/mqueue.c and pay special attention to mqueue_create() and msg_insert() functions, as it's a good place to get started if you want to understand how message queues are implemented in the linux kernel.
Related
I need help understanding what is wrong with my code or system. For some reason, when compiled and ran in WSL(Windows Subsystem for Linux), the following code generates a wrong result. When ran directly in the windows command terminal, it correctly generates the final output as HNDPNV and in WSL generates PNV. The airports file is simply a series of 50 lines, with 3 letters per line. g++ version is 7.5.0 in ubuntu 18.04 WSL and g++ windows version is 9.2.0.
compilation flags are: g++ -std=c++17 -Wall -Wextra -o test test.cpp
#include <iostream>
#include <cstdlib>
#include <string>
#include <fstream>
using namespace std;
int main() {
string airport_designations[50];
string air_companies_designations[20];
ifstream airports;
ifstream aircompanies;
string name = "apple";
string name2 = "orange";
airports.open("Airports.txt");
for (int i = 0; i < 50; i++){
getline(airports, airport_designations[i]);
cout << airport_designations[i] << endl;
}
airports.close();
aircompanies.open("Companies.txt");
for (int i = 0; i < 20; i++){
getline(aircompanies, air_companies_designations[i]);
cout << air_companies_designations[i] << endl;
}
aircompanies.close();
name = airport_designations[4];
cout << name << endl;
name += airport_designations[8];
cout << name << endl;
}
The line break on windows is \r\n,
On Linux \n.
To get a consistent conclusion, you need to replace all \r\ n in the Airports.txt file with \n.
I am trying to accomplish the following objectives:
Write video from my Raspberry Pi Camera to disk without any interference from streaming
Stream the same video through the network optimizing latency
It is important streaming does not interfere with the video being written to disk, since network connection may be unstable, such as WiFi router may be out of range, etc.
To accomplish that, the first thing I have tried is the following:
#Receiver side
FPS="30"
netcat -l -p 5000 | mplayer -vf scale -zoom -xy 1280 -fps $FPS -cache-min 50 -cache 1024 - &
#RPi side
FPS="30"
mkfifo netcat_fifo
raspivid -t 0 -md 5 -fps $FPS -o - | tee --output-error=warn netcat_fifo > $video_out &
cat netcat_fifo | netcat -v 192.168.0.101 5000 &> $netcat_log &
And the streaming works very well. However, when I switch off the router, simulating a problem with the network, my $video_out is cut. I think this is due to the back-pressure from the netcat_fifo.
I found a solution here at stackexchange concerning a non-blocking FIFO, by replacing tee by ftee:
Linux non-blocking fifo (on demand logging)
It now prevents my $video_out from being affected by the streaming, but the streaming itself is very unstable. The best results were using the following script:
#RPi side
FPS="30"
MULTIPIPE="ftee"
mkfifo netcat_fifo
raspivid -t 0 -md 5 -fps $FPS -o - | ./${MULTIPIPE} netcat_fifo > $video_out &
cat netcat_fifo | mbuffer --direct -t -s 2k 2> $mbuffer_log | netcat -v 192.168.0.101 5000 &> $netcat_log &
When I check the mbuffer log, I diagnose a FIFO that remains most of the time empty, but has peaks of 99-100% utilization. During these peaks, my mplayer receiver-side has many errors decoding the video and takes about 5 seconds to recover itself. After this interval, the mbuffer log shows again an empty FIFO.
The empty->full->empty goes on and on.
I have two questions:
Am I using the correct approach to solve my problem?
If so, how could I render my streaming more robust while keeping the $video_out file intact?
I had a little attempt at this and it seems to work pretty solidly on my Raspberry Pi 3. It is pretty well commented so it should be quite easy to understand, but you can always ask if there are any questions.
Basically there are 3 threads:
main program - it constantly reads its stdin from raspivid and cyclically puts the data into a bunch of buffers
disk writer thread - it constantly cycles through the list of buffers waiting for the next one to become full. When the buffer is full, it writes the contents to disk, marks the buffer as written and moves onto the next one
fifo writer thread - it constantly cycles through the list of buffers waiting for the next one to become full. When the buffer is full, it writes the contents to the fifo, flushes the fifo to reduce lag and marks the buffer as written and moves onto the next one. Errors are ignored.
So, here is the code:
////////////////////////////////////////////////////////////////////////////////
// main.cpp
// Mark Setchell
//
// Read video stream from "raspivid" and write (independently) to both disk file
// and stdout - for onward netcatting to another host.
//
// Compiles with:
// g++ main.cpp -o main -lpthread
//
// Run on Raspberry Pi with:
// raspivid -t 0 -md 5 -fps 30 -o - | ./main video.h264 | netcat -v 192.168.0.8 5000
//
// Receive on other host with:
// netcat -l -p 5000 | mplayer -vf scale -zoom -xy 1280 -fps 30 -cache-min 50 -cache 1024 -
////////////////////////////////////////////////////////////////////////////////
#include <iostream>
#include <chrono>
#include <thread>
#include <vector>
#include <unistd.h>
#include <atomic>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define BUFSZ 65536
#define NBUFS 64
class Buffer{
public:
int bytes=0;
std::atomic<int> NeedsWriteToDisk{0};
std::atomic<int> NeedsWriteToFifo{0};
unsigned char data[BUFSZ];
};
std::vector<Buffer> buffers(NBUFS);
////////////////////////////////////////////////////////////////////////////////
// This is the DiskWriter thread.
// It loops through all the buffers waiting in turn for each one to become ready
// then writes it to disk and marks the buffer as written before moving to next
// buffer.
////////////////////////////////////////////////////////////////////////////////
void DiskWriter(char* filename){
int bufIndex=0;
// Open output file
int fd=open(filename,O_CREAT|O_WRONLY|O_TRUNC,S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP);
if(fd==-1)
{
std::cerr << "ERROR: Unable to open output file" << std::endl;
exit(EXIT_FAILURE);
}
bool Error=false;
while(!Error){
// Wait for buffer to be filled by main thread
while(buffers[bufIndex].NeedsWriteToDisk!=1){
// std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
// Write to disk
int bytesToWrite=buffers[bufIndex].bytes;
int bytesWritten=write(fd,reinterpret_cast<unsigned char*>(&buffers[bufIndex].data),bytesToWrite);
if(bytesWritten!=bytesToWrite){
std::cerr << "ERROR: Unable to write to disk" << std::endl;
exit(EXIT_FAILURE);
}
// Mark buffer as written
buffers[bufIndex].NeedsWriteToDisk=0;
// Move to next buffer
bufIndex=(bufIndex+1)%NBUFS;
}
}
////////////////////////////////////////////////////////////////////////////////
// This is the FifoWriter thread.
// It loops through all the buffers waiting in turn for each one to become ready
// then writes it to the Fifo, flushes it for reduced lag, and marks the buffer
// as written before moving to next one. Errors are ignored.
////////////////////////////////////////////////////////////////////////////////
void FifoWriter(){
int bufIndex=0;
bool Error=false;
while(!Error){
// Wait for buffer to be filled by main thread
while(buffers[bufIndex].NeedsWriteToFifo!=1){
// std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
// Write to fifo
int bytesToWrite=buffers[bufIndex].bytes;
int bytesWritten=write(STDOUT_FILENO,reinterpret_cast<unsigned char*>(&buffers[bufIndex].data),bytesToWrite);
if(bytesWritten!=bytesToWrite){
std::cerr << "ERROR: Unable to write to fifo" << std::endl;
}
// Try to reduce lag
fflush(stdout);
// Mark buffer as written
buffers[bufIndex].NeedsWriteToFifo=0;
// Move to next buffer
bufIndex=(bufIndex+1)%NBUFS;
}
}
int main(int argc, char *argv[])
{
int bufIndex=0;
if(argc!=2){
std::cerr << "ERROR: Usage " << argv[0] << " filename" << std::endl;
exit(EXIT_FAILURE);
}
char * filename = argv[1];
// Start disk and fifo writing threads in parallel
std::thread tDiskWriter(DiskWriter,filename);
std::thread tFifoWriter(FifoWriter);
bool Error=false;
// Continuously fill buffers from "raspivid" on stdin. Mark as full and
// needing output to disk and fifo before moving to next buffer.
while(!Error)
{
// Check disk writer is not behind before re-using buffer
if(buffers[bufIndex].NeedsWriteToDisk==1){
std::cerr << "ERROR: Disk writer is behind by " << NBUFS << " buffers" << std::endl;
}
// Check fifo writer is not behind before re-using buffer
if(buffers[bufIndex].NeedsWriteToFifo==1){
std::cerr << "ERROR: Fifo writer is behind by " << NBUFS << " buffers" << std::endl;
}
// Read from STDIN till buffer is pretty full
int bytes;
int totalBytes=0;
int bytesToRead=BUFSZ;
unsigned char* ptr=reinterpret_cast<unsigned char*>(&buffers[bufIndex].data);
while(totalBytes<(BUFSZ*.75)){
bytes = read(STDIN_FILENO,ptr,bytesToRead);
if(bytes<=0){
Error=true;
break;
}
ptr+=bytes;
totalBytes+=bytes;
bytesToRead-=bytes;
}
// Signal buffer ready for writing
buffers[bufIndex].bytes=totalBytes;
buffers[bufIndex].NeedsWriteToDisk=1;
buffers[bufIndex].NeedsWriteToFifo=1;
// Move to next buffer
bufIndex=(bufIndex+1)%NBUFS;
}
}
There is some Windows/VS setting that will let debug output in VS correctly output non Latin characters. We are interested in Cyrillic letters. What is that setting?
Too brief question. However, a solution based on this answer could help:
#include "stdafx.h"
#include <iostream>
#include <io.h>
#include <fcntl.h>
int wmain(int argc, wchar_t* argv[])
{
_setmode(_fileno(stdout), _O_U16TEXT);
std::wcout
<< L"Unicode test -- Ελληνικά -- čeština -- русский язык -- Türkçesi"
<< std::endl;
for (int i = 1; i < argc; ++i)
{
std::wcout << L"param " << i << L": " << argv[i] << std::endl;
}
}
Verified using Visual Studio 2013:
==> ".\so36212399\Debug\so36212399.exe" qwertz ςερτυθ ěščřžý йцукен ğüşıöç
Unicode test -- Ελληνικά -- čeština -- русский язык -- Türkçesi
param 1: qwertz
param 2: ςερτυθ
param 3: ěščřžý
param 4: йцукен
param 5: ğüşıöç
==>
I need to perform character set conversion using iconv on windows. In this case this is transliteration to remove accents, etc. but the issue I am facing is the same for most any target encoding. Here is my Program:
#include "stdafx.h"
#include <vector>
#include <fstream>
#include <iconv.h>
#include <iostream>
int _tmain(int argc, _TCHAR* argv[])
{
std::ifstream ifs("test.txt", std::ios::binary | std::ios::ate);
std::ifstream::pos_type pos = ifs.tellg();
char * pIn = new char[(int)pos + 1];
ifs.seekg(0, std::ios::beg);
ifs.read(pIn, pos);
pIn[pos] = 0;
size_t srclen = strlen(pIn);
char dst[1000];
char * pOut = (char*)dst;
size_t dstlen = 1000;
iconv_t conv = iconv_open("UTF-8", "ASCII//TRANSLIT");
std::cout << srclen << " " << dstlen << std::endl;
auto ret = iconv(conv, (const char**)&pIn, &srclen, &pOut, &dstlen);
std::cout << (int)ret << " " << errno << " " << srclen << " " << dstlen << std::endl;
iconv_close(conv);
return 0;
}
The test.txt file looks like this (UTF-8 w/o BOM):
qwe
Tøyenbekken
Zażółć gęślą jaźń
ZAŻÓŁĆ GĘŚLĄ JAŹŃ
Unfortunately the iconv call stops processing at the first non ASCII character and program outputs:
75 1000
-1 0 69 994
The return value of -1 indicates error, but errno is set to 0 which gives no clue as to what may be wrong.
Any idea what am I doing wrong here? To make the matter more interesting here is the output of iconv.exe located in the same dir as the libiconv2.dll file:
> iconv -f utf-8 -t ascii//translit test.txt
qwe
Toyenbekken
Zaz'ol'c ge'sla ja'z'n
ZAZ'OL'C GE'SLA JA'Z'N
which is ok.
Update after testing on Linux:
The command line version of iconv does not work - it outputs some garbage to the console (in place of non ascii characters).
Using my own code it outputs error code of 84 (which is I guess EILSEQ - Illegal byte sequence) after processing ascii characters.
Any ideas what may be wrong here?
The issue was that I wanted to convert from UTF-8 to ASCII and opened the converter this way:
iconv_t conv = iconv_open("UTF-8", "ASCII//TRANSLIT");
whereas it should be done this way:
iconv_t conv = iconv_open("ASCII//TRANSLIT", "UTF-8");
(argument order). Still not sure why I did not get proper error code.
I using MPI on Windows with Cygwin. I try to use critical section for write log some one, but what I would not do I always get a mixed log.
setbuf(stdout, 0);
int totalProcess;
MPI_Comm_size(MPI_COMM_WORLD, &totalProcess);
int processRank;
MPI_Comm_rank(MPI_COMM_WORLD, &processRank);
int rank = 0;
while (rank < totalProcess) {
if (processRank == rank) {
printf("-----%d-----\n", rank);
printf("%s", logBuffer);
printf("-----%d-----\n", rank);
//fflush(stdout);
}
rank ++;
MPI_Barrier(MPI_COMM_WORLD);
}
I run mpi at single machine (emulation mode):
mpirun -v -np 2 ./bin/main.out
I want dedicated space log per process, what I do wrong?
(When I wrote it I think it would not work correctly...)
This is the same problem asked about here; there is enough buffering going on at various different layers that there's no guarantee that the final output will reflect the order that the individual processes wrote, although in practice it can work for "small enough" outputs.
But if the goal is something like a logfile, MPI-IO provides mechanisms for you to write to a file in exactly such a way - MPI_File_write_ordered, which writes output in order of processors to the file. As an example:
#include <string.h>
#include <stdio.h>
#include "mpi.h"
int main(int argc, char** argv)
{
int rank, size;
MPI_File logfile;
char mylogbuffer[1024];
char line[128];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &size);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_File_open(MPI_COMM_WORLD, "logfile.txt", MPI_MODE_WRONLY | MPI_MODE_CREATE,
MPI_INFO_NULL, &logfile);
/* write initial message */
sprintf(mylogbuffer,"-----%d-----\n", rank);
sprintf(line,"Message from proc %d\n", rank);
for (int i=0; i<rank; i++)
strcat(mylogbuffer, line);
sprintf(line,"-----%d-----\n", rank);
strcat(mylogbuffer, line);
MPI_File_write_ordered(logfile, mylogbuffer, strlen(mylogbuffer), MPI_CHAR, MPI_STATUS_IGNORE);
/* write another message */
sprintf(mylogbuffer,"-----%d-----\nAll done\n-----%d-----\n", rank, rank);
MPI_File_write_ordered(logfile, mylogbuffer, strlen(mylogbuffer), MPI_CHAR, MPI_STATUS_IGNORE);
MPI_File_close(&logfile);
MPI_Finalize();
return 0;
}
Compiling and running gives:
$ mpicc -o log log.c -std=c99
$ mpirun -np 5 ./log
$ cat logfile.txt
-----0-----
-----0-----
-----1-----
Message from proc 1
-----1-----
-----2-----
Message from proc 2
Message from proc 2
-----2-----
-----3-----
Message from proc 3
Message from proc 3
Message from proc 3
-----3-----
-----4-----
Message from proc 4
Message from proc 4
Message from proc 4
Message from proc 4
-----4-----
-----0-----
All done
-----0-----
-----1-----
All done
-----1-----
-----2-----
All done
-----2-----
-----3-----
All done
-----3-----
-----4-----
All done
-----4-----