I'm trying to re-implement this C code:
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#define ROWS 100000
#define COLS 10000
int twodarray[ROWS][COLS];
int main() {
for (int i = 0; i < ROWS; i++) {
for (int j = 0; j < COLS; j++) {
twodarray[i][j] = rand();
}
}
int64 sum = 0;
for (int i = 0; i < ROWS; i++) {
for (int j = 0; j < COLS; j++) {
sum += twodarray[i][j];
}
}
}
So, after a lot of trial and error I've come up with Rust code that at least compiles
extern crate rand;
const ROWS: usize = 100000;
const COLS: usize = 10000;
fn main() {
let mut twodarray: [[u128; COLS]; ROWS] = [[0; COLS]; ROWS];
for i in 0..ROWS {
for j in 0..COLS {
twodarray[i][j] = rand::random::<u8>() as u128;
}
}
let mut sum: u128 = 0;
for k in 0..ROWS {
for j in 0..COLS {
sum += twodarray[k][j] as u128;
}
}
println!("{}", sum);
}
But when I compile and execute it, I get the following error message: thread 'main' has overflowed its stack. To be honest, I have absolutelly no idea why this is happening. I'm currently learning rust, but there is not a lot of information on 2d arrays online... I DO NOT WANT TO USE VECTOR. This exercise is specifically aimed on arrays.
EDIT:
After reading the accepted answear, I've come up with Rust code that outputs expected results:
extern crate rand;
const ROWS: usize = 100000;
const COLS: usize = 10000;
fn main() {
// let mut twodarray: Box<[[u8; COLS]; ROWS]> = Box::new([[0; COLS]; ROWS]);
// for i in 0..ROWS {
// for j in 0..COLS {
// twodarray[i][j] = rand::random::<u8>();
// }
// }
// let mut sum: u32 = 0;
// for k in 0..ROWS {
// for l in 0..COLS {
// sum += twodarray[k][l] as u32;
// }
// }
let mut twodarray: Box<[[u8; ROWS]; COLS]> = Box::new([[0; ROWS]; COLS]);
for i in 0..COLS {
for j in 0..ROWS {
twodarray[i][j] = rand::random::<u8>();
}
}
let mut sum: u32 = 0;
for k in 0..COLS {
for l in 0..ROWS {
sum += twodarray[k][l] as u32;
}
}
println!("{}", sum);
}
This two-dimensional array is huge. Rust tries to allocate it on the stack (that's the way arrays in Rust work).
You explicitly wrote you didn't want to use vectors, but in reality vectors are allocated on the heap, so you wouldn't have such a problem with them. It's good to keep that in mind.
If you insist on using arrays, maybe put them in a Box, so that they go to the heap?
That's not a Rust-specific problem. Stacks are generally a lot more limited, compared to the heap, usually up to a few megabytes. In some other languages, such as Java, arrays are allocated on the heap, but that doesn't mean there is no similar stack size limit there as well.
Related
I am trying to monitor a binary file containing single 32 bit integer. I mapped the file into memory via MmapMut and read in a forever loop:
fn read(mmap: &MmapMut) {
let mut i = u32::from_ne_bytes(mmap[0..4].try_into().unwrap());
loop {
let j = u32::from_ne_bytes(mmap[0..4].try_into().unwrap());
if j != i {
assert!(false); // this assert should fail, but it never does
i = j;
}
}
}
However, compiler seems to optimise the loop away assuming neither i or j can ever change.
Is there a way to prevent this optimisation?
You can use MmapRaw with read_volatile().
fn read(mmap: &MmapRaw) {
let mut i = unsafe { mmap.as_ptr().cast::<u32>().read_volatile() };
loop {
let j = unsafe { mmap.as_ptr().cast::<u32>().read_volatile() };
if j != i {
assert!(false);
i = j;
}
}
}
You can also consider this as, is it possible to URLify a string in place in rust?
For example,
Problem statement: Replace whitespace with %20
Assumption: String will have enough capacity left to accommodate new characters.
Input: Hello how are you
Output: Hello%20how%20are%20you
I know there are ways to do this if we don't have to do this "in place". I am solving a problem that explicitly states that you have to update in place.
If there isn't any safe way to do this, is there any particular reason behind that?
[Edit]
I was able to solve this using unsafe approach, but would appreciate a better approach than this. More idiomatic approach if there is.
fn space_20(sentence: &mut String) {
if !sentence.is_ascii() {
panic!("Invalid string");
}
let chars: Vec<usize> = sentence.char_indices().filter(|(_, ch)| ch.is_whitespace()).map(|(idx, _)| idx ).collect();
let char_count = chars.len();
if char_count == 0 {
return;
}
let sentence_len = sentence.len();
sentence.push_str(&"*".repeat(char_count*2)); // filling string with * so that bytes array becomes of required size.
unsafe {
let bytes = sentence.as_bytes_mut();
let mut final_idx = sentence_len + (char_count * 2) - 1;
let mut i = sentence_len - 1;
let mut char_ptr = char_count - 1;
loop {
if i != chars[char_ptr] {
bytes[final_idx] = bytes[i];
if final_idx == 0 {
// all elements are filled.
println!("all elements are filled.");
break;
}
final_idx -= 1;
} else {
bytes[final_idx] = '0' as u8;
bytes[final_idx - 1] = '2' as u8;
bytes[final_idx - 2] = '%' as u8;
// final_idx is of type usize cannot be less than 0.
if final_idx < 3 {
println!("all elements are filled at start.");
break;
}
final_idx -= 3;
// char_ptr is of type usize cannot be less than 0.
if char_ptr > 0 {
char_ptr -= 1;
}
}
if i == 0 {
// all elements are parsed.
println!("all elements are parsed.");
break;
}
i -= 1;
}
}
}
fn main() {
let mut sentence = String::with_capacity(1000);
sentence.push_str(" hello, how are you?");
// sentence.push_str("hello, how are you?");
// sentence.push_str(" hello, how are you? ");
// sentence.push_str(" ");
// sentence.push_str("abcd");
space_20(&mut sentence);
println!("{}", sentence);
}
An O(n) solution that neither uses unsafe nor allocates (provided that the string has enough capacity), using std::mem::take:
fn urlify_spaces(text: &mut String) {
const SPACE_REPLACEMENT: &[u8] = b"%20";
// operating on bytes for simplicity
let mut buffer = std::mem::take(text).into_bytes();
let old_len = buffer.len();
let space_count = buffer.iter().filter(|&&byte| byte == b' ').count();
let new_len = buffer.len() + (SPACE_REPLACEMENT.len() - 1) * space_count;
buffer.resize(new_len, b'\0');
let mut write_pos = new_len;
for read_pos in (0..old_len).rev() {
let byte = buffer[read_pos];
if byte == b' ' {
write_pos -= SPACE_REPLACEMENT.len();
buffer[write_pos..write_pos + SPACE_REPLACEMENT.len()]
.copy_from_slice(SPACE_REPLACEMENT);
} else {
write_pos -= 1;
buffer[write_pos] = byte;
}
}
*text = String::from_utf8(buffer).expect("invalid UTF-8 during URL-ification");
}
(playground)
Basically, it calculates the final length of the string, sets up a reading pointer and a writing pointer, and translates the string from right to left. Since "%20" has more characters than " ", the writing pointer never catches up with the reading pointer.
Is it possible to do this without unsafe?
Yes like this:
fn main() {
let mut my_string = String::from("Hello how are you");
let mut insert_positions = Vec::new();
let mut char_counter = 0;
for c in my_string.chars() {
if c == ' ' {
insert_positions.push(char_counter);
char_counter += 2; // Because we will insert two extra chars here later.
}
char_counter += 1;
}
for p in insert_positions.iter() {
my_string.remove(*p);
my_string.insert(*p, '0');
my_string.insert(*p, '2');
my_string.insert(*p, '%');
}
println!("{}", my_string);
}
Here is the Playground.
But should you do it?
As discussed for example here on Reddit this is almost always not the recommended way of doing this, because both remove and insert are O(n) operations as noted in the documentation.
Edit
A slightly better version:
fn main() {
let mut my_string = String::from("Hello how are you");
let mut insert_positions = Vec::new();
let mut char_counter = 0;
for c in my_string.chars() {
if c == ' ' {
insert_positions.push(char_counter);
char_counter += 2; // Because we will insert two extra chars here later.
}
char_counter += 1;
}
for p in insert_positions.iter() {
my_string.remove(*p);
my_string.insert_str(*p, "%20");
}
println!("{}", my_string);
}
and the corresponding Playground.
I'm writing a Node.js extension in Rust. I called this toy library from C, checked it with Valgrind and no memory leak was found.
I called the same library from Python and Ruby, running in an infinite loop, and see no sign of a memory leak. (I can not post the picture due to insufficient reputation here).
When the same library is called from Node.js, the memory usage seems to increase with time:
The unit of time is the cycle of loop, not a real time.
Here is the Rust code:
#[repr(C)]
pub struct Matrix {
m: Vec<Vec<f64>>,
}
#[no_mangle]
pub extern "C" fn matrix_new(nrow: usize, ncol: usize) -> *mut Matrix {
let mut m = Vec::new();
for _ in 0..(nrow) {
let mut n = Vec::new();
for _ in 0..(ncol) {
n.push(0.0);
}
m.push(n);
}
Box::into_raw(Box::new(Matrix { m: m }))
}
#[no_mangle]
pub extern "C" fn matrix_free(matrix: *mut Matrix) {
if matrix.is_null() {
return
}
unsafe { Box::from_raw(matrix); }
}
Here is the original Node.js code:
var ref = require('ref');
var ffi = require('ffi');
var c_matrix = ref.types.void;
var c_matrix_ptr = ref.refType(c_matrix);
var libmatrix = ffi.Library('./target/release/libmatrix.so', {
'matrix_new': [ c_matrix_ptr, ['size_t', 'size_t']],
'matrix_get': [ 'double', [ c_matrix_ptr, 'size_t', 'size_t']],
'matrix_set': [ 'double', [ c_matrix_ptr, 'size_t', 'size_t', 'double']],
'matrix_free': [ 'void', [ c_matrix_ptr ]]
});
var matrix = function(nrow, ncol) {
"use strict";
var matrix = libmatrix.matrix_new(nrow, ncol);
Object.defineProperty(this, '_matrix', {
value: matrix,
writeable: false
});
return this;
};
matrix.prototype.get = function(row, col) {
"use strict";
return libmatrix.matrix_get(this._matrix, row, col);
};
matrix.prototype.set = function(row, col, value) {
"use strict";
libmatrix.matrix_set(this._matrix, row, col, value);
};
matrix.prototype.free = function() {
"use strict";
libmatrix.matrix_free(this._matrix);
};
module.exports = matrix;
if (!module.parent) {
while (true) {
var m = new matrix(3, 3);
m.free();
m = null;
delete global.m;
global.gc();
console.log(process.memoryUsage().rss);
}
}
Some information:
OS: Debian GNU/Linux (Jessie)
Node.js: 7.2.0
node-gyp: 7.2.0
ffi: 2.2.0
ref: 1.3.3
Rust: 1.13.0
I re-wrote the same library in C:
typedef struct Matrix {
size_t nrow;
size_t ncol;
double** mtx;
} Matrix;
void* matrix_new(size_t row, size_t col) {
Matrix* m = malloc(sizeof(Matrix));
m->nrow = row;
m->ncol = col;
m->mtx = malloc(m->nrow * sizeof(double*));
for (int i = 0; i < m->nrow; i++) {
*((m->mtx)+i) = (double*) malloc(m->ncol * sizeof(double));
}
for (int i = 0; i < m->nrow; i++) {
for (int j = 0; j < m->ncol; j++) {
m->mtx[i][j] = 0.0;
}
}
return (void*) m;
}
void matrix_free(void* m) {
if (m == NULL) {
return;
}
double** ptr = ((Matrix*) m)->mtx;
for (int i = 0; i < ((Matrix*) m)->nrow; i++) {
free((void*) (ptr[i]));
}
free((void*) ((Matrix*) m)->mtx);
free((void*) m);
}
I also changed the version of Node from 7.2.0 to 7.3.0. Here is the memory usage of the Node.js module with both C and Rust implementation:
I tried a no-op library without changing the Node.js code, finding something surprising myself:
I am new to Rust, and struggling to deal with all those wrapper types in Rust. I am trying to write code that is semantically equal to the following C code. The code tries to create a big table for book keeping, but will divide the big table so that every thread will only access their local small slices of that table. The big table will not be accessed unless other threads quit and no longer access their own slice.
#include <stdio.h>
#include <pthread.h>
void* write_slice(void* arg) {
int* slice = (int*) arg;
int i;
for (i = 0; i < 10; i++)
slice[i] = i;
return NULL;
}
int main()
{
int* table = (int*) malloc(100 * sizeof(int));
int* slice[10];
int i;
for (i = 0; i < 10; i++) {
slice[i] = table + i * 10;
}
// create pthread for each slice
pthread_t p[10];
for (i = 0; i < 10; i++)
pthread_create(&p[i], NULL, write_slice, slice[i]);
for (i = 0; i < 10; i++)
pthread_join(p[i], NULL);
for (i = 0; i < 100; i++)
printf("%d,", table[i]);
}
How do I use Rust's types and ownership to achieve this?
Let's start with the code:
// cargo-deps: crossbeam="0.7.3"
extern crate crossbeam;
const CHUNKS: usize = 10;
const CHUNK_SIZE: usize = 10;
fn main() {
let mut table = [0; CHUNKS * CHUNK_SIZE];
// Scoped threads allow the compiler to prove that no threads will outlive
// table (which would be bad).
let _ = crossbeam::scope(|scope| {
// Chop `table` into disjoint sub-slices.
for slice in table.chunks_mut(CHUNK_SIZE) {
// Spawn a thread operating on that subslice.
scope.spawn(move |_| write_slice(slice));
}
// `crossbeam::scope` ensures that *all* spawned threads join before
// returning control back from this closure.
});
// At this point, all threads have joined, and we have exclusive access to
// `table` again. Huzzah for 100% safe multi-threaded stack mutation!
println!("{:?}", &table[..]);
}
fn write_slice(slice: &mut [i32]) {
for (i, e) in slice.iter_mut().enumerate() {
*e = i as i32;
}
}
One thing to note is that this needs the crossbeam crate. Rust used to have a similar "scoped" construct, but a soundness hole was found right before 1.0, so it was deprecated with no time to replace it. crossbeam is basically the replacement.
What Rust lets you do here is express the idea that, whatever the code does, none of the threads created within the call to crossbeam::scoped will survive that scope. As such, anything borrowed from outside that scope will live longer than the threads. Thus, the threads can freely access those borrows without having to worry about things like, say, a thread outliving the stack frame that table is defined by and scribbling over the stack.
So this should do more or less the same thing as the C code, though without that nagging worry that you might have missed something. :)
Finally, here's the same thing using scoped_threadpool instead. The only real practical difference is that this allows us to control how many threads are used.
// cargo-deps: scoped_threadpool="0.1.6"
extern crate scoped_threadpool;
const CHUNKS: usize = 10;
const CHUNK_SIZE: usize = 10;
fn main() {
let mut table = [0; CHUNKS * CHUNK_SIZE];
let mut pool = scoped_threadpool::Pool::new(CHUNKS as u32);
pool.scoped(|scope| {
for slice in table.chunks_mut(CHUNK_SIZE) {
scope.execute(move || write_slice(slice));
}
});
println!("{:?}", &table[..]);
}
fn write_slice(slice: &mut [i32]) {
for (i, e) in slice.iter_mut().enumerate() {
*e = i as i32;
}
}
I'm implementing combsort. I'd like to create fixed-size array on the stack, but it shows stack overflow. When I change it to be on the heap (Rust by Example says to allocate in the heap we must use Box), it still shows stack overflow.
fn new_gap(gap: usize) -> usize {
let ngap = ((gap as f64) / 1.3) as usize;
if ngap == 9 || ngap == 10 {
return 11;
}
if ngap < 1 {
return 1;
}
return ngap;
}
fn comb_sort(a: &mut Box<[f64]>) {
// previously: [f64]
let xlen = a.len();
let mut gap = xlen;
let mut swapped: bool;
let mut temp: f64;
loop {
swapped = false;
gap = new_gap(gap);
for i in 0..(xlen - gap) {
if a[i] > a[i + gap] {
swapped = true;
temp = a[i];
a[i] = a[i + gap];
a[i + gap] = temp;
}
}
if !(gap > 1 || swapped) {
break;
}
}
}
const N: usize = 10000000;
fn main() {
let mut arr: Box<[f64]> = Box::new([0.0; N]); // previously: [f64; N] = [0.0; N];
for z in 0..(N) {
arr[z] = (N - z) as f64;
}
comb_sort(&mut arr);
for z in 1..(N) {
if arr[z] < arr[z - 1] {
print!("!")
}
}
}
The output:
thread '<main>' has overflowed its stack
Illegal instruction (core dumped)
Or
thread 'main' has overflowed its stack
fatal runtime error: stack overflow
I know that my stack size is not enough, the same as C++ when creating a non-heap array that is too big inside a function, but this code is using heap but still shows stack overflow. What's really wrong with this code?
As far as I can tell, it seems like that code is still trying to allocate the array on the stack first, and then move it into the box after.
It works for me if I switch to Vec<f64> in place of Box<[f64]> like this:
fn new_gap(gap: usize) -> usize {
let ngap = ((gap as f64) / 1.3) as usize;
if ngap == 9 || ngap == 10 {
return 11;
}
if ngap < 1 {
return 1;
}
return ngap;
}
fn comb_sort(a: &mut [f64]) {
// previously: [f64]
let xlen = a.len();
let mut gap = xlen;
let mut swapped: bool;
let mut temp: f64;
loop {
swapped = false;
gap = new_gap(gap);
for i in 0..(xlen - gap) {
if a[i] > a[i + gap] {
swapped = true;
temp = a[i];
a[i] = a[i + gap];
a[i + gap] = temp;
}
}
if !(gap > 1 || swapped) {
break;
}
}
}
const N: usize = 10000000;
fn main() {
let mut arr: Vec<f64> = std::iter::repeat(0.0).take(N).collect();
//let mut arr: Box<[f64]> = Box::new([0.0; N]); // previously: [f64; N] = [0.0; N];
for z in 0..(N) {
arr[z] = (N - z) as f64;
}
comb_sort(arr.as_mut_slice());
for z in 1..(N) {
if arr[z] < arr[z - 1] {
print!("!")
}
}
}
In the future, the box syntax will be stabilized. When it is, it will support this large allocation, as no function call to Box::new will be needed, thus the array will never be placed on the stack. For example:
#![feature(box_syntax)]
fn main() {
let v = box [0i32; 5_000_000];
println!("{}", v[1_000_000])
}