I'm trying to make some operations on Pointcloud but when calling VoxelDownSample function i'm getting Segmentation fault signal. I see that the problem is that program want to allocate some astronomic amount of data but i don't know where it come from as actuall PointCloud object size is 120.
Code:
open3d::geometry::PointCloud open3d_source_original;
open3d::geometry::PointCloud open3d_source_down;
createPointCloud(open3d_source_original);
open3d_source_down = *open3d_source_original.VoxelDownSample(0.01);
createPointCloud is my function which creates PointCloud from csv file and it works fine.
gdb output:
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff6eb4921 in __GI_abort () at abort.c:79
#2 0x00007ffff7ad6957 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff7adcae6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff7adcb21 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff7adcd54 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff7b05012 in std::__throw_bad_alloc() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x0000555555563476 in __gnu_cxx::new_allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> >::allocate(unsigned long, void const*) ()
#8 0x0000555555562bec in std::allocator_traits<std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::allocate(std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> >&, unsigned long) ()
#9 0x0000555555562088 in std::_Vector_base<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::_M_allocate(unsigned long) ()
#10 0x00005555555613ac in Eigen::Matrix<double, 3, 1, 0, 3, 1>* std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<Eigen::Matrix<double, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > > > >(unsigned long, __gnu_cxx::__normal_iterator<Eigen::Matrix<double, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > > >, __gnu_cxx::__normal_iterator<Eigen::Matrix<double, 3, 1, 0, 3, 1> const*, std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > > >) ()
#11 0x00005555555606e2 in std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::operator=(std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > > const&) ()
#12 0x000055555555fcfa in open3d::geometry::PointCloud::operator=(open3d::geometry::PointCloud const&) ()
#13 0x000055555555f400 in main ()
Valgrind output:
==14534== Invalid read of size 8
==14534== at 0x114D59: std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::begin() const (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
==14534== by 0x1147EA: std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::operator=(std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > > const&) (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
==14534== by 0x113CF9: open3d::geometry::PointCloud::operator=(open3d::geometry::PointCloud const&) (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
==14534== by 0x1133FF: main (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
==14534== Address 0x61da940 is 0 bytes after a block of size 112 alloc'd
==14534== at 0x4C3217F: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14534== by 0x12C358: allocate (new_allocator.h:111)
==14534== by 0x12C358: allocate (alloc_traits.h:436)
==14534== by 0x12C358: __allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<open3d::geometry::PointCloud, std::allocatoropen3d::geometry::PointCloud, (__gnu_cxx::_Lock_policy)2> > > (allocated_ptr.h:104)
==14534== by 0x12C358: __shared_count<open3d::geometry::PointCloud, std::allocatoropen3d::geometry::PointCloud > (shared_ptr_base.h:635)
==14534== by 0x12C358: __shared_ptr<std::allocatoropen3d::geometry::PointCloud > (shared_ptr_base.h:1295)
==14534== by 0x12C358: shared_ptr<std::allocatoropen3d::geometry::PointCloud > (shared_ptr.h:344)
==14534== by 0x12C358: allocate_shared<open3d::geometry::PointCloud, std::allocatoropen3d::geometry::PointCloud > (shared_ptr.h:691)
==14534== by 0x12C358: make_sharedopen3d::geometry::PointCloud (shared_ptr.h:707)
==14534== by 0x12C358: open3d::geometry::PointCloud::VoxelDownSample(double) const (PointCloud.cpp:306)
==14534== by 0x1133DB: main (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
==14534==
==14534==
==14534== HEAP SUMMARY:
==14534== in use at exit: 8 bytes in 1 blocks
==14534== total heap usage: 19 allocs, 18 frees, 107,576 bytes allocated
==14534==
==14534== LEAK SUMMARY:
==14534== definitely lost: 0 bytes in 0 blocks
==14534== indirectly lost: 0 bytes in 0 blocks
==14534== possibly lost: 0 bytes in 0 blocks
==14534== still reachable: 8 bytes in 1 blocks
==14534== suppressed: 0 bytes in 0 blocks
==14534== Rerun with --leak-check=full to see details of leaked memory
==14534==
==14534== For counts of detected and suppressed errors, rerun with: -v
==14534== Use --track-origins=yes to see where uninitialised values come from
==14534== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)
Operating system: Ubuntu 18.04
Open3D build from source
Compiler version: gcc 7.5
I tried to change macros from Eigen and change compilers but nothing worked.
This part shows a SIGABRT due to too much memory, not a SIGSEGV:
#0 __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff6eb4921 in __GI_abort () at abort.c:79
Valgrind is indicating that uninitialized data was used, not unmapped memory:
==14534== Invalid read of size 8
It appears to be pointing at the internal pointer to the start of the array which underlies the vector:
==14534== at 0x114D59: std::vector<Eigen::Matrix<double, 3, 1, 0, 3, 1>, std::allocator<Eigen::Matrix<double, 3, 1, 0, 3, 1> > >::begin() const (in /home/rufus/CLionProjects/isolatedSegFault/build/isolatedSegFault)
Then it says that the vector was created by make_shared inside voxelDownSample:
==14534== by 0x12C358: make_sharedopen3d::geometry::PointCloud (shared_ptr.h:707)
==14534== by 0x12C358: open3d::geometry::PointCloud::VoxelDownSample(double) const (PointCloud.cpp:306)
My intuition says,
If a weak_ptr is involved, then perhaps the vector was destroyed by losing all its shared_ptr references. Valgrind can't see this because it wouldn't be deallocated until the weak_ptr references are gone, too.
Otherwise, there's some other kind of memory error, such as
trying to use the vector before it has actually been constructed
using a raw pointer after the shared_ptr is destroyed
a buffer overflow, or other randomly spraying memory (rather less likely to hit a shared_ptr object, but you never know)
In any case, it looks like a library bug, if this object was created after the last time any of your code ran, and your parameters were numerically valid.
Related
I have an LSTM that receives as input a padded tensor with pad values on both sides (assume that pad values are 0):
[0, 0, 0, 3, 2, 1, 4, 0, 0, 0]
[1, 9, 5, 3, 2, 1, 4, 7, 2, 3]
[0, 0, 0, 3, 2, 1, 4, 2, 3, 3]
Typically, one would use TORCH.NN.UTILS.RNN.PACK_PADDED_SEQUENCE to let the LSTM know which values to pad, but I've only seen it used for cases where tensors are right-padded. There's also no indication from the documentation that it will be able to recognize padded values on the left-hand side. Is there an alternative?
How initialize in pytorch hybrid tensor torch.sparse_coo_tensor (one dimension is sparse and other is not), which have the following dense representation?
array([[1, 0, 5, 0],
[2, 0, 6, 0],
[3, 0, 7, 0],
[4, 0, 8, 0]])
What should I put into the indices argument?
How to initialize
Something like this:
import torch
indices = torch.tensor([[0, 0, 1, 1, 2, 2, 3, 3], [0, 2, 0, 2, 0, 2, 0, 2]])
tensor = torch.sparse_coo_tensor(
indices, torch.tensor([1, 2, 3, 4, 5, 6, 7, 8]), size=(4, 4)
)
Given above:
indices - first dimension specifies row, second column, where non-zero value(s) will be located. Those become pairs, in this case: (0, 0), (0, 2), (1, 0), (1, 2)... and so on
values - values located at those pairs, so 1 will be under (0, 0) coordinate, 2 under (0, 2) and so it goes.
size - total size of the matrix, optional, might be inferred in this case from your input
8 pairs, 8 values, there are also other ways to specify it, but the idea holds.
And a quick check:
print(tensor)
print(tensor.to_dense())
Gives us:
tensor(indices=tensor([[0, 0, 1, 1, 2, 2, 3, 3],
[0, 2, 0, 2, 0, 2, 0, 2]]),
values=tensor([1, 2, 3, 4, 5, 6, 7, 8]),
size=(4, 4), nnz=8, layout=torch.sparse_coo)
tensor([[1, 0, 2, 0],
[3, 0, 4, 0],
[5, 0, 6, 0],
[7, 0, 8, 0]])
Why to initialize
If your actual data is 50% sparse, you shouldn't use COO tensor.
It will save some memory, but operations will be way slower, so keep that in mind.
I'm trying to somehow test my rbd storage with random read, random write, mixed randrw, but the output is not correct, it is a sequential growing number.
What is wrong with my steps?
This is the fio file that I ran:
; fio-rand-write.job for fiotest
[global]
name=fio-rand-write
filename=fio-rand-write
rw=randwrite
bs=4K
direct=1
write_iops_log=rand-read
[file1]
size=1G
ioengine=libaio
iodepth=16
And the result is this:
head rand-read_iops.1.log
2, 1, 1, 4096, 0
2, 1, 1, 4096, 0
2, 1, 1, 4096, 0
2, 1, 1, 4096, 0
2, 1, 1, 4096, 0
3, 1, 1, 4096, 0
4, 1, 1, 4096, 0
5, 1, 1, 4096, 0
5, 1, 1, 4096, 0
5, 1, 1, 4096, 0
tail rand-read_iops.1.log
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
30700, 1, 1, 4096, 0
I'm using fio 3.18.
Why I don't get the iops that is the real one?
(Note this isn't really a programming question so Stackoverflow is the wrong place to ask this... Maybe Super User or Serverfault would be a better choice and get faster answers?)
but the output is not correct, it is a sequential growing number
Which column are you referring to? If you mean the left most column then isn't that time per the fio Log File Formats documentation?
Fio supports a variety of log file formats, for logging latencies, bandwidth, and IOPS. The logs share a common format, which looks like this:
time (msec), value, data direction, block size (bytes), offset (bytes)
Doesn't time generally monotonically increase relative to prior readings (accounting for precision)?
Also see the documentation for write_iops_log that says:
Because fio defaults to individual I/O logging, the value entry in the IOPS log will be 1 unless windowed logging (see log_avg_msec) has been enabled
How do you serialized/deserialize a numpy array?
A = np.random.randint(0, 10, 40).reshape(8, 5)
print(A)
print (A.dtype)
snapshot = A
serialized = snapshot.tobytes()
[[9 5 5 7 4]
[3 8 8 1 0]
[5 7 1 0 2]
[2 2 7 1 2]
[2 6 3 5 4]
[7 5 4 8 3]
[2 4 2 4 7]
[3 4 2 6 2]]
int64
Returns
deserialized = np.frombuffer(serialized).astype(np.int64)
print (deserialized)
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0]
There is a mismatch between the default dtype used to generate A and in np.frombuffer. Works as expected when using the correct dtype (may depend on the machine / Python / numpy version):
# Python 3.6 64-bits with numpy 1.12.1 64-bits
A = np.random.randint(0, 10, 40).reshape(8, 5)
print(A)
>>> array([[3, 3, 5, 3, 9],
[1, 4, 7, 1, 8],
[1, 7, 4, 3, 0],
[9, 2, 9, 1, 2],
[2, 8, 9, 1, 1],
[3, 3, 5, 2, 6],
[5, 0, 2, 7, 6],
[2, 8, 8, 0, 7]])
A.dtype
>>> dtype('int32')
deserialized = np.frombuffer(A.tobytes(), dtype=np.int32).reshape(A.shape)
print(deserialized)
>>> array([[3, 3, 5, 3, 9],
[1, 4, 7, 1, 8],
[1, 7, 4, 3, 0],
[9, 2, 9, 1, 2],
[2, 8, 9, 1, 1],
[3, 3, 5, 2, 6],
[5, 0, 2, 7, 6],
[2, 8, 8, 0, 7]])
I have spent the last day looking for a memory leak in my code. For each frame in my loop I can see in the task manager that about 200 Mb of memory is lost. Eventually the program of course crashes.
The pseudo-code looks like this:
for frame = 0:NBROFFRAMES
cv::Mat = getImage(frame);
cout<<"Before optimization"<<endl;
getchar();
optFunction.optimizeColor(img);
cout<<"After optimization"<<endl;
getchar();
Do other things
end
Just looking in the task manager shows that in optimizeColor approximately 200 Mb are lost for every frame.
In optimizeColorI run Eigens ConjugateGradient several times. When I changed to Cholesky Factorization, SimplicialLDLT, the memory consumption is between 4.5-4.7 Gb over the entire sequence. Using ConjugateGradient it starts with 4.5 Gb and after 10 images the amount of memory used according to my Task Manager is 11.2 Gb.
I have also run Valgrind and one of the messages I get is the following:
==22576== 110,430,720 bytes in 18 blocks are possibly lost in loss record 30,186 of 30,189
==22576== at 0x4C2E80F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22576== by 0x913CCD: Eigen::internal::scoped_array<int>::scoped_array(long) (Memory.h:666)
==22576== by 0x910329: Eigen::internal::CompressedStorage<float, int>::reallocate(long) (CompressedStorage.h:230)
==22576== by 0x910695: Eigen::internal::CompressedStorage<float, int>::resize(long, double) (CompressedStorage.h:96)
==22576== by 0x91CC41: Eigen::SparseMatrix<float, 0, int>& Eigen::SparseMatrix<float, 0, int>::operator=<Eigen::SparseMatrix<float, 1, int> >(Eigen::SparseMatrixBase<Eigen::SparseMatrix<float, 1, int> > const&) (SparseMatrix.h:1085)
==22576== by 0x91A911: Eigen::SparseMatrix<float, 0, int>::SparseMatrix<Eigen::SparseMatrix<float, 1, int> >(Eigen::SparseMatrixBase<Eigen::SparseMatrix<float, 1, int> > const&) (SparseMatrix.h:674)
==22576== by 0x91837C: void Eigen::Ref<Eigen::SparseMatrix<float, 0, int> const, 0, Eigen::OuterStride<-1> >::construct<Eigen::SparseMatrix<float, 1, int> >(Eigen::SparseMatrix<float, 1, int> const&, Eigen::internal::false_type) (SparseRef.h:217)
==22576== by 0x91631C: Eigen::Ref<Eigen::SparseMatrix<float, 0, int> const, 0, Eigen::OuterStride<-1> >::Ref<Eigen::SparseMatrix<float, 1, int> >(Eigen::SparseMatrixBase<Eigen::SparseMatrix<float, 1, int> > const&) (SparseRef.h:184)
==22576== by 0x913801: void Eigen::internal::generic_matrix_wrapper<Eigen::SparseMatrix<float, 0, int>, false>::grab<Eigen::SparseMatrix<float, 1, int> >(Eigen::EigenBase<Eigen::SparseMatrix<float, 1, int> > const&) (IterativeSolverBase.h:81)
==22576== by 0x90FEFE: void Eigen::IterativeSolverBase<Eigen::ConjugateGradient<Eigen::SparseMatrix<float, 0, int>, 3, Eigen::DiagonalPreconditioner<float> > >::grab<Eigen::SparseMatrix<float, 1, int> >(Eigen::SparseMatrix<float, 1, int> const&) (IterativeSolverBase.h:375)
==22576== by 0x90C03D: Eigen::ConjugateGradient<Eigen::SparseMatrix<float, 0, int>, 3, Eigen::DiagonalPreconditioner<float> >& Eigen::IterativeSolverBase<Eigen::ConjugateGradient<Eigen::SparseMatrix<float, 0, int>, 3, Eigen::DiagonalPreconditioner<float> > >::analyzePattern<Eigen::SparseMatrix<float, 1, int> >(Eigen::EigenBase<Eigen::SparseMatrix<float, 1, int> > const&) (IterativeSolverBase.h:199)
==22576== by 0x9077F3: OptimizeAlbedo::optimize_rho(cv::Mat const&, std::vector<Eigen::Matrix<float, -1, 1, 0, -1, 1>, std::allocator<Eigen::Matrix<float, -1, 1, 0, -1, 1> > > const&, std::vector<Eigen::Matrix<int, 2, 1, 0, 2, 1>, std::allocator<Eigen::Matrix<int, 2, 1, 0, 2, 1> > > const&, Eigen::Matrix<float, -1, 1, 0, -1, 1> const&, float, float, Eigen::Matrix<float, -1, 1, 0, -1, 1> const&) (OptimizeAlbedo.cpp:586)
The only thing I do that allocates memory is that I define the sizes of some matrices when I create the instance optFunction and reserve space for the entries. That is of course only done once and that I have checked several times. The only explanation I see is that ConjugateGradient is consuming memory.
Here is my code to call ConjugateGradient:
function optmize()
Eigen::ConjugateGradient<Eigen::SparseMatrix<float>, Eigen::Lower | Eigen::Upper> cg;
cg.setTolerance(0.001);
cg.setMaxIterations(200);
cg.analyzePattern(A_tot);
cg.factorize(A_tot);
Eigen::VectorXf opt = cg.solveWithGuess(b_tot, rho_current);
end
I would have expected that cg is cleared when the function goes out of scope, but it seems that is not the case.
I think I use Eigen version: 3.2.92, this is in the Macros.h:
#define EIGEN_WORLD_VERSION 3
#define EIGEN_MAJOR_VERSION 2
#define EIGEN_MINOR_VERSION 92
I am using Ubuntu 16.04 and g++ 5.4.0.
EDIT: As pointed out by Avi Ginsburg, I was using a beta-version of Eigen. By upgrading to 3.3.2 it works without memory leaks.