Strange behavior from V8's usage of v8::EmbedderHeapTracer - garbage-collection
I'm having a bit of a hard time with using the v8::EmbedderHeapTracer API with V8 version 6.3.
My issue is the following sequence of events happening in the following order:
V8 will call TracePrologue.
V8 will register a V8 reference for embedder field pointer with RegisterV8References.
Optional: I have tried both calling and not calling RegisterExternalReference on the object the embedder field is stored in here. Intuitively, I would expect this to be redundant information, as V8 has already detected it as reachable, but I have tried calling it out of paranoia.
I will add pointer to my frontier (e.g. the stack for DFS traversal of the embedder heap).
V8 will then run the SetWeak callback on the v8::Persistent<v8::Object> that it just registered before. This will cause me to delete the object.
Note that neither EnterFinalPause nor TraceEpilogue have been called yet.
pointer is removed from the frontier, and I attempt to expand it. This results in a heap use after free (detected by ASan).
Is objects that are determined to be reachable being deleted within the same GC cycle something that I should have to be worried about? I would have thought that anything that has RegisterExternalReference called on it would be kept alive until at least the next GC cycle.
Additional information:
ASan dump:
=================================================================
==184290==ERROR: AddressSanitizer: heap-use-after-free on address 0x60c0001687c0 at pc 0x0000053ad97e bp 0x7f36d0676f30 sp 0x7f36d0676f28
READ of size 8 at 0x60c0001687c0 thread T30 (MainWebModule)
#0 0x53ad97d in cobalt::script::v8c::V8cHeapTracer::AdvanceTracing(double, v8::EmbedderHeapTracer::AdvanceTracingActions) out/linux-x64x11-v8_debug/../../cobalt/script/v8c/v8c_heap_tracer.cc:76:16
#1 0x675d8e2 in v8::internal::LocalEmbedderHeapTracer::Trace(double, v8::EmbedderHeapTracer::AdvanceTracingActions) out/linux-x64x11-v8_debug/../../v8/src/heap/embedder-tracing.cc:45:26
#2 0x67975e7 in v8::internal::MarkCompactCollector::ProcessEphemeralMarking(bool) out/linux-x64x11-v8_debug/../../v8/src/heap/mark-compact.cc:1847:46
#3 0x678ed88 in v8::internal::MarkCompactCollector::MarkLiveObjects() out/linux-x64x11-v8_debug/../../v8/src/heap/mark-compact.cc:2558:7
#4 0x678d983 in v8::internal::MarkCompactCollector::CollectGarbage() out/linux-x64x11-v8_debug/../../v8/src/heap/mark-compact.cc:525:3
#5 0x66ab63d in v8::internal::Heap::MarkCompact() out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1684:29
#6 0x66a835f in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1557:9
#7 0x66a6c48 in v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1216:11
#8 0x66a4b78 in v8::internal::Heap::CollectAllGarbage(int, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1069:3
#9 0x66a4725 in v8::internal::Heap::HandleGCRequest() out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:993:5
#10 0x659cb9a in v8::internal::StackGuard::HandleInterrupts() out/linux-x64x11-v8_debug/../../v8/src/execution.cc:480:23
#11 0x710ac22 in v8::internal::__RT_impl_Runtime_StackGuard(v8::internal::Arguments, v8::internal::Isolate*) out/linux-x64x11-v8_debug/../../v8/src/runtime/runtime-internal.cc:306:34
#12 0x710a4f3 in v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) out/linux-x64x11-v8_debug/../../v8/src/runtime/runtime-internal.cc:296:1
#13 0x7f36af5047a3 (<unknown module>)
0x60c0001687c0 is located 0 bytes inside of 120-byte region [0x60c0001687c0,0x60c000168838)
freed by thread T30 (MainWebModule) here:
#0 0x2308fd2 in operator delete(void*) (out/linux-x64x11-v8_debug/cobalt+0x2308fd2)
#1 0x3b532d1 in cobalt::dom::KeyboardEvent::~KeyboardEvent() out/linux-x64x11-v8_debug/../../cobalt/dom/keyboard_event.h:99:29
#2 0x238f94c in base::RefCounted<cobalt::script::Wrappable>::Release() const out/linux-x64x11-v8_debug/../../base/memory/ref_counted.h:91:7
#3 0x24ced4f in scoped_refptr<cobalt::script::Wrappable>::~scoped_refptr() out/linux-x64x11-v8_debug/../../base/memory/ref_counted.h:249:13
#4 0xb928ca2 in cobalt::script::v8c::WrapperPrivate::~WrapperPrivate() out/linux-x64x11-v8_debug/../../cobalt/script/v8c/wrapper_private.h:86:3
#5 0xb9285ba in cobalt::script::v8c::WrapperPrivate::Callback(v8::WeakCallbackInfo<cobalt::script::v8c::WrapperPrivate> const&) out/linux-x64x11-v8_debug/../../cobalt/script/v8c/wrapper_private.h:38:5
#6 0x667f45c in v8::internal::GlobalHandles::PendingPhantomCallback::Invoke(v8::internal::Isolate*) out/linux-x64x11-v8_debug/../../v8/src/global-handles.cc:848:3
#7 0x6680408 in v8::internal::GlobalHandles::DispatchPendingPhantomCallbacks(bool) out/linux-x64x11-v8_debug/../../v8/src/global-handles.cc:813:16
#8 0x668084b in v8::internal::GlobalHandles::PostGarbageCollectionProcessing(v8::internal::GarbageCollector, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/global-handles.cc:869:18
#9 0x66a8a7b in v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1601:37
#10 0x66a6c48 in v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) out/linux-x64x11-v8_debug/../../v8/src/heap/heap.cc:1216:11
#11 0x65bdfc9 in v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) out/linux-x64x11-v8_debug/../../v8/src/factory.cc:91:3
#12 0x710c27d in v8::internal::__RT_impl_Runtime_AllocateInNewSpace(v8::internal::Arguments, v8::internal::Isolate*) out/linux-x64x11-v8_debug/../../v8/src/runtime/runtime-internal.cc:322:31
#13 0x710b9c3 in v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) out/linux-x64x11-v8_debug/../../v8/src/runtime/runtime-internal.cc:315:1
#14 0x7f36af5047a3 (<unknown module>)
#15 0x7f36af5513fb (<unknown module>)
#16 0x7f36af5be594 (<unknown module>)
#17 0x7f36af5b583e (<unknown module>)
#18 0x7f36af5be594 (<unknown module>)
#19 0x7f36af5b583e (<unknown module>)
#20 0x7f36af5be594 (<unknown module>)
#21 0x7f36af5bb818 (<unknown module>)
#22 0x7f36af5040fe (<unknown module>)
#23 0x6599b9a in v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, bool, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*, v8::internal::Handle<v8::internal::Object>, v8::internal::Execution::MessageHandling) out/linux-x64x11-v8_debug/../../v8/src/execution.cc:146:13
#24 0x6598454 in v8::internal::(anonymous namespace)::CallInternal(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*, v8::internal::Execution::MessageHandling) out/linux-x64x11-v8_debug/../../v8/src/execution.cc:182:10
#25 0x6597dd6 in v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) out/linux-x64x11-v8_debug/../../v8/src/execution.cc:192:10
#26 0x54a5627 in v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) out/linux-x64x11-v8_debug/../../v8/src/api.cc:5449:7
#27 0xb96677e in cobalt::dom::V8cEventListener::HandleEvent(scoped_refptr<cobalt::script::Wrappable> const&, scoped_refptr<cobalt::dom::Event> const&, bool*) const out/linux-x64x11-v8_debug/gen/bindings/browser/source/cobalt/dom/v8c_event_listener.cc:90:20
#28 0x3a2962b in cobalt::dom::EventListener::HandleEvent(scoped_refptr<cobalt::dom::Event> const&, cobalt::dom::EventListener::Type) const out/linux-x64x11-v8_debug/../../cobalt/dom/event_listener.cc:95:7
#29 0x3a1e705 in cobalt::dom::EventTarget::FireEventOnListeners(scoped_refptr<cobalt::dom::Event> const&) out/linux-x64x11-v8_debug/../../cobalt/dom/event_target.cc:200:31
previously allocated by thread T30 (MainWebModule) here:
#0 0x23083d2 in operator new(unsigned long) (out/linux-x64x11-v8_debug/cobalt+0x23083d2)
#1 0x24b530b in cobalt::browser::WebModule::Impl::InjectKeyboardEvent(scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&) out/linux-x64x11-v8_debug/../../cobalt/browser/web_module.cc:791:7
#2 0x253ca5e in base::internal::RunnableAdapter<void (cobalt::browser::WebModule::Impl::*)(scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&)>::Run(cobalt::browser::WebModule::Impl*, scoped_refptr<cobalt::dom::Element> const&, base::Token const&, cobalt::dom::KeyboardEventInit const&) out/linux-x64x11-v8_debug/../../base/bind_internal.h:317:12
#3 0x2556929 in base::internal::InvokeHelper<false, void, base::internal::RunnableAdapter<void (cobalt::browser::WebModule::Impl::*)(scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&)>, void (cobalt::browser::WebModule::Impl*, cobalt::dom::Element*, base::Token const&, cobalt::dom::KeyboardEventInit const&)>::MakeItSo(base::internal::RunnableAdapter<void (cobalt::browser::WebModule::Impl::*)(scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&)>, cobalt::browser::WebModule::Impl*, cobalt::dom::Element*, base::Token const&, cobalt::dom::KeyboardEventInit const&) out/linux-x64x11-v8_debug/../../base/bind_internal.h:966:14
#4 0x25565d6 in base::internal::Invoker<4, base::internal::BindState<base::internal::RunnableAdapter<void (cobalt::browser::WebModule::Impl::*)(scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&)>, void (cobalt::browser::WebModule::Impl*, scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&), void (base::internal::UnretainedWrapper<cobalt::browser::WebModule::Impl>, scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit)>, void (cobalt::browser::WebModule::Impl*, scoped_refptr<cobalt::dom::Element>, base::Token, cobalt::dom::KeyboardEventInit const&)>::Run(base::internal::BindStateBase*) out/linux-x64x11-v8_debug/../../base/bind_internal.h:1571:12
#5 0x2322063 in base::Callback<void ()>::Run() const out/linux-x64x11-v8_debug/../../base/callback.h:412:12
#6 0x25d3932 in MessageLoop::RunTask(base::PendingTask const&) out/linux-x64x11-v8_debug/../../base/message_loop.cc:548:21
#7 0x25d41f9 in MessageLoop::DeferOrRunPendingTask(base::PendingTask const&) out/linux-x64x11-v8_debug/../../base/message_loop.cc:560:5
#8 0x25d451e in MessageLoop::DoWork() out/linux-x64x11-v8_debug/../../base/message_loop.cc:744:13
#9 0x25f1e1b in base::MessagePumpDefault::Run(base::MessagePump::Delegate*) out/linux-x64x11-v8_debug/../../base/message_pump_default.cc:31:31
#10 0x25d2648 in MessageLoop::RunInternal() out/linux-x64x11-v8_debug/../../base/message_loop.cc:491:10
#11 0x25d23f4 in MessageLoop::RunHandler() out/linux-x64x11-v8_debug/../../base/message_loop.cc:463:3
#12 0x265e81f in base::RunLoop::Run() out/linux-x64x11-v8_debug/../../base/run_loop.cc:47:10
#13 0x25d0cbc in MessageLoop::Run() out/linux-x64x11-v8_debug/../../base/message_loop.cc:363:12
#14 0x26f2398 in base::Thread::Run(MessageLoop*) out/linux-x64x11-v8_debug/../../base/threading/thread.cc:159:17
#15 0x26f28a3 in base::Thread::ThreadMain() out/linux-x64x11-v8_debug/../../base/threading/thread.cc:212:5
#16 0x26e58bc in base::(anonymous namespace)::ThreadFunc(void*) out/linux-x64x11-v8_debug/../../base/threading/platform_thread_starboard.cc:44:13
#17 0x2d1f982 in (anonymous namespace)::ThreadFunc(void*) out/linux-x64x11-v8_debug/../../starboard/shared/pthread/thread_create.cc:73:10
#18 0x7f37370ab183 in start_thread /build/eglibc-ripdx6/eglibc-2.19/nptl/pthread_create.c:312
Thread T30 (MainWebModule) created by T0 here:
#0 0x22c381d in __interceptor_pthread_create (out/linux-x64x11-v8_debug/cobalt+0x22c381d)
#1 0x2d1f489 in SbThreadCreate out/linux-x64x11-v8_debug/../../starboard/shared/pthread/thread_create.cc:124:12
#2 0x26e5504 in base::(anonymous namespace)::CreateThread(unsigned long, SbThreadPriority, int, bool, char const*, base::PlatformThread::Delegate*, unsigned long*) out/linux-x64x11-v8_debug/../../base/threading/platform_thread_starboard.cc:59:21
#3 0x26e5787 in base::PlatformThread::CreateWithOptions(base::PlatformThread::PlatformThreadOptions const&, base::PlatformThread::Delegate*, unsigned long*) out/linux-x64x11-v8_debug/../../base/threading/platform_thread_starboard.cc:142:10
#4 0x26f195e in base::Thread::StartWithOptions(base::Thread::Options const&) out/linux-x64x11-v8_debug/../../base/threading/thread.cc:96:8
#5 0x24bb450 in cobalt::browser::WebModule::WebModule(GURL const&, base::ApplicationState, base::Callback<void (cobalt::layout::LayoutManager::LayoutResults const&)> const&, base::Callback<void (GURL const&, std::string const&)> const&, base::Callback<void (base::TimeDelta)> const&, base::Callback<void ()> const&, cobalt::media::CanPlayTypeHandler*, cobalt::media::WebMediaPlayerFactory*, cobalt::network::NetworkModule*, cobalt::math::Size const&, float, cobalt::render_tree::ResourceProvider*, float, cobalt::browser::WebModule::Options const&) out/linux-x64x11-v8_debug/../../cobalt/browser/web_module.cc:1226:11
#6 0x23495ab in cobalt::browser::BrowserModule::Navigate(GURL const&) out/linux-x64x11-v8_debug/../../cobalt/browser/browser_module.cc:519:25
#7 0x2343f7c in cobalt::browser::BrowserModule::BrowserModule(GURL const&, base::ApplicationState, base::EventDispatcher*, cobalt::account::AccountManager*, cobalt::browser::BrowserModule::Options const&) out/linux-x64x11-v8_debug/../../cobalt/browser/browser_module.cc:372:3
#8 0x23124f7 in cobalt::browser::Application::Application(base::Callback<void ()> const&, bool) out/linux-x64x11-v8_debug/../../cobalt/browser/application.cc:614:11
#9 0x230cb48 in (anonymous namespace)::StartApplication(int, char**, char const*, base::Callback<void ()> const&) out/linux-x64x11-v8_debug/../../cobalt/browser/main.cc:78:25
#10 0x230c0b2 in void cobalt::wrap_main::BaseEventHandler<&(anonymous namespace)::PreloadApplication, &(anonymous namespace)::StartApplication, &(anonymous namespace)::HandleStarboardEvent, &(anonymous namespace)::StopApplication>(SbEvent const*) out/linux-x64x11-v8_debug/../../cobalt/base/wrap_main_starboard.h:74:7
#11 0x230b214 in SbEventHandle out/linux-x64x11-v8_debug/../../cobalt/browser/main.cc:105:1
#12 0x2d246b4 in starboard::shared::starboard::Application::DispatchAndDelete(starboard::shared::starboard::Application::Event*) out/linux-x64x11-v8_debug/../../starboard/shared/starboard/application.cc:307:3
#13 0x2d23685 in starboard::shared::starboard::Application::DispatchStart() out/linux-x64x11-v8_debug/../../starboard/shared/starboard/application.cc:194:3
#14 0x2d2242b in starboard::shared::starboard::Application::RunLoop() out/linux-x64x11-v8_debug/../../starboard/shared/starboard/application.cc:382:5
#15 0x2d228b8 in starboard::shared::starboard::Application::Run(starboard::shared::starboard::CommandLine) out/linux-x64x11-v8_debug/../../starboard/shared/starboard/application.cc:111:10
#16 0x277ca1f in starboard::shared::starboard::Application::Run(int, char**) out/linux-x64x11-v8_debug/../../starboard/shared/starboard/application.h:166:12
#17 0x277c849 in main out/linux-x64x11-v8_debug/../../starboard/linux/x64x11/main.cc:31:26
#18 0x7f37360cff44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
SUMMARY: AddressSanitizer: heap-use-after-free out/linux-x64x11-v8_debug/../../cobalt/script/v8c/v8c_heap_tracer.cc:76:16 in cobalt::script::v8c::V8cHeapTracer::AdvanceTracing(double, v8::EmbedderHeapTracer::AdvanceTracingActions)
Shadow bytes around the buggy address:
0x0c18800250a0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800250b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c18800250c0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c18800250d0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c18800250e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c18800250f0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
0x0c1880025100: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
0x0c1880025110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
0x0c1880025120: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1880025130: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c1880025140: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==184290==ABORTING
Log output (logging GC cycle begin/end, object marking, and the weak callback running):
$ cat log.txt | grep -P '(Begin|End|0x60c000199540)'
[0204/150945:INFO:v8c_heap_tracer.cc(53)] Begin GC cycle 1
[0204/151001:INFO:v8c_heap_tracer.cc(37)] 0x60c000199540 got registered by RegisterV8References during GC cycle 1
[0204/151001:INFO:v8c_heap_tracer.cc(172)] 0x60c000199540 got marked during GC cycle 1
[0204/151001:INFO:wrapper_private.h(78)] 0x60c000199540 is dying in GC cycle 1
==190737==ERROR: AddressSanitizer: heap-use-after-free on address 0x60c000199540 at pc 0x0000053ad9de bp 0x7fe4ebb76270 sp 0x7fe4ebb76268
READ of size 8 at 0x60c000199540 thread T30 (MainWebModule)
0x60c000199540 is located 0 bytes inside of 120-byte region [0x60c000199540,0x60c0001995b8)
With the default configuration V8 marks wrappers, i.e., objects pointing into embedder memory, incrementally.
The requirements are
An v8::EmbedderHeapTracer implementation figuring out (transitive) liveness of V8 wrappers based on references passed to it from V8.
Consistency mechanisms, .e.g., write barriers, when DOM is mutated during incremental marking.
Informing V8's young generation garbage collector to keep v8::Persistent alive during incremental marking.
3 is probably subtle. The steps involved are:
Register a callback for Scavenges using Isolate::AddGCPrologueCallback
Visit weak persistent handles using Isolate::VisitWeakHandles
Mark weak persistents as active using v8::PersistentBase<T>::MarkActive
Steps 2 and 3 can be omitted when turning off incremental tracing of wrappers using the flag incremental_marking_wrappers.
Related
How to Set the Number of Threads on PyTorch Hosted on AWS Lambda
I'm trying to set the number of threads via torch.set_num_threads(multiprocessing.cpu_count()) to speed up inference on AWS Lambda. However, it gives me the following stack trace.
2022-03-28T16:48:11.625-07:00 Error in cpuinfo: failed to parse the list of possible processors in /sys/devices/system/cpu/possible 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:11.625-07:00 Error in cpuinfo: failed to parse the list of present processors in /sys/devices/system/cpu/present 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:11.625-07:00 Error in cpuinfo: failed to parse both lists of possible and present processors 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:11.801-07:00 /var/lang/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:11.801-07:00 warn(f"Failed to load image Python extension: {e}") 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00
Copy
terminate called after throwing an instance of 'c10::Error'
terminate called after throwing an instance of 'c10::Error' 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 what(): [enforce fail at ThreadPool.cc:44] cpuinfo_initialize(). cpuinfo initialization failed 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x50 (0x7faf9dc2a0 in /var/lang/lib/python3.9/site-packages/torch/lib/libc10.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00
Copy
frame #1: c10::ThrowEnforceNotMet(char const*, int, char const*, char const*, void const*) + 0x50 (0x7faf9dc440 in /var/lang/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::ThrowEnforceNotMet(char const*, int, char const*, char const*, void const*) + 0x50 (0x7faf9dc440 in /var/lang/lib/python3.9/site-packages/torch/lib/libc10.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 frame #2: <unknown function> + 0x1d6cb7c (0x7fb17abb7c in /var/lang/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 frame #3: <unknown function> + 0x1d6fa34 (0x7fb17aea34 in /var/lang/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 frame #4: at::set_num_threads(int) + 0x2c (0x7fb02d082c in /var/lang/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 frame #5: <unknown function> + 0x498864 (0x7fb52ff864 in /var/lang/lib/python3.9/site-packages/torch/lib/libtorch_python.so) 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 <omitting python frames> 2022/03/28/[$LATEST]5935023a5e514dbd8cc15036a23cedfe
2022-03-28T16:48:14.200-07:00 Fatal Python error: Aborted
Does anyone know how to fix this? For context, I'm deploying a docker image that runs the PyTorch model but the one line I linked above results in this error.
I expected that the program would speed up. Instead, it crashed without a helpful stack trace.
Segfault inserting 64 bit number into stringstream on 32 bit Linux
I have some code as follows:
uint64_t now;
std::stringstream timestamp;
...
now = GetElapsedTime();
timestamp << "#" << std::setfill('0') << std::setw(6) << (now / 1000u) << ".";
On a 64-bit Windows system this compiles and runs fine.
On a 32-bit Ubuntu Linux system the code compiles and runs fine, but then I get a segfault. The backtrace is as follows:
#0 0xb7fe132d in check_match (
undef_name=undef_name#entry=0xb7e53396 "__udivdi3",
ref=ref#entry=0xb7e1fc70, version=version#entry=0xb7fd2690, flags=1,
type_class=1, sym=0x401638, symidx=323, strtab=0x401678 "",
map=0xb7fff940, versioned_sym=0xbfffecc4, num_versions=0xbfffecc0)
at dl-lookup.c:120
#1 0xb7fe173d in do_lookup_x (
undef_name=undef_name#entry=0xb7e53396 "__udivdi3",
new_hash=new_hash#entry=4161312091, old_hash=old_hash#entry=0xbfffed44,
ref=0xb7e1fc70, result=0xbfffed4c, scope=0xb7fffa9c, i=<optimised out>,
version=0xb7fd2690, flags=1, skip=0x0, type_class=1, undef_map=0xb7fd1400)
at dl-lookup.c:406
#2 0xb7fe1f9b in _dl_lookup_symbol_x (undef_name=0xb7e53396 "__udivdi3",
undef_map=0xb7fd1400, ref=0xbfffedc4, symbol_scope=0xb7fd15b8,
version=0xb7fd2690, type_class=1, flags=1, skip_map=0x0) at dl-lookup.c:813
#3 0xb7fe6ff8 in _dl_fixup (l=<optimised out>, reloc_arg=<optimised out>)
at dl-runtime.c:112
#4 0xb7fece20 in _dl_runtime_resolve () at ../sysdeps/i386/dl-trampoline.S:43
#5 0xb7f0a15c in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#6 0xb7f0b2fe in std::ostreambuf_iterator<char, std::char_traits<char> >
std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >
::_M_insert_int<unsigned long long>(std::ostreambuf_iterator<char,
std::char_traits<char> >, std::ios_base&, char, unsigned long long) const ()
from /usr/lib/i386-linux-gnu/libstdc++.so.6
#7 0xb7f0b4b4 in std::num_put<char, std::ostreambuf_iterator<char,
std::char_traits<char> > >::do_put(std::ostreambuf_iterator<char,
std::char_traits<char> >, std::ios_base&, char, unsigned long long) const ()
from /usr/lib/i386-linux-gnu/libstdc++.so.6
#8 0xb7f17403 in std::ostream& std::ostream::_M_insert<unsigned long long>(unsigned long
long) () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#9 0xb7f17590 in std::ostream::operator<<(unsigned long long) ()
from /usr/lib/i386-linux-gnu/libstdc++.so.6
As a guess, it seems that the gcc compiler thinks that there is an overload for the << operator with an unsigned long long value, but thinks it needs to dynamically load a library to execute it. However it doesn't seem to be able to locate the symbol. I am not sure if it has loaded a library at this point or not.
Can anyone please give me some pointers on how to solve this issue? I tried casting the value to a uint32_t and that fixed the issue, but it is a nasty work-around. There must be a proper solution.
Unreal Engine Random Crashing on arch Linux Unhandled Exception: SIGABRT
I am running arch with an Nvidia graphics card Around every six minutes Unreal Engine will randomly close and give me a bunch of warning and errors in the log Here is the UE4.Log [2017.05.19-00.47.14:159][579]LogLinux: === Critical error: === Unhandled Exception: SIGABRT: abort() called [2017.05.19-00.47.14:159][579]LogLinux: Fatal error! [Callstack] 00 0x00007f2d9cc8fdaf FLinuxPlatformStackWalk::CaptureStackBackTrace(unsigned long long*, unsigned int, void*) [Callstack] 01 0x00007f2d9cb71f86 FGenericPlatformStackWalk::StackWalkAndDump(char*, unsigned long, int, void*) [Callstack] 02 0x00007f2d9cc509c2 FLinuxCrashContext::CaptureStackTrace() [Callstack] 03 0x00007f2d92d3e1f0 CommonLinuxCrashHandler(FGenericCrashContext const&) [Callstack] 04 0x00007f2d9cc55185 PlatformCrashHandler(int, siginfo_t*, void*) [Callstack] 05 0x00007f2da297efe0 /usr/lib/libpthread.so.0(+0x11fe0) [0x7f2da297efe0] [Callstack] 06 0x00007f2d9211ea10 /usr/lib/libc.so.6(gsignal+0x110) [0x7f2d9211ea10] [Callstack] 07 0x00007f2d9212013a /usr/lib/libc.so.6(abort+0x17a) [0x7f2d9212013a] [Callstack] 08 0x00007f2d9215d2b0 /usr/lib/libc.so.6(+0x722b0) [0x7f2d9215d2b0] [Callstack] 09 0x00007f2d9216390e /usr/lib/libc.so.6(+0x7890e) [0x7f2d9216390e] [Callstack] 10 0x00007f2d9216411e /usr/lib/libc.so.6(+0x7911e) [0x7f2d9216411e] [Callstack] 11 0x00007f2d75722287 FOpenGLRHIState::CleanupResources() [Callstack] 12 0x00007f2d7571d58c FOpenGLDynamicRHI::Cleanup() [Callstack] 13 0x00007f2d7571d370 FOpenGLDynamicRHI::Shutdown() [Callstack] 14 0x00007f2d96c00fe2 RHIExit() [Callstack] 15 0x0000000000424fcc FEngineLoop::Exit() [/home/jayden/newU/UnrealEngine-release/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp, line 319] [Callstack] 16 0x0000000000425492 GuardedMain(wchar_t const*) [/home/jayden/newU/UnrealEngine-release/Engine/Source/Runtime/Launch/Private/Launch.cpp, line 178] [Callstack] 17 0x00007f2d92d3f0b6 CommonLinuxMain(int, char**, int (*)(wchar_t const*)) [Callstack] 18 0x00007f2d9210b511 /usr/lib/libc.so.6(__libc_start_main+0xf1) [0x7f2d9210b511] [Callstack] 19 0x00000000004153ba ./UE4Editor(_start+0x2a) [0x4153ba] The Full log is here https://pastebin.com/PuKxNSXK Any Help would be greatly appreciated, Thanks =)
Here is the standard procedure for this : Reboot Your Computer (Then try again) Google it! If you can use other projects without any problem, then you did something with the project, in which case you should tell me the procedure of making the project. IF EVERYTHING FAILS... Reinstall Unreal Engine Try another operating system.
GDB debug output for multi-thread program
All, I am debuging a 24-thread program with GDB, now I have find which line in the code the error occurs, but I cannot tell what the error is from the output of GDB. The followsing line of code leads to the error, it's just a normal insertion to a map structure. current_node->children.insert(std::pair<string, ComponentTrieNode*>(comps[j], temp_node)); I used GDB to find out in which thread the error happens and switched to that thread, the backtrace command shows the function calls in the stack. (The last several lines try to print the value of some variables in a function, but failed.) What should I do to clear know what error is happening? [root#localhost nameComponentEncoding]# gdb NCE_david GNU gdb (GDB) Fedora (7.2.90.20110429-36.fc15) Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david...done. (gdb) r /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt Starting program: /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt [Thread debugging using libthread_db enabled] [New Thread 0x7fffd2bf5700 (LWP 13129)] [New Thread 0x7fffd23f4700 (LWP 13130)] [New Thread 0x7fffd1bf3700 (LWP 13131)] [New Thread 0x7fffd13f2700 (LWP 13132)] [New Thread 0x7fffd0bf1700 (LWP 13133)] [New Thread 0x7fffd03f0700 (LWP 13134)] [New Thread 0x7fffcfbef700 (LWP 13135)] [New Thread 0x7fffcf3ee700 (LWP 13136)] [New Thread 0x7fffcebed700 (LWP 13137)] [New Thread 0x7fffce3ec700 (LWP 13138)] [New Thread 0x7fffcdbeb700 (LWP 13139)] [New Thread 0x7fffcd3ea700 (LWP 13140)] [New Thread 0x7fffccbe9700 (LWP 13141)] [New Thread 0x7fffcc3e8700 (LWP 13142)] [New Thread 0x7fffcbbe7700 (LWP 13143)] [New Thread 0x7fffcb3e6700 (LWP 13144)] [New Thread 0x7fffcabe5700 (LWP 13145)] [New Thread 0x7fffca3e4700 (LWP 13146)] [New Thread 0x7fffc9be3700 (LWP 13147)] [New Thread 0x7fffc93e2700 (LWP 13148)] [New Thread 0x7fffc8be1700 (LWP 13149)] [New Thread 0x7fffc83e0700 (LWP 13150)] [New Thread 0x7fffc7bdf700 (LWP 13151)] this is thread 1 this is thread 7 this is thread 14 this is thread 18 this is thread 2 this is thread 19 this is thread 6 this is thread 8 this is thread 24 base: 64312646 this is thread 11 this is thread 5 this is thread 12 this is thread 13 this is thread 3 this is thread 15 this is thread 16 this is thread 17 this is thread 4 this is thread 20 this is thread 21 this is thread 22 this is thread 23 this is thread 9 this is thread 10 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffc8be1700 (LWP 13149)] std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=#0x608118) at ../../../../libstdc++-v3/src/tree.cc:126 126 __x->_M_right = __y->_M_left; (gdb) info threads Id Target Id Frame 24 Thread 0x7fffc7bdf700 (LWP 13151) "NCE_david" compare (__n=<optimized out>, __s2=<optimized out>, __s1=<optimized out>) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/char_traits.h:257 (... other 22 threads not listed) 2 Thread 0x7fffd2bf5700 (LWP 13129) "NCE_david" compare (__n=<optimized out>, __s2=<optimized out>, __s1=<optimized out>) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/char_traits.h:257 1 Thread 0x7ffff7fe57a0 (LWP 13126) "NCE_david" strtok () at ../sysdeps/x86_64/strtok.S:76 (gdb) thread 22 [Switching to thread 22 (Thread 0x7fffc8be1700 (LWP 13149))] #0 std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=#0x608118) at ../../../../libstdc++-v3/src/tree.cc:126 126 __x->_M_right = __y->_M_left; (gdb) bt #0 std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=#0x608118) at ../../../../libstdc++-v3/src/tree.cc:126 #1 0x0000003cdd26e848 in std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x7fffc0005ba0, __p=<optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:266 #2 0x00000000004029ca in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*> > >::_M_insert_ (this=0x608108, __x=<optimized out>, __p=0x16cd3e30, __v=...) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_pair.h:87 #3 0x0000000000402b7d in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*>, std::_Select1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ComponentTrieNode*> > >::_M_insert_unique (this=0x608108, __v=...) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_tree.h:1281 #4 0x000000000040444c in insert (__x=..., this=0x608108) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_map.h:518 #5 ComponentTrie::add_prefix (this=0x7fffffffe2e0, prefix_input=<optimized out>, port=10) at ComponentTrie_david.cpp:112 #6 0x0000000000401c3b in main._omp_fn.0 () at NameComponentEncoding_david.cpp:277 #7 0x0000003cd2607fea in gomp_thread_start (xdata=<optimized out>) at ../../../libgomp/team.c:115 #8 0x0000003cd0607cd1 in start_thread (arg=0x7fffc8be1700) at pthread_create.c:305 #9 0x0000003cd02dfd3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) p 'ComponentTrie::add_prefix(char*, int)'::comps[j] No symbol "comps" in specified context. (gdb) p 'ComponentTrie::add_prefix(char*, int)'::prefix No symbol "prefix" in specified context. Edit: I have run the code with valgrind --tool=memcheck, the following is the result. [root#localhost nameComponentEncoding]# valgrind --tool=memcheck ./NCE_david /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt (... many lines omitted) ==13261== ==13261== Thread 11: ==13261== Invalid read of size 1 ==13261== at 0x3CD02849BC: strtok (strtok.S:141) ==13261== by 0x40426A: ComponentTrie::add_prefix(char*, int) (ComponentTrie_david.cpp:99) ==13261== by 0x40242C: main._omp_fn.0 (NameComponentEncoding_david.cpp:531) ==13261== by 0x3CD2607FE9: gomp_thread_start (team.c:115) ==13261== by 0x3CD0607CD0: start_thread (pthread_create.c:305) ==13261== by 0x3CD02DFD3C: clone (clone.S:115) ==13261== Address 0x234422c02 is not stack'd, malloc'd or (recently) free'd ==13261== ==13261== Invalid read of size 1 ==13261== at 0x3CD02849EC: strtok (strtok.S:167) ==13261== by 0x40426A: ComponentTrie::add_prefix(char*, int) (ComponentTrie_david.cpp:99) ==13261== by 0x40242C: main._omp_fn.0 (NameComponentEncoding_david.cpp:531) ==13261== by 0x3CD2607FE9: gomp_thread_start (team.c:115) ==13261== by 0x3CD0607CD0: start_thread (pthread_create.c:305) ==13261== by 0x3CD02DFD3C: clone (clone.S:115) ==13261== Address 0x234422c02 is not stack'd, malloc'd or (recently) free'd ==13261== Insertion and lookup cost time(us): 994669532 67108864 14.821731 0.067469 component number:4849478, state number: 2545847 Parallel threads:24 ==13261== ==13261== HEAP SUMMARY: ==13261== in use at exit: 4,239,081,584 bytes in 76,746,193 blocks ==13261== total heap usage: 80,050,114 allocs, 3,303,921 frees, 4,323,622,103 bytes allocated ==13261== ==13261== LEAK SUMMARY: ==13261== definitely lost: 0 bytes in 0 blocks ==13261== indirectly lost: 0 bytes in 0 blocks ==13261== possibly lost: 4,111,951,106 bytes in 74,746,429 blocks ==13261== still reachable: 127,130,478 bytes in 1,999,764 blocks ==13261== suppressed: 0 bytes in 0 blocks ==13261== Rerun with --leak-check=full to see details of leaked memory ==13261== ==13261== For counts of detected and suppressed errors, rerun with: -v ==13261== Use --track-origins=yes to see where uninitialised values come from ==13261== ERROR SUMMARY: 45 errors from 30 contexts (suppressed: 6 from 6)
We know that the program is segfaulting on this line: current_node->children.insert(std::pair<string, ComponentTrieNode*>(comps[j], temp_node)); From the stack trace, we know that the segfault happens deep in the red black tree implementation of std::map: #0 std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=#0x608118) at ../../../../libstdc++-v3/src/tree.cc:126 126 __x->_M_right = __y->_M_left; This implies that: The segfault could be caused by: evaluating __x->_M_right evaluating __y->_M_left storing the right hand side to the left hand side of __x->_M_right = __y->_M_left std::map::insert() being called implies that the segfault was NOT caused while building the arguments to the call. In particular comps[j] is not out of bounds. This leads me to think that your heap was already corrupted by previous memory operation errors by this time and that the crash in std::map::insert() is a symptom and not a cause. Run your program under the Valgrind memcheck tool: $ valgrind --tool=memcheck /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt and carefully read Valgrind's output afterwards to find the first memory error in your program. Valgrind is implemented as a virtual CPU, so your program would slow down by a factor of ~30. This is time consuming but should allow you to make progress in troubleshooting the problem. In addition to Valgrind, you might also want to try enabling debug mode for the libstdc++ containers: To use the libstdc++ debug mode, compile your application with the compiler flag -D_GLIBCXX_DEBUG. Note that this flag changes the sizes and behavior of standard class templates such as std::vector, and therefore you can only link code compiled with debug mode and code compiled without debug mode if no instantiation of a container is passed between the two translation units. If your program uses no external libraries then rebuilding the whole thing with -D_GLIBCXX_DEBUG added to CXXFLAGS in the Makefile should work. Otherwise you'd need to know whether C++ containers are passed between components compiled with and without the debug flag. Valgrind Log Review I'm surprised that you're using strtok() in a multi-threaded program. Is ComponentTrie::add_prefix() never called from two threads concurrently? While fixing the invalid read by inspecting how strtok() is used on ComponentTrie_david.cpp:99, you might want to replace strtok() with strtok_r() as well. Concurrent Access to STL Containers The standard C++ containers are explicitly documented to not do thread synchronization: The user code must guard against concurrent function calls which access any particular library object's state when one or more of those accesses modifies the state. An object will be modified by invoking a non-const member function on it or passing it as a non-const argument to a library function. An object will not be modified by invoking a const member function on it or passing it to a function as a pointer- or reference-to-const. Typically, the application programmer may infer what object locks must be held based on the objects referenced in a function call and whether the objects are accessed as const or non-const. (That's from the GNU libstdc++ documentation but the C++11 standard essentially specifies the same behavior) Concurrent modifications of std::map and other containers is a serious error and likely the culprit that caused the crash. Guard each container with their own pthread_mutex_t or use the OpenMP synchronization mechanisms.
Why is ImageMagick within node.js crashing?
I'm using node.js, node-vips and libvips compiled with ImageMagick to convert and resize images. I'm getting segmentation faults and failed assertions when I try and resize more than a couple of images. I've had so many different crashes I'm not sure where to begin. I started off with libvips 7.26.8, I've also tried 7.30.7. This is with node v0.8.17 compiled from source, on a fairly standard, clean ubuntu box. #0 0x0000158ac3765c59 in ?? () Cannot access memory at address 0x7fffa837ec90 #0 0x0000000000000000 in ?? () #1 0x0000000000000000 in ?? () node: ../deps/uv/src/unix/stream.c:729: uv__stream_io: Assertion `!!(events & EV_READ) ^ !!(events & EV_WRITE)' failed. Aborted (core dumped) #0 0x00007f5ed4df9425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f5ed4dfcb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f5ed4df20ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007f5ed4df2192 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00000000005d1bc8 in uv__stream_io (loop=<optimized out>, w=<optimized out>, events=<optimized out>) at ../deps/uv/src/unix/stream.c:729 #5 0x00000000005c6ac2 in ev_invoke_pending (loop=0xdb34c0 <default_loop_struct>) at ../deps/uv/src/unix/ev/ev.c:2145 #6 0x00000000005c2986 in uv__poll (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:246 #7 uv__run (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:257 #8 0x00000000005c2c60 in uv_run (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:265 #9 0x000000000057d9f7 in node::Start(int, char**) () #10 0x00007f5ed4de476d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #11 0x0000000000575245 in _start () #0 0x00000000006b13b8 in v8::Object::SetHiddenValue(v8::Handle<v8::String>, v8::Handle<v8::Value>) () #1 0x0000000000593f5a in node::SlabAllocator::Allocate(v8::Handle<v8::Object>, unsigned int) () #2 0x0000000000591d04 in node::StreamWrap::OnAlloc(uv_handle_s*, unsigned long) () #3 0x00000000005d08fb in uv__read (stream=0x1fa7f90) at ../deps/uv/src/unix/stream.c:575 #4 0x00000000005d1a1a in uv__stream_io (loop=<optimized out>, w=<optimized out>, events=<optimized out>) at ../deps/uv/src/unix/stream.c:745 #5 0x00000000005c6ac2 in ev_invoke_pending (loop=0xdb34c0 <default_loop_struct>) at ../deps/uv/src/unix/ev/ev.c:2145 #6 0x00000000005c29df in uv__poll (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:248 #7 uv__run (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:257 #8 0x00000000005c2c60 in uv_run (loop=0xdb27e0 <default_loop_struct>) at ../deps/uv/src/unix/core.c:265 #9 0x000000000057d9f7 in node::Start(int, char**) () #10 0x00007f065aa1176d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #11 0x0000000000575245 in _start () More often than not I get one of the first two errors – ie, no stack trace. These all occurred while trying to resize 6 or so images – occasionally they all succeed without error, but usually it seg faults after the first one or two have been resized. How on earth do I go about debugging this? In the unit test for the node-vips plugin there's a comment that reads: this test will crash if vips is compiled with imagemagick support because imagemagick crashes when called from libeio Why is this? Is this still true? I thought ImageMagick was perfectly thread-safe, what about it makes it not safe to be called from libeio/libuv?