PyO3 linker error when using "extension-module" feature - rust

When I try to build very simple a PyO3 module, the compilation stops at the link stage and returns a very long error message. I compile this code on Linux.
The Rust project looks like this:
├ Cargo.toml
└ src/
└ lib.rs
Cargo.toml:
[package]
name = "dytest"
version = "0.1.0"
edition = "2021"
[lib]
name = "dytest"
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.16", features = ["extension-module"] }
lib.rs:
use pyo3::prelude::*;
#[test]
fn test_print() {
pyo3::prepare_freethreaded_python();
Python::with_gil(|py| py.run( "print('Hello World')", None, None ) );
}
When I now run cargo test, I get a linker error:
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" [...many, many paths...]
= note: /usr/bin/ld: /home/user/dytest/target/debug/deps/libpyo3-c98e45c2daa36b66.rlib(pyo3-c98e45c2daa36b66.pyo3.b04632a8-cgu.2.rcgu.o): in function `pyo3_ffi::object::Py_DECREF':
/home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.16.2/src/object.rs:407: undefined reference to `_Py_Dealloc'
/usr/bin/ld: /home/user/dytest/target/debug/deps/libpyo3-c98e45c2daa36b66.rlib(pyo3-c98e45c2daa36b66.pyo3.b04632a8-cgu.2.rcgu.o): in function `pyo3_ffi::object::Py_None':
/home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.16.2/src/object.rs:472: undefined reference to `_Py_NoneStruct'
/usr/bin/ld: /home/user/dytest/target/debug/deps/libpyo3-c98e45c2daa36b66.rlib(pyo3-c98e45c2daa36b66.pyo3.b04632a8-cgu.2.rcgu.o): in function `pyo3::err::PyErr::take':
/home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.16.2/src/err/mod.rs:264: undefined reference to `PyErr_Fetch'
[...many, many error messages...]
/usr/bin/ld: /home/user/dytest/target/debug/deps/libpyo3-c98e45c2daa36b66.rlib(pyo3-c98e45c2daa36b66.pyo3.b04632a8-cgu.15.rcgu.o): in function `pyo3::types::string::PyString::to_string_lossy':
/home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.16.2/src/types/string.rs:199: undefined reference to `PyUnicode_AsEncodedString'
collect2: error: ld returned 1 exit status
= help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
= note: use the `-l` flag to specify native libraries to link
= note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
error: could not compile `dytest` due to previous error
If I replace
pyo3 = { version = "0.16", features = ["extension-module"] }
with
pyo3 = { version = "0.16" }
everything works, but the PyO3 User Guide explicitly states this feature.
Trying this code on a Windows machine, works.
I assume I am missing some kind of dependency, but I can't figure out, what is missing.

It looks like this is a known PyO3 issue. It is mentioned in their FAQ:
Cargo.toml:
[dependencies.pyo3]
version = "0.16.3"
[features]
extension-module = ["pyo3/extension-module"]
default = ["extension-module"]
cargo test fails with linking errors when the extension-module feature is activated. A workaround is to make the extension-module feature optional and running the tests with
cargo test --no-default-features.

Related

Rust linker errors: using "-Wl,--as-needed" when linking system libraries

I'm trying to write an application primarily in rust that uses a gtk-based frontend written in C++. I've gotten pretty far in getting the build setup in Cargo, but it's failing in the linking stage.
I've created a minimal reproducible example.
// build.rs
extern crate cc;
extern crate pkg_config;
fn main() {
let gtk = pkg_config::probe_library("gtk+-3.0").unwrap();
cc::Build::new()
.cpp(true)
.file("src/gui.cc")
.includes(gtk.include_paths)
.compile("gui");
}
// src/gui.cc
// This is essentially the first example from https://docs.gtk.org/gtk3/getting_started.html
#include <gtk/gtk.h>
extern "C" {
int run_gui();
}
static void activate(GtkApplication* app, gpointer) {
GtkWidget* window = gtk_application_window_new (app);
gtk_window_set_title (GTK_WINDOW (window), "Window");
gtk_window_set_default_size (GTK_WINDOW (window), 200, 200);
gtk_widget_show_all (window);
}
int run_gui() {
GtkApplication* app = gtk_application_new ("org.gtk.example", G_APPLICATION_DEFAULT_FLAGS);
g_signal_connect (app, "activate", G_CALLBACK (activate), nullptr);
int status = g_application_run (G_APPLICATION (app), 0, nullptr);
g_object_unref (app);
return status;
}
// src/main.rs
extern "C" {
fn run_gui() -> core::ffi::c_int;
}
fn main() {
unsafe { run_gui(); }
}
When I cargo build the package, compiling the c++ and rust files seems to work fine, but then I get undefined reference errors on the link stage.
/usr/bin/ld: /home/pete/workspaces/csvtk/so-minimal-gtk/target/debug/build/so-minimal-gtk-75c7a9a61d32d2e6/out/libgui.a(gui.o): in function `activate(_GtkApplication*, void*)':
/home/pete/workspaces/csvtk/so-minimal-gtk/src/gui.cc:9: undefined reference to `gtk_application_window_new'
/usr/bin/ld: /home/pete/workspaces/csvtk/so-minimal-gtk/src/gui.cc:10: undefined reference to `gtk_window_get_type'
...
The link line, which is quite long, is also logged before that error. It includes all the required gtk libraries, -lgtk-3, -lcairo, etc. When I run it outside of cargo, it fails with the same errors, but, when I remove the -Wl,--as-needed flag from the link line, it links correctly and the program runs fine.
Why did rustc add -Wl,--as-needed? Is there a reason those gtk symbols aren't "needed" from the perspective of the linker? Any tips on how to fix this problem elegantly?
In general, using --as-needed as a flag to the linker is extremely helpful because it prevents your binary from being linked to libraries it doesn't use directly. As an example of a reason why this is beneficial, if you depend on libfoo 1.0, which depends on libbar 1.0, and later on libfoo 1.1 updates to libbar 2.0, then as long as you don't use libbar directly, your code will continue to work with --as-needed, but won't work without it (unless libbar has symbol versioning).
The kind of impact is visible here when you use objdump -x on the binary after making the change I suggest below:
NEEDED libgtk-3.so.0
NEEDED libgio-2.0.so.0
NEEDED libgobject-2.0.so.0
NEEDED libgcc_s.so.1
NEEDED libc.so.6
NEEDED ld-linux-x86-64.so.2
ldd lists 70 libraries linked into your program, and you're only directly requiring to 6, including the dynamic linker.
You can usually rely on other crates to specify their link dependencies correctly. However, you need to be careful because when you're creating your own dependency on a library using C or C++, you have to specify the libraries to link with yourself. This can be seen if you use cargo build -v, where you can notice that your gui library is linked after the dependencies of libgtk-3:
Running `rustc --crate-name test_repo --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=c60f7e6ab8348a08 -C extra-filename=-c60f7e6ab8348a08 --out-dir /tmp/user/1000/test-repo/target/debug/deps -C incremental=/tmp/user/1000/test-repo/target/debug/incremental -L dependency=/tmp/user/1000/test-repo/target/debug/deps -L native=/usr/lib/x86_64-linux-gnu -L native=/tmp/user/1000/test-repo/target/debug/build/test-repo-c89dae0927f2103a/out -l gtk-3 -l gdk-3 -l z -l pangocairo-1.0 -l pango-1.0 -l harfbuzz -l atk-1.0 -l cairo-gobject -l cairo -l gdk_pixbuf-2.0 -l gio-2.0 -l gobject-2.0 -l glib-2.0 -l static=gui -l stdc++`
So you'd want to do something more like this in your build.rs:
// build.rs
extern crate cc;
extern crate pkg_config;
fn main() {
let gtk = pkg_config::probe_library("gtk+-3.0").unwrap();
cc::Build::new()
.cpp(true)
.file("src/gui.cc")
.includes(gtk.include_paths)
.compile("gui");
for path in gtk.link_paths {
println!("cargo:rustc-link-search={}", path.display());
}
for lib in gtk.libs {
println!("cargo:rustc-link-lib={}", lib);
}
}

I'm unable to run rust winit application on Alpine (Wayland)

I'm following this tutorial to create a winit window with Rust on Alpine Linux.
When I started the demo application described in the tutorial using cargo run it not compile. But after installing build-base cmake musl-dev libpng-dev freetype-dev fontconfig-dev it was compiling. However, it fails to run, and it throws the following error:
thread 'main' panicked at 'Failed to initialize any backend! Wayland status: NoWaylandLib X11 status: LibraryOpenError(OpenError { kind: Library, detail: "opening library failed (Dynamic loading not supported); opening library failed (Dynamic loading not supported)" })', /home/user/.cargo/registry/src/github.com-1ecc6299db9ec823/winit-0.27.5/src/platform_impl/linux/mod.rs:719:9
It seems to me that it is unable to use a dynamic library because it's MUSL. I don't know how to fix this. (It does compile, but does not run!)
System info:
OS: Linux 5.15.80-0-lts, Alpine v3.17 x86_64
WM: sway (wayland)
Rust: Toolchain stable-x86_64-unknown-linux-musl, rustc 1.65.0
Shell: ash
What I've tried:
I have tried to install some Xorg and Gui dev libraries like libxtst-dev libxext-dev libxrender-dev freetype-dev fontconfig-dev libxslt glib-dev musl-dev libxcursor-dev libxi-dev libx11-dev glu-dev glew-dev mesa-dev libxcb-dev libxkbcommon-dev libx11-dev xproto lbxft-dev libxext-dev libxcb-dev libxkbcommon-dev just to be sure. That did not work
I tried to run the application using RUSTFLAGS="-C target-feature=-crt-static" but that did not work
I've tried to run the application using env var RUSTFLAGS="-C target-feature=+crt-static" to compile it static instead of linked (I don't know if this was correct) but that throws an exception
I've tried to run the application using env var RUSTFLAGS="-C target-feature=-crt-static". That did compile and run, but no window shows up. And I'm not sure if that is the way to go. Also that bugs cargo to recompile all dependencies every time I've tried to change something in the code.
I've found no documentation or anything on the internet that could help me out, telling how to run this on Alpine or musl.
I was expecting that with the correct dependencies the program compiles and a window shows up.
Can anybody help me out?
This is my cargo.toml:
[package]
name = "gpu-programming"
version = "0.1.0"
edition = "2021"
[dependencies]
wgpu = ">=0.14"
winit = ">=0.27"
env_logger = ">=0.10"
log = ">=0.4"
And this is the code I'm talking about (main.rs)
use winit::{
event::{ElementState, Event, KeyboardInput, VirtualKeyCode, WindowEvent},
event_loop::{ControlFlow, EventLoop},
window::WindowBuilder,
};
pub fn main() {
env_logger::init(); // Make sure WGPU errors are printed to console. Else it will fail silently!
// Create event loop and window
let event_loop = EventLoop::new();
let window = WindowBuilder::new()
.build(&event_loop)
.expect("Failed to create window");
println!("Window created! {:?}", window.id());
window.set_visible(true);
println!("Visible {:?}", window.is_visible());
println!("Monitor: {:?}", window.current_monitor());
event_loop.run(move |event, _, control_flow| match event {
Event::WindowEvent {
ref event,
window_id,
} if window_id == window.id() => match event {
WindowEvent::CloseRequested
| WindowEvent::KeyboardInput {
input:
KeyboardInput {
state: ElementState::Pressed,
virtual_keycode: Some(VirtualKeyCode::Escape),
..
},
..
} => *control_flow = ControlFlow::Exit,
_ => {}
},
_ => {}
});
}

oxipng throwing RuntimeError: unreachable when called

I'm trying to create a small WASM project for image compression.
After some search in github, I noticed that oxipng 2.2.2 has a target for wasm32-unknown-unknown, hence why I'm using that.
I'm using wasm-pack for creating the wasm file + JS bindings with target -t web
This is the code:
extern crate oxipng;
mod utils;
use std::error::Error;
use wasm_bindgen::prelude::*;
use oxipng::*;
#[wasm_bindgen]
extern "C" {
// Use `js_namespace` here to bind `console.log(..)` instead of just
// `log(..)`
#[wasm_bindgen(js_namespace = console)]
fn log(s: &str);
}
// Next let's define a macro that's like `println!`, only it works for
// `console.log`. Note that `println!` doesn't actually work on the wasm target
// because the standard library currently just eats all output. To get
// `println!`-like behavior in your app you'll likely want a macro like this.
#[macro_export]
macro_rules! console_log {
// Note that this is using the `log` function imported above during
// `bare_bones`
($($t:tt)*) => (crate::log(&format_args!($($t)*).to_string()))
}
// When the `wee_alloc` feature is enabled, use `wee_alloc` as the global
// allocator.
#[cfg(feature = "wee_alloc")]
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;
#[wasm_bindgen]
pub fn compress(data: &[u8]) -> Vec<u8> {
console_log!("{}", data.len());
let opts = Options::from_preset(6);
console_log!("after options");
let res = match optimize_from_memory(data, &&opts) {
Ok(res) => Ok(res),
Err(err) => Err(err),
};
match &res {
Ok(_) => console_log!("Optimized"),
Err(err) => console_log!("Error: {}", err),
}
return res.unwrap();
}
I don't ever get an error message, the last log I have is "after options".
In a nutshell, I'm using a Flutter web application that gets a PNG file, converts it into a Uint8List, and I send it as an integer List to the JS bindings.
When called, the following error happens:
RuntimeError: unreachable
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[1019]:0x5c6be
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[414]:0x4cd37
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[619]:0x54c96
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[915]:0x5b4ba
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[986]:0x5c139
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[645]:0x55885
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[569]:0x5324b
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[594]:0x53ff1
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[2]:0x554f
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[84]:0x2cbf2
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[73]:0x2a501
at http://localhost:3000/pkg/rust_png_module_bg.wasm:wasm-function[563]:0x52eaa
at compress (http://localhost:3000/pkg/rust_png_module.js:49:14)
at compressImage (http://localhost:3000/packages/rust_wasm/ui/screens/home/home_page.dart.lib.js:568:72)
at compressImage.next (<anonymous>)
at http://localhost:3000/dart_sdk.js:38640:33
at _RootZone.runUnary (http://localhost:3000/dart_sdk.js:38511:59)
at _FutureListener.thenAwait.handleValue (http://localhost:3000/dart_sdk.js:33713:29)
at handleValueCallback (http://localhost:3000/dart_sdk.js:34265:49)
at Function._propagateToListeners (http://localhost:3000/dart_sdk.js:34303:17)
at _Future.new.[_completeWithValue] (http://localhost:3000/dart_sdk.js:34151:23)
at async._AsyncCallbackEntry.new.callback (http://localhost:3000/dart_sdk.js:34172:35)
at Object._microtaskLoop (http://localhost:3000/dart_sdk.js:38778:13)
at _startMicrotaskLoop (http://localhost:3000/dart_sdk.js:38784:13)
at http://localhost:3000/dart_sdk.js:34519:9
Since this version is old, I don't know if I should revert back to an older version of Rust
$ rustup --version
rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.55.0 (c8dfcfe04 2021-09-06)`
Thank you in advance
The problem is you're using a very old version of oxipng (v2.2.2) that didn't support wasm yet. I believe wasm support was added in v2.3.0 (link to issue that was fixed). Anyways, you should be able to use the latest version just fine with wasm, just make sure you disable the default features when adding the crate to your Cargo.toml:
[dependencies]
oxipng = { version = "5", default-features = false }

Why do I get the error "there is no reactor running, must be called from the context of Tokio runtime" even though I have #[tokio::main]?

I'm following the mdns Rust documentation and pasted the example code but it throws the following error:
thread 'main' panicked at 'there is no reactor running, must be called from the context of Tokio runtime'
Here's the code that I have:
use futures_util::{pin_mut, stream::StreamExt};
use mdns::{Error, Record, RecordKind};
use std::{net::IpAddr, time::Duration};
const SERVICE_NAME: &'static str = "_googlecast._tcp.local";
#[tokio::main]
async fn main() -> Result<(), Error> {
// Iterate through responses from each Cast device, asking for new devices every 15s
let stream = mdns::discover::all(SERVICE_NAME, Duration::from_secs(15))?.listen();
pin_mut!(stream);
while let Some(Ok(response)) = stream.next().await {
let addr = response.records().filter_map(self::to_ip_addr).next();
if let Some(addr) = addr {
println!("found cast device at {}", addr);
} else {
println!("cast device does not advertise address");
}
}
Ok(())
}
fn to_ip_addr(record: &Record) -> Option<IpAddr> {
match record.kind {
RecordKind::A(addr) => Some(addr.into()),
RecordKind::AAAA(addr) => Some(addr.into()),
_ => None,
}
}
Dependencies:
[dependencies]
mdns = "1.1.0"
futures-util = "0.3.8"
tokio = { version = "0.3.3", features = ["full"] }
What am I missing? I tried looking online but haven't found how to create a reactor for this use case.
You are using a newer version of Tokio, such as 0.3 or 1.x, and many packages, including mdns 1.1.0, rely on an older version of Tokio, such as 0.2.
% cargo tree -d
tokio v0.2.22
└── mdns v1.1.0
└── example_project v0.1.0
tokio v0.3.3
└── example_project v0.1.0
For now, you will need to match versions of the Tokio runtime. The easiest way is to use Tokio 0.2 yourself. The tokio-compat-02 crate may also be useful in some cases.
See also:
Why is a trait not implemented for a type that clearly has it implemented?
Various error messages with the same root cause:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
there is no reactor running, must be called from the context of Tokio runtime
not currently running on the Tokio runtime
Fix for me was adding this to Cargo.toml:
[dependencies]
async-std = { version = "1", features = ["attributes", "tokio1"] }
https://github.com/ATiltedTree/ytextract/issues/25
At the time of writing, a fair amount of crates are already using Tokio v1, but others might still be under an experimental phase. Check your crates for prerelease versions which might have already upgraded their tokio runtime compatibility.
A relevant example of this was actix-web, which uses runtime 1.0 of Tokio since version 4. Although there were prereleases of this major increment since 2022-01-07, version 4.0.0 was only released in 2022-02-25.
actix-web = { version = "4.0.0-beta.10" }

What dependencies are needed to write rust/gstreamer plugins for gstreamer crate 0.14?

I am trying to follow the tutorial at https://coaxion.net/blog/2018/01/how-to-write-gstreamer-elements-in-rust-part-1-a-video-filter-for-converting-rgb-to-grayscale/ about writing gstreamer plugins using rust.
If you follow the tutorial to the first point where I have compilable code Cargo.toml is
[package]
name = "gst-plugin-tutorial"
version = "0.1.0"
authors = ["Sebastian Dröge <sebastian#centricular.com>"]
repository = "https://github.com/sdroege/gst-plugin-rs"
license = "MIT/Apache-2.0"
[dependencies]
glib = "0.4"
gstreamer = "0.10"
gstreamer-base = "0.10"
gstreamer-video = "0.10"
gst-plugin = "0.1"
[lib]
name = "gstrstutorial"
crate-type = ["cdylib"]
path = "src/lib.rs"
and src/lib.rs is
extern crate glib;
#[macro_use]
extern crate gstreamer as gst;
extern crate gstreamer_base as gst_base;
extern crate gstreamer_video as gst_video;
#[macro_use]
extern crate gst_plugin;
plugin_define!(
b"rstutorial\0",
b"Rust Tutorial Plugin\0",
plugin_init,
b"1.0\0",
b"MIT/X11\0",
b"rstutorial\0",
b"rstutorial\0",
b"https://github.com/sdroege/gst-plugin-rs\0",
b"2017-12-30\0"
);
fn plugin_init(plugin: &gst::Plugin) -> bool {
true
}
This compiles, but the project for which I need to write a plugin uses gstreamer 1.16, so it needs rust crate gstreamer 0.14.
When I alter the Cargo.toml to reference recent versions of the gstreamer crate:
[dependencies]
#glib = "0.4"
gstreamer = "0.14"
gstreamer-base = "0.14"
gstreamer-video = "0.14"
gst-plugin = "0.3.2"
I get errors at build time:
Updating crates.io index
error: failed to select a version for `glib-sys`.
... required by package `gstreamer-base v0.14.0`
... which is depended on by `gst-plugin-tutorial v0.1.0 (/home/thoth/src/rust-gst-plugin-exp/coaxion-plugin)`
versions that meet the requirements `^0.9` are: 0.9.0
the package `glib-sys` links to the native library `glib`, but it conflicts with a previous package which links to `glib` as well:
package `glib-sys v0.7.0`
... which is depended on by `gst-plugin v0.3.2`
... which is depended on by `gst-plugin-tutorial v0.1.0 (/home/thoth/src/rust-gst-plugin-exp/coaxion-plugin)`
failed to select a version for `glib-sys` which could resolve this conflict
What is the proper mix of crate versions to write gstreamer plugins in rust for use with gstreamer 1.16 ?
You can find new versions of the tutorials here and the latest version of the code here.
Your problem is that you're still using the gst-plugin crate, but that's obsolete nowadays and everything's part of the glib / gstreamer / gstreamer-base / etc crates now if you enable the subclass feature of them. See the links above for the details.
Depending on the old version of the gst-plugin crate will pull in an older version of the glib-sys (and others) crate, and you can't have two different versions of a -sys crate in the same project.
You'll have the same problem again if you uncomment the glib dependency. Once you update that to the 0.8 version of glib, that error would also go away.
As a variation on Sebastian's answer, I experimented with a Cargo.toml that doesn't point at git and uses released crates.
glib = "0.8"
gstreamer = "0.14"
gstreamer-base = "0.14"
gstreamer-video = "0.14"
#gst-plugin = "0.3.2"
That failed to provide a definition for gst_plugin_define! . It seems that is part of the subclassing feature. Switching to the following dependencies:
glib = { version = "0.8", features = [ "subclassing"] }
gstreamer = { version = "0.14", features = [ "subclassing"] }
gstreamer-base = { version = "0.14", features = [ "subclassing"] }
gstreamer-video = "0.14"
activated the code defining the gst_plugin_define! macro.

Resources