Running markdown example with wit-bindgen - rust

The WebAssembly Interface Types was removed in favor of wit-bindgen as discussed here: https://github.com/bytecodealliance/wasmtime/issues/677#issuecomment-1024087373.
I am trying to compile the old markdown demo using interface types support available at https://github.com/bytecodealliance/wasmtime-demos, and import the wasm file in a python runtime. While the compilation works, I am unable to import the module since the feature was removed. Therefore I am trying to rewrite it with wit-bindgen: https://github.com/bytecodealliance/wit-bindgen
This is my project setup:
Cargo.toml
[package]
name = "markdown"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ['cdylib']
[dependencies]
pulldown-cmark = "0.5.3"
wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen" }
src/render.wit
render: func(name: string) -> string
src/lib.rs
use pulldown_cmark::{html, Parser};
wit_bindgen_rust::export!("./src/render.wit");
struct Render;
impl render::Render for Render {
fn render(input: String) -> String {
let parser = Parser::new(&input);
let mut html_output = String::new();
html::push_html(&mut html_output, parser);
return html_output;
}
}
I can compile this with cargo build and move the wasm file to the root directory with cp /target/wasm32-unknown-unknown/release/markdown.wasm .. The python bindings file bindings.py is generated by issuing wit-bindgen wasmtime-py --import src/render.wit.
How do I finally import the wasm file in python? I have tried the following:
import wasmtime.loader
import markdown
but this gives me the error:
File ~/.local/lib/python3.10/site-packages/wasmtime/loader.py:72, in _WasmtimeLoader.exec_module(self, module)
70 break
71 field_name = wasm_import.name
---> 72 imported_module = importlib.import_module(module_name)
73 item = imported_module.__dict__[field_name]
74 if not isinstance(item, Func) and \
75 not isinstance(item, Table) and \
76 not isinstance(item, Global) and \
77 not isinstance(item, Memory):
File /usr/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
124 break
125 level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '__wbindgen_placeholder__'
I believe this is because the bindings.py file is not being used. How do i specify it using python-wasmtime ?

I recently did something similar with wit-bindgen and python.
In your Python code you can do something like this to make it work:
from wasmtime import Engine, Store, Config, Module, Linker, WasiConfig
from md import bindings
config = Config()
engine = Engine(config)
store = Store(engine)
store.set_wasi(WasiConfig())
module = Module.from_file(engine, path)
linker = Linker(engine)
linker.define_wasi()
markdown = bindings.Markdown(store, linker, module)
res = markdown.render(store, text)
You can also take a look here for the full example:
https://github.com/olavvatne/python-webassembly-rust

Related

pyo3 rust module kills Python on import

I'm writing my first Rust-based Python module, and it kills the Python process on import. I've got it down to a pretty minimal example, based loosely on the html-py-ever example (which does run for me without crashing).
I'm running Python 3.8 on an M1 macbook, Python is compiled for arm64.
% python -c "import platform;print(platform.machine())"
arm64
My output, reproducer command using the files pasted below. The install should take care of any python requirements:
(rust) jeremytemp#Jeremy-McGibbons-MacBook-Pro minimal % pip install -e . && python test.py
Obtaining file:///Users/jeremytemp/rust/minimal
Installing collected packages: minimal
Attempting uninstall: minimal
Found existing installation: minimal 0.1.0
Uninstalling minimal-0.1.0:
Successfully uninstalled minimal-0.1.0
Running setup.py develop for minimal
Successfully installed minimal-0.1.0
zsh: killed python test.py
src/lib.rs:
use pyo3::{prelude::*, wrap_pyfunction};
#[pyfunction]
fn foo() -> PyResult<u64>{
let u: u64 = 1;
Ok(u)
}
#[pymodule]
fn minimal(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(foo, m)?)?;
Ok(())
}
Cargo.toml:
[package]
name = "minimal"
version = "0.1.0"
edition = "2021"
[dependencies]
pyo3 = { features = ["extension-module"] }
[lib]
name = "minimal"
crate-type = ["cdylib"]
setup.py:
from setuptools import setup
from setuptools_rust import RustExtension
setup(
rust_extensions=[RustExtension("minimal.minimal")],
)
setup.cfg:
[metadata]
name = minimal
version = 0.1.0
license = MIT
[options]
packages = minimal
zip_safe = False
setup_requires = setuptools-rust >= 0.12.1;
python_requires = >=3.8
include_package_data = True
minimal/__init__.py:
from .minimal import *
test.py:
import minimal
pip freeze output is
(rust) jeremytemp#Jeremy-McGibbons-MacBook-Pro minimal % pip freeze
attrs==21.4.0
beautifulsoup4==4.11.1
certifi==2021.10.8
iniconfig==1.1.1
# Editable Git install with no remote (minimal==0.1.0)
-e /Users/jeremytemp/rust/minimal
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.8
pytest==7.1.2
semantic-version==2.9.0
setuptools-rust==1.3.0
soupsieve==2.3.2.post1
tomli==2.0.1
typing_extensions==4.2.0
What am I doing wrong? Are there any steps I can take to get more helpful debugging output than "killed"?
Not a very satisfying answer, but the example executes fine on my windows machine. I'm assuming this is an issue with pyo3 on M1 Macs.

MyPy can't find types for my local package

I have two python packages. One is a util library, and one is an application that will use the util library (eventually I will have more apps sharing the library.
I am using poetry for both, and the app specifies the common library as a dependency using the path and develop properties.
For example, my layout looks something like this:
- common/
- common/
- __init__.py
- py.typed
- pyproject.toml
- myapp/
- myapp/
- __init__.py
- py.typed
- pyproject.toml
And the myapp\pyproject.toml looks something like this:
[tool.poetry]
name = "myapp"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
[tool.poetry.dependencies]
python = "^3.9"
common = { path = "../common", develop = true }
[tool.poetry.dev-dependencies]
mypy = "^0.910"
flake8 = "^4.0.1"
black = {version = "^21.12b0", allow-prereleases = true}
pytest = "^6.2.5"
pytest-cov = "^3.0.0"
pytest-mock = "^3.6.1"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
When I run mypy on myapp I get something like:
myapp/__init__.py:1:1: error: Cannot find implementation or library stub for module named "common" [import]
Assuming the "util" package has type hints, you'll want to add a py.typed file to it (in the root directory of the package) so mypy understands it comes with type hints. py.typed should be empty, it's just a flag file.
If your package does not have type hints, then you'd have to add stubs files (.pyi).
More details: https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages

How to deal with unresolved import 'concrete' issues in RUST

I am new to RUST, and I am trying to use this concrete library: https://github.com/zama-ai/concrete/tree/master/concrete. I am trying to create a simple "Hello World" in RUST to see if concrete imports correctly. I followed the instructions in the aforementioned link.
Specifically, I:
Cloned the GitHub repo.
Cd into concrete folder (/concrete/concrete)
ran "cargo new play_with_fhe"
Updated the "Cargo.toml" file with the new member "play_with_me"
[workspace]
members = [
"concrete",
"concrete-npe",
"concrete-core",
"concrete-csprng",
"concrete-commons",
"concrete-tasks",
"concrete-boolean",
"play_with_fhe",
]
[profile.bench]
opt-level = 3
debug = true
lto="fat"
[patch.crates-io]
concrete = {path="concrete"}
concrete-npe = {path="concrete-npe"}
concrete-core = {path="concrete-core"}
concrete-csprng = {path="concrete-csprng"}
concrete-commons = {path="concrete-commons"}
concrete-boolean = {path= "concrete-boolean"}
play_with_fhe = {path= "play_with_fhe"}
Cd into "/concrete/concrete/play_with_fhe" and updated the "Cargo.toml" file with
[package]
name = "play_with_fhe"
version = "0.1.11"
authors = ["FHE Curious"]
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
concrete = "^0.1.11"
Cd into /concrete/concrete/play_with_fhe_/src and created a main.rs file running a simple code:
use concrete::*;
fn main() {
println!("Hello, world!");
}
When I try compiling it with rustc main.rs, I get told that:
error[E0432]: unresolved import `concrete`
--> main.rs:2:5
|
2 | use concrete::*;
| ^^^^^^^^ maybe a missing crate `concrete`?
error: aborting due to previous error
For more information about this error, try `rustc --explain E0432`.
How can I address this issue? Any advice would be appreciated.
Since you're trying to create your own hello world project, you don't need to clone the repository. You just need to create a project, include concrete as a dependency, and then import it. Those instructions are on the concrete page (as Stargateur notes):
% cargo new play_with_fhe
% cd play_with_fhe
Add concrete to your dependencies in Cargo.toml:
[package]
name = "play_with_fhe"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
concrete = "^0.1" # <--- This is the only change you make. The rest is template.
Then add use cargo::* to the top of your main.rs, and build it:
% cargo build
This will install everything. For more on Cargo, see Dependencies in Rust By Example.
Note that this package likely won't build correctly unless you're on an x86 architecture. For example, it won't run on an Apple M1 without Rosetta2.

How can I add the build version to a scons build

At the moment I'm using some magic to get the current git revision into my scons builds.. I just grab the version a stick it into CPPDEFINES.
It works quite nicely ... until the version changes and scons wants to rebuild everything, rather than just the files that have changed - becasue the define that all files use has changed.
Ideally I'd generate a file using a custom builder called git_version.cpp and
just have a function in there that returns the right tag. That way only that one file would be rebuilt.
Now I'm sure I've seen a tutorial showing exactly how to do this .. but I can't seem to track it down. And I find the custom builder stuff a little odd in scons...
So any pointers would be appreciated...
Anyway just for reference this is what I'm currently doing:
# Lets get the version from git
# first get the base version
git_sha = subprocess.Popen(["git","rev-parse","--short=10","HEAD"], stdout=subprocess.PIPE ).communicate()[0].strip()
p1 = subprocess.Popen(["git", "status"], stdout=subprocess.PIPE )
p2 = subprocess.Popen(["grep", "Changed but not updated\\|Changes to be committed"], stdin=p1.stdout,stdout=subprocess.PIPE)
result = p2.communicate()[0].strip()
if result!="":
git_sha += "[MOD]"
print "Building version %s"%git_sha
env = Environment()
env.Append( CPPDEFINES={'GITSHAMOD':'"\\"%s\\""'%git_sha} )
You don't need a custom Builder since this is just one file. You can use a function (attached to the target version file as an Action) to generate your version file. In the example code below, I've already computed the version and put it into an environment variable. You could do the same, or you could put your code that makes git calls in the version_action function.
version_build_template="""/*
* This file is automatically generated by the build process
* DO NOT EDIT!
*/
const char VERSION_STRING[] = "%s";
const char* getVersionString() { return VERSION_STRING; }
"""
def version_action(target, source, env):
"""
Generate the version file with the current version in it
"""
contents = version_build_template % (env['VERSION'].toString())
fd = open(target[0].path, 'w')
fd.write(contents)
fd.close()
return 0
build_version = env.Command('version.build.cpp', [], Action(version_action))
env.AlwaysBuild(build_version)

How to create a symbolic link with SCons?

I'm using SCons for building a project and need to add a symbolic link to a file it is installing via env.Install. What command(s) will make a link that's the equivalent of running ln -s on the command line?
SCons doesn't have a dedicated symbolic link command, but you can use os.symlink(src, dst) from Python's os module:
import os
env = Environment()
def SymLink(target, source, env):
os.symlink(os.path.abspath(str(source[0])), os.path.abspath(str(target[0])))
env.Command("file.out", "file.in", SymLink)
This may not work correctly on Windows, I've only tried it on Linux.
There seems to be little advancement in the SCons core code for symbolic link support and I wasn't satisfied any one solution I found on the web. Here is a potential builder which incorporates aspects of both Nick's and richq's answers. Additionally, it will catch name changes (due to the emitter method) and is as platform-agnostic as I could get it.
I prefer this builder because it will make links relative to the directory in which they are installed. One could add an option to force the link to be absolute I suppose, but I have not needed or wanted that yet.
Currently, if the OS doesn't support symlinks, I just pass and do nothing, but one could use os.copytree() for example however the dependency becomes messy if the source is a directory so the emitter would need to do something fancy. I'm up for any suggestions here.
One can put the following code into the file site_scons/site_tools/symlink.py (with blank _init_.py files in the appropriate places). Then do this in the SConstruct file:
SConstruct:
env = Environment()
env.Tool('symlink')
env.SymLink('link_name.txt', 'real_file.txt')
symlink.py:
import os
from os import path
from SCons.Node import FS
from SCons.Script import Action, Builder
def generate(env):
'''
SymLink(link_name,source)
env.SymLink(link_name,source)
Makes a symbolic link named "link_name" that points to the
real file or directory "source". The link produced is always
relative.
'''
bldr = Builder(action = Action(symlink_builder,symlink_print),
target_factory = FS.File,
source_factory = FS.Entry,
single_target = True,
single_source = True,
emitter = symlink_emitter)
env.Append(BUILDERS = {'SymLink' : bldr})
def exists(env):
'''
we could test if the OS supports symlinks here, or we could
use copytree as an alternative in the builder.
'''
return True
def symlink_print(target, source, env):
lnk = path.basename(target[0].abspath)
src = path.basename(source[0].abspath)
return 'Link: '+lnk+' points to '+src
def symlink_emitter(target, source, env):
'''
This emitter removes the link if the source file name has changed
since scons does not seem to catch this case.
'''
lnk = target[0].abspath
src = source[0].abspath
lnkdir,lnkname = path.split(lnk)
srcrel = path.relpath(src,lnkdir)
if int(env.get('verbose',0)) > 3:
ldir = path.relpath(lnkdir,env.Dir('#').abspath)
if rellnkdir[:2] == '..':
ldir = path.abspath(ldir)
print ' symbolic link in directory: %s' % ldir
print ' %s -> %s' % (lnkname,srcrel)
try:
if path.exists(lnk):
if os.readlink(lnk) != srcrel:
os.remove(lnk)
except AttributeError:
# no symlink available, so we remove the whole tree? (or pass)
#os.rmtree(lnk)
print 'no os.symlink capability on this system?'
return (target, source)
def symlink_builder(target, source, env):
lnk = target[0].abspath
src = source[0].abspath
lnkdir,lnkname = path.split(lnk)
srcrel = path.relpath(src,lnkdir)
if int(env.get('verbose',0)) > 4:
print 'target:', target
print 'source:', source
print 'lnk:', lnk
print 'src:', src
print 'lnkdir,lnkname:', lnkdir, lnkname
print 'srcrel:', srcrel
if int(env.get('verbose',0)) > 4:
print 'in directory: %s' % path.relpath(lnkdir,env.Dir('#').abspath)
print ' symlink: %s -> %s' % (lnkname,srcrel)
try:
os.symlink(srcrel,lnk)
except AttributeError:
# no symlink available, so we make a (deep) copy? (or pass)
#os.copytree(srcrel,lnk)
print 'no os.symlink capability on this system?'
return None
This creates a builder to perform the job:
mylib = env.SharedLibrary("foobar", SRCS)
builder = Builder(action = "ln -s ${SOURCE.file} ${TARGET.file}", chdir = True)
env.Append(BUILDERS = {"Symlink" : builder})
mylib_link = env.Symlink("_foobar.so", mylib)
env.Default(mylib)
env.Default(mylib_link)
Again, this solution is for Linux.
If you wanted to issue the command directly to the shell and know the OS, subprocess can be used as well.
E.g.: subprocess.call(['ln', '-s', '</src/path>', '</dest/path>'])
In addition to Nicks solution, you can add a directory symlink by using a file as a directory name carrier. It's not the cleanest solution and debugging path names is a pain, but this works well:
def symlink_last(target_source_env):
src = os.path.basename(os.path.dirname(str(source[0])))
link = "deliverables/last"
print "Symlinking "+ src + "as" + link
os.symlink(src, link)
BUILD_TARGETS.append('link')
install_dir = "deliverables/subdir"
carrier_file = "filename"
builder = Builder(action = symlink_last, chdir=False)
env.Append(BUILDERS={ "Symlink" : builder })
env.Alias(target="link", source=env.Symlink(dir="deliverables", source = install_dir + carrier_file)
This will make a link to deliverables/subdir named deliverables/last, provided that a file deliverables/subdir/filename exists.

Resources