go_remote_library usage in Pants - pants

I am currently attempting to use the go_remote_library target??, package??, plugin?? in Pants. Real simple question, here:
If in my code I have the import listed as:
import(
"github.com/golang/groupcache"
)
is it valid for me to specify a name of simply "groupcache" instead of the full import path? Here is what my BUILD file looks like:
go_remote_library(name="groupcache",
rev="d781998583680cda80cf61e0b37dd0cd8da2eb52"
)
Am I doing this right? As a side note, is there a Pants target that I can use to test that my BUILD file is valid? Thanks!

You are doing it right. All of the go targets - go_remote_library in this case, but also go_library and go_binary - currently take a name parameter and it must be the name of the directory the BUILD file lives in. The next release of pants (0.0.44) should remove the name parameter taking the choice away from you.
The 1st line of defense is the BUILD Dictionary.
For go_remote_library you'll find this doc.
As to testing, the simplest test is checking syntax, and for that this does the trick:
./pants list path/to/BUILD:
Note the trailing colon attached to the path
This says "List all the targets defined in path/to/BUILD. Here the : means all - its equivalent to the * wildcard in bourne shells for pants targets in BUILD files.
If you want to check more targets all at once you could say:
./pants list ::
Here the recursive glob is used - equivalent to ** in zsh, and so this asks pants to list all the targets in the repo.
If the syntax checks out, you may still have more subtle issues, like defining a go_remote_library that does not point to a valid github project. These issues will only show up when you try to do more than act on the target's metadata like list and depmap goals do. For a go_remote_library, the simplest way to exercise it is to try and resolve the library:
./pants resolve 3rdparty/go/github.com/bitly/go-simplejson2
If you have this BUILD file contents at that path:
go_remote_library(name='go-simplejson2')
Running the resolve will fail since no such github repo exists.
You can do a similar higher-level check with go_library and go_binary targets, instead running ./pants compile .... This will smoke out whether you're missing any required go_remote_library BUILD files or dependencies.

Related

Project-specific override for Cargo

I primarily want to use Debian's Rust packages, rather than fetching some random code from the wider Internet (I'm old-fashioned, I know, let's not get into that part). To this end, my ~/.cargo/config.toml looks like
[net]
offline = true
[source]
[source.crates-io]
replace-with = "debian"
[source.debian]
directory = "/usr/share/cargo/registry"
This works great after I install the librust-*-dev packages that I desire. However, in some specific projects, I'd like to lift this rule and tell Cargo "hey, you can in fact go wild and get whatever you want from crates.io". According to the Cargo book, a project-specific /project/.cargo/config.toml should take precedence over my user one. Assume this project-specific .cargo/config.toml:
[net]
offline = false
[source]
[source.crates-io]
I'm still not able to cargo build a project with dependencies from outside of my replacement source. If for example, I make a Cargo.toml that depends on yew (a randomly chosen crate that I know isn't available in my replacement source) I get
$ cargo build
error: no matching package found
searched package name: `yew`
What am I misunderstanding about Cargo's sources, replacement and per-project overrides?
The answer suggested by #blackgreen is one possible workaround for the underlying problem until issues 10045 and 10057 (or a combination thereof) are solved. Another, perhaps slightly less ugly, workaround follows below for those who need it.
I ended up working around the problem using UnionFS (I guess the more modern OverlayFS should work well too).
I simply add
[source.crates-io]
replace-with = "union"
[source.union]
directory = "/home/gspr/.cargo-overlay/union-registry"
to my ~/.cargo/config.toml and then do
unionfs -o ro /usr/share/cargo/registry:/home/gspr/.cargo-overlay/local-registry /home/gspr/.cargo-overlay/union-registry
Now /home/gspr/.cargo-overlay/union-registry reflects the union of /usr/share/cargo/registry and /home/gspr/.cargo-overlay/local-registry, with priority to the former in case of conflicts.
So what goes in ~/.cargo-overlay/local-registry? Individual extra crates, in the same way as in Debian's /usr/share/cargo/registry. That is to say, directories named cratename-version as they are distributed by upstream – but with a single extra file, namely .cargo-checksum.json added to them. The content of that extra file can be extracted from the crates.io index as follows.
Suppose we have cloned the crates.io index into ~/.cargo-overlay/crates.io-index, i.e.
git clone https://github.com/rust-lang/crates.io-index.git ~/.cargo-overlay/crates.io-index
Then suppose we've extracted a crate foo at version 0.1.2 into ~/.cargo-overlay/local-registry/foo-0.1.2. We can generate the missing .cargo-checksum.json like so:
cd ~/.cargo-overlay
index_file=$(find crates.io-index -type f -name foo)
cksum=$(jq -r "select(.name == \"foo\" and .vers == \"0.1.2\" ) | .cksum" ${index_file})
jo package="${cksum}" files="{}" > local-registry/foo-0.1.2/.cargo-checksum.json
It looks as if you are suffering from this issue: https://github.com/rust-lang/cargo/issues/8687
You would like to unset a config key on a upper-level config.toml but this is not supported.
I've played a bit with the config, and the only way I got it to work was to overwrite in the project-local config.toml the properties that were set in the upper-level config.toml.
In your case your upper-level config.toml specifies replace-with, so you have to overwrite that. But you can't overwrite it with crates-io, which is the registry you want to use, because that is exactly the registry with the replace-with key.
So until the above issue gets acted upon, we have to, essentially, use a mirror, both in the config and as an actual registry to download from:
[net]
offline = false
[source]
[source.crates-io]
replace-with = "crates-io-mirror"
[source.crates-io-mirror]
registry = "https://gitlab.com/integer32llc/crates.io-index"
As we both tested, it seems it's not possible to reuse the normal crates.io registry url because that is already defined and will fail with:
error: source crates-io-mirror defines source registry https://github.com/rust-lang/crates.io-index, but that source is already defined by crates-io note: Sources are not allowed to be defined multiple times.
So instead the URL above is an actual mirror server of crates.io. Then you can run cargo build successfully in the local project.
The recently released Cargo 1.56 adds a feature that should let one do what my question asks for: patch tables can now be specified in a project-specific .cargo/config.toml, which means that [patch] stanzas can now be introduced outside of Cargo.toml. That should do the trick! I haven't yet verified this, as I am stuck with an older Cargo for a little while still.

How to find which Yocto Project recipe populates a particular file on an image root filesystem

I work with the Yocto Project quite a bit and a common challenge is determining why (or from what recipe) a file has been included on the rootfs. This is something that can hopefully be derived from the build system's environment, log & meta data. Ideally, a set of commands would allow linking a file back to a source (ie. recipe).
My usual strategy is to perform searches on the meta data (e.g. grep -R filename ../layers/*) and searches on the internet of said filenames to find clues of possible responsible recipes. However, this is not always very effective. In many cases, filenames are not explicitly stated within a recipe. Additionally, there are many cases where a filename is provided by multiple recipes which leads to additional work to find which recipe ultimately supplied it. There are of course many other clues available to find the answer. Regardless, this investigation is often quite laborious when it seems the build system should have enough information to make resolving the answer simple.
This is exact use case for oe-pkgdata-util script and its subcommand find-path. That script is part of openembedded-core.
See this example (executed in OE build environment, i.e. bitbake works):
tom#pc:~/oe/build> oe-pkgdata-util find-path /lib/ld-2.24.so
glibc: /lib/ld-2.24.so
You can clearly see that this library belongs to glibc recipe.
oe-pkgdata-util has more useful subcommands to see information about packages and recipes, it worth to check the --help.
If you prefer a graphical presentation, the Toaster web UI will also show you this, plus dependency information.
The candidate files deployed for each recipe are placed in each $WORKDIR/image
So you can cd to
$ cd ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}
and perform a
$ find . -path '*/image/*/fileYouAreLookingFor'
from the result you should be able to infer the ${PN} of the recipe which deploys such file.
For example:
$ find . -path '*/image/*/mc'
./bash-completion/2.4-r0/image/usr/share/bash-completion/completions/mc
./mc/4.8.18-r0/image/usr/share/mc
./mc/4.8.18-r0/image/usr/bin/mc
./mc/4.8.18-r0/image/usr/libexec/mc
./mc/4.8.18-r0/image/etc/mc

Scons command/explicit dependency

I have a code snippet similar to this:
# Compile protobuf headers
env.Protoc(...)
# Move headers to 'include' (compiled via protobuf)
env.Command([include headers...], [headers...], move_func)
# Compile program (depends on 'include' files)
out2 = SConscript('src/SConscript')
Depends(out2, [include headers...])
Basically, I have Protoc() compiling protobuf files, then the headers are moved to the 'include' directory by env.Command() and finally the program is compiled through a SConscript file in the 'src'.
Since these are header files that are being moved (that the src compilation depends on), they are not explicitly defined as a dependency by scons (as far as I understand). Thus, the compilation runs, but the header files haven't been moved so it fails. I have tried exposing the dependency via Depends() and Requires() without success.
I understand that in the usual case, scons should "figure-out" dependencies, but I don't know how it could do that here.
Thanks!
You seem to be thinking in "make" ways about your build process, which is the wrong approach when using SCons. You can't order single build steps by putting them in different SConscripts, and then including those in a special order. You have to define proper dependencies between your actual sources (C/CPP files for example) and a target like a program or PDF file. Then SCons is able to figure out the correct build order, and will traverse through the folder structure of your project automatically. If required, it will enter subfolders more than once when the dependency graph (DAG) dictates this. Defining this kind of dependencies between inputs and outputs is usually done, using a Builder...and in your case the Install() builder would be a good fit. Please also regard the hints for #2 in the list of "most frequently-asked FAQs" ( https://bitbucket.org/scons/scons/wiki/FrequentlyAskedQuestions).
Further, I can only recommend to read a little more in the UserGuide ( http://www.scons.org/doc/production/HTML/scons-user.html ) to get a better feeling for how to do things in a more "SConsy" way. If you get stuck, feel free to ask further questions on our mailing list at scons-users#scons.org (see http://www.scons.org/lists.php ).
Finally, if you have a lot of steps that you want to execute in serial, and that don't require any special input/output files, SCons is probably not the right tool for your current task. It's designed as a file-oriented build system with automatic parallelization in mind, a simple (Python?) script might be better at the mere serial stuff...

Node.js/npm - dynamic service discovery in packages

I was wondering whether Node.js/npm include any kind of exension mechanism comparable to Python setuptools' "entry points".
So, in short:
is there any way I can do dynamic discovery of services provided by other packages using npm?
if not, what would be the best way to implement something similar? Specifying the extension name in the main module's configuration file seems to be the logical solution, but I wonder whether something "automatic" can be done.
I'm not aware of any builtin mechanism to do this.
One viable way of doing it yourself:
I made a small tool (Jumpstart) to quickly create project scaffolding from templates with placeholders, and I used a kind of plugin mechanism for that. It basically comes down to that the Jumpstart script searches for modules named jumpstart-* "adjacent" to where the module itself is installed. So it would work for both local and global installations. If installed locally, it would search the other local modules (on the same level) and if global, it searches the other global modules.
Note that here, "search" comes down to a simple fs.exists check to see if there's a Jumpstart template module with a particular name installed. However, nothing would stand in the way to actually get a full list of all installed packages matching the jumpstart-* pattern, and loading all at once. I could also search up the entire directory tree for node_modules directories and do the same. There's no point in doing this for this particular program, however.
See https://npmjs.org/package/jumpstart for docs.
The only limitation to this technique is that all modules must be named in a consistent fashion. Start with some string, end with some string, something like that. Any rogue packages polluting the namespace could be detected by doing further checks on a package contents: What files does it contain? What kind of object does its main module export? etc.
Brunch also uses a plugin mechanism. This one actually deals with file extensions, so is more relevant: https://github.com/brunch/brunch/wiki/Plugins . See for example source of the CoffeeScript plugin https://github.com/brunch/coffee-script-brunch/blob/master/src/index.coffee .

On GNU/Linux systems, Where should I load application data from?

In this instance I'm using c with autoconf, but the question applies elsewhere.
I have a glade xml file that is needed at runtime, and I have to tell the application where it is. I'm using autoconf to define a variable in my code that points to the "specified prefix directory"/app-name/glade. But that only begins to work once the application is installed. What if I want to run the program before that point? Is there a standard way to determine what paths should be checked for application data?
Thanks
Thanks for the responses. To clarify, I don't need to know where the app data is installed (eg by searching in /usr,usr/local,etc etc), the configure script does that. The problem was more determining whether the app has been installed yet. I guess I'll just check in install location first, and if not then in "./src/foo.glade".
I dont think there's any standard way on how to locate such data.
I'd personally do it in a way that i'd have a list of paths and i'd locate if i can find the file from anyone of those and the list should containt the DATADIR+APPNAME defined from autoconf and CURRENTDIRECTORY+POSSIBLE_PREFIX where prefix might be some folder from your build root.
But in any case, dont forget to use those defines from autoconf for your data files, those make your software easier to package (like deb/rpm)
There is no prescription how this should be done in general, but Debian packagers usually installs the application data somewhere in /usr/share, /usr/lib, et cetera. They may also patch the software to make it read from appropriate locations. You can see the Debian policy for more information.
I can however say a few words how I do it. First, I don't expect to find the file in a single directory; I first create a list of directories that I iterate through in my wrapper around fopen(). This is the order in which I believe the file reading should be done:
current directory (obviously)
~/.program-name
$(datadir)/program-name
$(datadir) is a variable you can use in Makefile.am. Example:
AM_CPPFLAGS = $(ASSERT_FLAGS) $(DEBUG_FLAGS) $(SDLGFX_FLAGS) $(OPENGL_FLAGS) -DDESTDIRS=\"$(prefix):$(datadir)/:$(datadir)/program-name/\"
This of course depends on your output from configure and how your configure.ac looks like.
So, just make a wrapper that will iterate through the locations and get the data from those dirs. Something like a PATH variable, except you implement the iteration.
After writing this post, I noticed I need to clean up our implementation in this project, but it can serve as a nice start. Take a look at our Makefile.am for using $(datadir) and our util.cpp and util.h for a simple wrapper (yatc_fopen()). We also have yatc_find_file() in case some third-party library is doing the fopen()ing, such as SDL_image or libxml2.
If the program is installed globally:
/usr/share/app-name/glade.xml
If you want the program to work without being installed (i.e. just extract a tarball), put it in the program's directory.
I don't think there is a standard way of placing files. I build it into the program, and I don't limit it to one location.
It depends on how much customising of the config file is going to be required.
I start by constructing a list of default directories and work through them until I find an instance of glade.xml and stop looking, or not find it and exit with an error. Good candidates for the default list are /etc, /usr/share/app-name, /usr/local/etc.
If the file is designed to be customizable, before I look through the default directories, I have a list of user files and paths and work through them. If it doesn't find one of the user versions, then I look in the list of default directories. Good candidates for the user config files are ~/.glade.xml or ~/.app-name/glade.xml or ~/.app-name/.glade.xml.

Resources