At the moment I'm using some magic to get the current git revision into my scons builds.. I just grab the version a stick it into CPPDEFINES.
It works quite nicely ... until the version changes and scons wants to rebuild everything, rather than just the files that have changed - becasue the define that all files use has changed.
Ideally I'd generate a file using a custom builder called git_version.cpp and
just have a function in there that returns the right tag. That way only that one file would be rebuilt.
Now I'm sure I've seen a tutorial showing exactly how to do this .. but I can't seem to track it down. And I find the custom builder stuff a little odd in scons...
So any pointers would be appreciated...
Anyway just for reference this is what I'm currently doing:
# Lets get the version from git
# first get the base version
git_sha = subprocess.Popen(["git","rev-parse","--short=10","HEAD"], stdout=subprocess.PIPE ).communicate()[0].strip()
p1 = subprocess.Popen(["git", "status"], stdout=subprocess.PIPE )
p2 = subprocess.Popen(["grep", "Changed but not updated\\|Changes to be committed"], stdin=p1.stdout,stdout=subprocess.PIPE)
result = p2.communicate()[0].strip()
if result!="":
git_sha += "[MOD]"
print "Building version %s"%git_sha
env = Environment()
env.Append( CPPDEFINES={'GITSHAMOD':'"\\"%s\\""'%git_sha} )
You don't need a custom Builder since this is just one file. You can use a function (attached to the target version file as an Action) to generate your version file. In the example code below, I've already computed the version and put it into an environment variable. You could do the same, or you could put your code that makes git calls in the version_action function.
version_build_template="""/*
* This file is automatically generated by the build process
* DO NOT EDIT!
*/
const char VERSION_STRING[] = "%s";
const char* getVersionString() { return VERSION_STRING; }
"""
def version_action(target, source, env):
"""
Generate the version file with the current version in it
"""
contents = version_build_template % (env['VERSION'].toString())
fd = open(target[0].path, 'w')
fd.write(contents)
fd.close()
return 0
build_version = env.Command('version.build.cpp', [], Action(version_action))
env.AlwaysBuild(build_version)
Related
The example is from Chapter 8 of Grokking Deep Reinforcement Learning written by Miguel Morales.
Please click here
The wrappers.Monitor is deprecated after the book is published. The code in question is as below:
env = wrappers.Monitor(
env, mdir, force=True,
mode=monitor_mode,
video_callable=lambda e_idx: record) if monitor_mode else env
I searched the internet and tried 2 methods but failed.
1- gnwrapper.Monitor
I pip install gym-notebook-wrapper and import gnwrapper, then rewrite the code
env = gnwrapper.Monitor(gym.make(env_name),directory="./")
A [WinError 2] The system cannot find the file specified error message is returned.
2- gym.wrappers.RecordVideo
I from gym.wrappers import RecordVideo then rewrite the code
env = RecordVideo(gym.make(env_name), "./")
A AttributeError: 'CartPoleEnv' object has no attribute 'videos' error message is returned.
Is there any way to solve the problem?
After some experiments, the best way to work around the problem is to downgrade the gym version to 0.20.0, in order to preserve the wrappers.Monitor function.
Moreover, the subprocess.Popen and subprocess.check_output does not work as the original author suggested, so I use moviepy (please pip install moviepy if you do not have this libarary and from moviepy.editor import VideoFileClip at the very beginning) to change MP4 to GIF files. The code in get_gif_html is amended as follows.
gif_path = basename + '.gif'
if not os.path.exists(gif_path):
videoClip = VideoFileClip(video_path)
videoClip.write_gif(gif_path, logger=None)
gif = io.open(gif_path, 'r+b').read()
The program now completely works.
Edit (24 Jul 2022):
The following solution is for gym version 0.25.0, as I hope the program works for later versions. Windows 10 and Juptyer Notebook is used for the demonstration.
(1) Maintain the moviepy improvement stated above
(2) import from gym.wrappers.record_video import RecordVideo and substitute
env = wrappers.Monitor(
env, mdir, force=True,
mode=monitor_mode,
video_callable=lambda e_idx: record) if monitor_mode else
with this
if monitor_mode:
env = RecordVideo(env, mdir, episode_trigger=lambda e_idx:record)
env.reset()
env.start_video_recorder()
else:
env = env
RecordVideo function takes (env, video_folder, episode_trigger) arguments, and video_callable and episode_trigger are the same thing.
(3) Loss of env.videos in get_gif_html function in new versions.
In the old setting, after the video is recorded, the MP4 and meta.json files are stored in the following address with a random file name
C:\Users\xxx\AppData\Local\Temp
https://i.stack.imgur.com/Za3Sd.jpg
This env.videos function returns a list of tuples, each tuples consisted of address path of MP4 and meta.json.
I tried to recreate the env.videos by
(a) declare the mdir as global variable
def get_make_env_fn(**kargs):
def make_env_fn(env_name, seed=None, render=None, record=False,
unwrapped=False, monitor_mode=None,
inner_wrappers=None, outer_wrappers=None):
global mdir
mdir = tempfile.mkdtemp()
(b) in the demo_progression and demo_last function, grouping all MP4 and meta_json files, zipping them and use list comprehension to make a new env_videos variable (as opposed to env.videos)
env.close()
mp4_files = ([(mdir + '\\' + file) for file in os.listdir(mdir) if file.endswith('.mp4')])
meta_json_files = ([(mdir + '\\' + file) for file in os.listdir(mdir) if file.endswith('.meta.json')])
env_videos = [(mp4, meta_json) for mp4, meta_json in zip(mp4_files, meta_json_files)]
data = get_gif_html(env_videos=env_videos,
These are all the changes.
I'm trying to create a Substfile rule that will expand a key to the transformed contents of another file. I'm not clear on the setup here to ensure the source file is registered as a dependency.
Logically I want something like:
out = env.Substfile( 'file.in', SUBST_DICT = {
'%SOME_CONTENT%': transform( readfile('depends.txt') ),
}
I'm using a combination of Action and Command to do what I want. I ended up not using Substfile, though it could probably be chained to the command.
This RawStringIt action loads a text file and emits a C++ encoded raw string for the content.
def RawStringIt(varName):
def Impl(target, source, env):
content = source[0].get_text_contents()
with open(target[0].get_path(), 'w') as target_file:
target_file.write( "std::string {} = R\"~~~~({})~~~~\";".format(varName,content))
return 0
return Action(Impl, "creating C++ Raw String $TARGET from $SOURCE" )
base_leaf = env.Command( 'include/runner/base.leaf.hpp', '../share/base.leaf', RawStringIt("dataBaseLeaf") )
Seems -u doesn't work on for me ( I am using scons-2.3.6).
To simplify the context, you can imagine my project structure like,
+root
+project
- bar.vcxproj (generated vs project)
-SConstruct
-bar.c
Inside SConstruct, I have put code like:
env_base = Environment()
...
env_base.StaticLibrary(target = 'bar', source = ['bar.c'])
...
If I execute command "scons" in root folder, everything works perfectly.
But If I execute command "scons -u" in project folder, scons can find my SConstruct up in root folder, but no file get compiled.
BTW : The reason for me to execute "scons -u" in project folder is because I want to put the generated vsproj in projet folder and use BuildCommandLine to compile the project.
I guess I didn't use "-u" correctly, what will be the elegant solution for my situation?
1st edit:
As bdbaddog asked, I have put the SConstruct here:
def BuildConfig(env, config):
env.Append(CCFLAGS = '/W 4')
env.Append(CCFLAGS = '/WX')
if config == "debug":
env.Append(CCFLAGS = '/DEBUG')
#env.Append(CCFLAGS = '-Zi /Fd${TARGET}.pdb')
env.Append(CCFLAGS = '/Z7')
elif config == "release":
pass
env_base = Environment()
lib = env_base.StaticLibrary(target = 'bar', source = ['bar.c'])
opts=Variables()
opts.Add('target', 'Compile Target (debug/release).', "debug")
# there is more in my project....
opts.Update(env_base) # update environment
# here I want to use my own command to build the project, so it can support different build option that is defined by me.
env_base['MSVSBUILDCOM'] = "scons -u target=$(Configuration)"
target = env_base["target"]
BuildConfig(env_base, env_base['target'])
env_base.MSVSProject(target = "project\\bar" + env_base['MSVSPROJECTSUFFIX'],
srcs = ["..\\bar.c"],
incs = [],
localincs = "",
resources = "",
misc = "",
buildtarget = lib,
variant = ['debug'],
auto_build_solution=0)
SCons only builds files under the current directory by default.
If you you wanted to only build files in a certain directory (for which there are rules that build the targets there), you can invoke SCons as follows:
scons the_target_directory_I_want_to_build
Though this may cause sources for targets in that directory to also be built.
I'm looking to monkey-patch require() to replace its file loading with my own function. I imagine that internally require(module_id) does something like:
Convert module_id into a file path
Load the file path as a string
Compile the string into a module object and set up the various globals correctly
I'm looking to replace step 2 without reimplementing steps 1 + 3. Looking at the public API, there's require() which does 1 - 3, and require.resolve() which does 1. Is there a way to isolate step 2 from step 3?
I've looked at the source of require mocking tools such as mockery -- all they seem to be doing is replacing require() with a function that intercepts certain calls and returns a user-supplied object, and passes on other calls to the native require() function.
For context, I'm trying to write a function require_at_commit(module_id, git_commit_id), which loads a module and any of that module's requires as they were at the given commit.
I want this function because I want to be able to write certain functions that a) rely on various parts of my codebase, and b) are guaranteed to not change as I evolve my codebase. I want to "freeze" my code at various points in time, so thought this might be an easy way of avoiding having to package 20 copies of my codebase (an alternative would be to have "my_code_v1": "git://..." in my package.json, but I feel like that would be bloated and slow with 20 versions).
Update:
So the source code for module loading is here: https://github.com/joyent/node/blob/master/lib/module.js. Specifically, to do something like this you would need to reimplement Module._load, which is pretty straightforward. However, there's a bigger obstacle, which is that step 1, converting module_id into a file path, is actually harder than I thought, because resolveFilename needs to be able to call fs.exists() to know where to terminate its search... so I can't just substitute out individual files, I have to substitute entire directories, which means that it's probably easier just to export the entire git revision to a directory and point require() at that directory, as opposed to overriding require().
Update 2:
Ended up using a different approach altogether... see answer I added below
You can use the require.extensions mechanism. This is how the coffee-script coffee command can load .coffee files without ever writing .js files to disk.
Here's how it works:
https://github.com/jashkenas/coffee-script/blob/1.6.2/lib/coffee-script/coffee-script.js#L20
loadFile = function(module, filename) {
var raw, stripped;
raw = fs.readFileSync(filename, 'utf8');
stripped = raw.charCodeAt(0) === 0xFEFF ? raw.substring(1) : raw;
return module._compile(compile(stripped, {
filename: filename,
literate: helpers.isLiterate(filename)
}), filename);
};
if (require.extensions) {
_ref = ['.coffee', '.litcoffee', '.md', '.coffee.md'];
for (_i = 0, _len = _ref.length; _i < _len; _i++) {
ext = _ref[_i];
require.extensions[ext] = loadFile;
}
}
Basically, assuming your modules have a set of well-known extensions, you should be able to use this pattern of a function that takes the module and filename, does whatever loading/transforming you need, and then returns an object that is the module.
This may or may not be sufficient to do what you are asking, but honestly from your question it sounds like you are off in the weeds somewhere far from the rest of the programming world (don't take that harshly, it's just my initial reaction).
So rather than mess with the node require() module, what I ended up doing is archiving the given commit I need to a folder. My code looks something like this:
# commit_id is the commit we want
# (note that if we don't need the whole repository,
# we can pass "commit_id path_to_folder_we_need")
#
# path is the path to the file you want to require starting from the repository root
# (ie 'lib/module.coffee')
#
# cb is called with (err, loaded_module)
#
require_at_commit = (commit_id, path, cb) ->
dir = 'old_versions' #make sure this is in .gitignore!
dir += '/' + commit_id
do_require = -> cb null, require dir + '/' + path
if not fs.existsSync(dir)
fs.mkdirSync(dir)
cmd = 'git archive ' + commit_id + ' | tar -x -C ' + dir
child_process.exec cmd, (error) ->
if error
cb error
else
do_require()
else
do_require()
I'm using SCons for building a project and need to add a symbolic link to a file it is installing via env.Install. What command(s) will make a link that's the equivalent of running ln -s on the command line?
SCons doesn't have a dedicated symbolic link command, but you can use os.symlink(src, dst) from Python's os module:
import os
env = Environment()
def SymLink(target, source, env):
os.symlink(os.path.abspath(str(source[0])), os.path.abspath(str(target[0])))
env.Command("file.out", "file.in", SymLink)
This may not work correctly on Windows, I've only tried it on Linux.
There seems to be little advancement in the SCons core code for symbolic link support and I wasn't satisfied any one solution I found on the web. Here is a potential builder which incorporates aspects of both Nick's and richq's answers. Additionally, it will catch name changes (due to the emitter method) and is as platform-agnostic as I could get it.
I prefer this builder because it will make links relative to the directory in which they are installed. One could add an option to force the link to be absolute I suppose, but I have not needed or wanted that yet.
Currently, if the OS doesn't support symlinks, I just pass and do nothing, but one could use os.copytree() for example however the dependency becomes messy if the source is a directory so the emitter would need to do something fancy. I'm up for any suggestions here.
One can put the following code into the file site_scons/site_tools/symlink.py (with blank _init_.py files in the appropriate places). Then do this in the SConstruct file:
SConstruct:
env = Environment()
env.Tool('symlink')
env.SymLink('link_name.txt', 'real_file.txt')
symlink.py:
import os
from os import path
from SCons.Node import FS
from SCons.Script import Action, Builder
def generate(env):
'''
SymLink(link_name,source)
env.SymLink(link_name,source)
Makes a symbolic link named "link_name" that points to the
real file or directory "source". The link produced is always
relative.
'''
bldr = Builder(action = Action(symlink_builder,symlink_print),
target_factory = FS.File,
source_factory = FS.Entry,
single_target = True,
single_source = True,
emitter = symlink_emitter)
env.Append(BUILDERS = {'SymLink' : bldr})
def exists(env):
'''
we could test if the OS supports symlinks here, or we could
use copytree as an alternative in the builder.
'''
return True
def symlink_print(target, source, env):
lnk = path.basename(target[0].abspath)
src = path.basename(source[0].abspath)
return 'Link: '+lnk+' points to '+src
def symlink_emitter(target, source, env):
'''
This emitter removes the link if the source file name has changed
since scons does not seem to catch this case.
'''
lnk = target[0].abspath
src = source[0].abspath
lnkdir,lnkname = path.split(lnk)
srcrel = path.relpath(src,lnkdir)
if int(env.get('verbose',0)) > 3:
ldir = path.relpath(lnkdir,env.Dir('#').abspath)
if rellnkdir[:2] == '..':
ldir = path.abspath(ldir)
print ' symbolic link in directory: %s' % ldir
print ' %s -> %s' % (lnkname,srcrel)
try:
if path.exists(lnk):
if os.readlink(lnk) != srcrel:
os.remove(lnk)
except AttributeError:
# no symlink available, so we remove the whole tree? (or pass)
#os.rmtree(lnk)
print 'no os.symlink capability on this system?'
return (target, source)
def symlink_builder(target, source, env):
lnk = target[0].abspath
src = source[0].abspath
lnkdir,lnkname = path.split(lnk)
srcrel = path.relpath(src,lnkdir)
if int(env.get('verbose',0)) > 4:
print 'target:', target
print 'source:', source
print 'lnk:', lnk
print 'src:', src
print 'lnkdir,lnkname:', lnkdir, lnkname
print 'srcrel:', srcrel
if int(env.get('verbose',0)) > 4:
print 'in directory: %s' % path.relpath(lnkdir,env.Dir('#').abspath)
print ' symlink: %s -> %s' % (lnkname,srcrel)
try:
os.symlink(srcrel,lnk)
except AttributeError:
# no symlink available, so we make a (deep) copy? (or pass)
#os.copytree(srcrel,lnk)
print 'no os.symlink capability on this system?'
return None
This creates a builder to perform the job:
mylib = env.SharedLibrary("foobar", SRCS)
builder = Builder(action = "ln -s ${SOURCE.file} ${TARGET.file}", chdir = True)
env.Append(BUILDERS = {"Symlink" : builder})
mylib_link = env.Symlink("_foobar.so", mylib)
env.Default(mylib)
env.Default(mylib_link)
Again, this solution is for Linux.
If you wanted to issue the command directly to the shell and know the OS, subprocess can be used as well.
E.g.: subprocess.call(['ln', '-s', '</src/path>', '</dest/path>'])
In addition to Nicks solution, you can add a directory symlink by using a file as a directory name carrier. It's not the cleanest solution and debugging path names is a pain, but this works well:
def symlink_last(target_source_env):
src = os.path.basename(os.path.dirname(str(source[0])))
link = "deliverables/last"
print "Symlinking "+ src + "as" + link
os.symlink(src, link)
BUILD_TARGETS.append('link')
install_dir = "deliverables/subdir"
carrier_file = "filename"
builder = Builder(action = symlink_last, chdir=False)
env.Append(BUILDERS={ "Symlink" : builder })
env.Alias(target="link", source=env.Symlink(dir="deliverables", source = install_dir + carrier_file)
This will make a link to deliverables/subdir named deliverables/last, provided that a file deliverables/subdir/filename exists.