In Test Dir
Test (Dir)
-> 20171101 (Dir)
-> 20171102 (Dir)
-> images (Dir)
-> test.apple.1.txt
-> 20171103 (Dir)
-> images (Dir)
-> test.apple.2.txt
I want to make symbolic link below :
Test2 (Test's symbolic link)
-> 20171101 (Dir)
-> 20171102 (Dir)
-> images (Dir)
-> test.banana.1.txt
-> 20171103 (Dir)
-> images (Dir)
-> test.banana.2.txt
I want to change file name pattern apple to banana while making symbolic link.
In result,
In original dir, test.apple.1.txt remains
In symbolic link, test.banana.1.txt remains
I find show pattern change shell script below:
for i in `find .`; do echo `echo $i | sed -e 's/apple/banana/'`; done;
But making symbolic link is difficult..
Is it possible?
Related
For some reason I cannot make cd command work in shake/command Haskell library. It thinks directory I called with cd does not exist even though it is present in the filesystem.
Here is an excerpt of my code:
dataDir = "data"
data Settings = Settings {
url :: String
} deriving (Eq, Show, Generic, JSON.ToJSON, JSON.FromJSON)
settings :: String -> Handler Settings
settings subfolder = let
gitPath = dataDir ++ "/" ++ subfolder ++ "/git/.git"
in do
pathExists <- liftIO $ doesPathExist gitPath
-- Stdout pwdOut <- liftIO $ cmd ("pwd" :: String)
-- liftIO $ putStrLn $ pwdOut
if not pathExists
then do
liftIO $ (cmd_ ("mkdir -p" :: String) [gitPath] :: IO ())
liftIO $ (cmd_ ("cd" :: String) [gitPath] :: IO ())
liftIO $ (cmd_ ("git init" :: String) :: IO ())
return $ Settings { url = ""}
else do
liftIO $ (cmd_ (Cwd ".") ("cd" :: String) [gitPath] :: IO ())
Stdout out <- liftIO $ (cmd ("git config --get remote.origin.url" :: String))
return $ Settings {url = out}
It fails with an error cd: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory) in both cases: if dir exists and when mkdir command is executed.
Cannot wrap my head around it. But before I submit a bug to the shake's github page, I want to make sure with you I am not doing anything stupid that might cause this kind of behavior.
Thanks in advance for help.
As described in the other answer, cd is not an executable, so if you wanted to run it, you would have to pass Shell to cmd.
However, it is almost certainly the case that you don't want to call cd in a command, as it does not change the directory for any subsequent command. Each cmd is a separate process, with a separate environment, so the subsequent command will be in a fresh environment, and the same working directory as before the cd. The solution is to pass (Cwd gitPath) to each command you want to operate with the given directory.
Shake's Haddock page describes cmd_, and links to its source. There we can see that cmd_ eventually calls commandExplicitIO, which constructs a ProcessOpts with RawCommand and passes it to process. process then takes that ProcessOpts, pattern-matches it as a RawCommand (via cmdSpec), and calls proc. We have now entered the well-documented zone: you must give proc an executable, and cd is not an executable. (Why? Since processes cannot change the working directory of their parent, cd must be a shell builtin.)
I have a directory traversal function in Haskell, but I want it to ignore symlinks. I figured out how to filter out the files alone, albeit with a slightly inelegant secondary filterM. But after some diagnosis I realize that I'm failing to filter symlinked directories.
I'd like to be able to write something like this:
-- Lazily return (normal) files from rootdir
getAllFiles :: FilePath -> IO [FilePath]
getAllFiles root = do
nodes <- pathWalkLazy root
-- get file paths from each node
let files = [dir </> file | (dir, _, files) <- nodes,
file <- files,
not . pathIsSymbolicLink dir]
normalFiles <- filterM (liftM not . pathIsSymbolicLink) files
return normalFiles
However, all the variations I have tried get some version of the "Couldn't match expected type ‘Bool’ with actual type ‘IO Bool’" message (without the filter clause in the comprehension it works, but fails to filter those linked dirs).
Various hints at ways I might completely restructure the function are in partial form at online resources, but I'm pretty sure that every such variation will run into some similar issue. The list comprehension would certainly be the most straightforward way... if I could just somehow exclude those dirs that are links.
Followup: Unfortunately, the solution kindly provided by ChrisB behaves (almost?!) identically to my existing version. I defined three functions, and run them within a test program:
-- XXX: debugging
files <- getAllFilesRaw rootdir
putStrLn ("getAllFilesRaw: " ++ show (length files))
files' <- getAllFilesNoSymFiles rootdir
putStrLn ("getAllFilesNoSymFiles: " ++ show (length files'))
files'' <- getAllFilesNoSymDirs rootdir
putStrLn ("getAllFilesNoSymDirs: " ++ show (length files''))
The first is my version with the normalFiles filter removed. The second is my original version (minus the type error in the listcomp). The final one is ChrisB's suggestion.
Running that, then also the system find utility:
% find $CONDA_PREFIX -type f | wc -l
449667
% find -L $CONDA_PREFIX -type f | wc -l
501153
% haskell/find-dups $CONDA_PREFIX
getAllFilesRaw : 501153
getAllFilesNoSymFiles: 464553
getAllFilesNoSymDirs: 464420
Moreover, this question came up because—for my own self-education—I've implemented the same application in a bunch of languages: Python; Golang; Rust; Julia; TypeScript; Bash, except the glitch, Haskell; others are planned. The programs actually do something more with the files, but that's not the point of this question.
The point of this is that ALL other languages report the same number as the system find tool. Moreover, the specific issue is things like this:
% ls -l /home/dmertz/miniconda3/pkgs/ncurses-6.2-he6710b0_1/lib/terminfo
lrwxrwxrwx 1 dmertz dmertz 17 Apr 29 2020 /home/dmertz/miniconda3/pkgs/ncurses-6.2-he6710b0_1/lib/terminfo -> ../share/terminfo
There are about 16k examples here (on my system currently), but looking at some in the other version of the tool, I see specifically that all the other languages are excluding the contents of that symlink directory.
EDIT:
Instead of just fixing a Bool / IO Bool issue we now want to mach find's behavior.
After looking at the documentation,
this seems to be quite hard to implement reasonably performantly
with the PathWalk library, so i just handrolled it.
(Using do-notation, as requested in the comments.)
In my quick and dirty tests the results match those of find:
import System.FilePath
import System.Directory
getAllFiles' :: FilePath -> IO [FilePath]
getAllFiles' path = do
isSymlink <- pathIsSymbolicLink path
if isSymlink
-- if this is a symlink, return the empty list.
-- even if this was the original root. (matches find's behavior)
then return []
else do
isFile <- doesFileExist path
if isFile
then return [path] -- if this is a file, return it
else do
-- if it's not a file, we assume it to be a directory
dirContents <- listDirectory path
-- run this function recursively on all the children
-- and accumulate the results
fmap concat $ mapM (getAllFiles' . (path </>)) dirContents
Original Answer solving the IO Bool / Bool issue
getAllFiles :: FilePath -> IO [FilePath]
getAllFiles root = pathWalkLazy root
-- remove dirs that are symlinks
>>= filterM (\(dir, _, _) -> fmap not $ pathIsSymbolicLink dir)
-- flatten to list of files
>>= return . concat . map (\(dir, _, files) -> map (\f -> dir </> f) files)
-- remove files that are symlinks
>>= filterM (fmap not . pathIsSymbolicLink)
I'm new to IIS and i've a PHP project with all files in a public directory, in such a structure:
Root/public/index.phplogin.phpsub1/index.phpedit.phpsub2/index.phpsrc/js/script.jscss/theme.css
I need some rewrite rules for:
hide extension for just PHP files:example.com -> example.com/public/index.phpexample.com/login -> example.com/public/login.phpexample.com/sub1/edit -> example.com/public/sub1/edit.phpexample.com/src/css/theme.css -> example.com/src/css/theme.cssdefault index.php for subdirectories, if page is not specified:example.com/sub1/ -> example.com/public/sub1/index.phpexample.com/sub1/edit -> example.com/public/sub1/edit.phpoptional trailing slash for both pages and subdirectories:example.com/login -> example.com/public/login.phpexample.com/login/ -> example.com/public/login.phpexample.com/sub1 -> example.com/public/sub1/index.phpexample.com/sub1/ -> example.com/public/sub1/index.php example.com/sub1/edit -> example.com/public/sub1/edit.phpexample.com/sub1/edit/ -> example.com/public/sub1/edit.php
I'm playing with Turtle and I'm faced with the following problem.
I want to do something like (in shell)
ls | grep 'foo'
My attempt using Turtle is
grep (prefix "foo") (ls ".") & view
But I got the following message
Couldn't match type ‘Turtle.FilePath’ with ‘Text’
Expected type: Shell Text
Actual type: Shell Turtle.FilePath
In the second argument of ‘grep’, namely ‘(ls ".")’
In the first argument of ‘(&)’, namely
‘grep (prefix "foo") (ls ".")’
I understand ls returns FilePath whereas grep works on Text, so what can I do ?
Update
There are obviously solutions which involves converting back and forth from FilePath to Text. That's beyond the simplicity I would expect shell-like program.
Someone mentioned the find function, which somehow could solves the problem.
However find is the equivalent to the find shell function and I was trying just to do ls | grep "foo". I'm not trying to solve a real life problem (if I were, I would switch to bash instead) but trying to combine simple bricks as I would do in bash. Unfortunately, it doesn't seem that bricks in Turtle are that easy to combine :-(.
Instead of grep, we can use match, in combination with the MonadPlus instance of Shell for filtering:
filterByPattern :: MonadPlus m => Pattern x -> FilePath -> m FilePath
filterByPattern somepattern somepath =
case match somepattern (either id id (toText somepath)) of
[] -> mzero
otherwise -> return somepath
greppedls :: FilePath -> Pattern x -> Shell FilePath
greppedls somepath somepattern =
ls somepath >>= filterByPattern somepattern
Edit: Instead of using the unnecesarily general MonadPlus, here's an implementation that filters using the turtle-specific combinator select:
filterByPattern :: Pattern x -> FilePath -> Shell FilePath
filterByPattern somepattern somepath =
case match somepattern (either id id (toText somepath)) of
[] -> select [] -- no matches, so filter this path
otherwise -> select [somepath] -- let this path pass
A value foo :: Shell a is a bit like a "list of as". If we have a function genlist :: a -> Shell b that for each a generates a (perhaps empty!) list of bs, we can obtain a list of bs using the (>>=) operator: foo >>= genlist.
Edit#2: The standard turtle function find already filters files using a pattern. It is recursive and searches in subdirectories.
To convert from FilePath into Text you use:
fp :: Format r (FilePath -> r)
Here is an example:
format fp ("usr" </> "lib")
There is a couple of issues about this so Gabriel has decided to update the tutorial a few days ago:
https://github.com/Gabriel439/Haskell-Turtle-Library/commit/a2fff2acf912cc7adb2e02671340822feb0e9172
To answer your (updated) question, the best I can come up is:
format fp <$> ls "." & grep (has "foo") & view
& is playing the role of |.
As a personal note, it is of course not as short as ls | grep 'foo' but still quite elegant given that Haskell is a typed language.
The literal answer is this one-liner:
example =
view (ls "." & fmap (format fp) & grep (prefix "foo") & fmap toText)
The idiomatic answer is to use the find utility
Try to use repr
repr :: Show a => a -> Text
I'm currently testing porting our build system from make to shake and have hit a roadblock:
Given the following project structure:
static/a.js
static/b.coffee
build/a.js
build/b.js
That is, various input extensions map to identical output extensions, so a straightforward "build//*.js" %> rule isn't going to work.
I wanted to avoid using priority if possible, and writing an ad-hoc build rule that checks for the existence of either possible input felt clunky (especially since this situation occurs with other filetypes as well), so I wrote the following:
data StaticFileMapping a = StaticFileMapping String String (FilePath -> FilePath -> Action a)
staticInputs :: FilePath -> StaticFileMapping a -> Action [FilePath]
staticInputs dir (StaticFileMapping iExt _ _) = (findFiles (dir </> "static") [iExt])
staticInputToOutput :: StaticFileMapping a -> FilePath -> FilePath
staticInputToOutput (StaticFileMapping _ oExt _) = (remapDir ["build"]) . (-<.> oExt)
staticTargets :: FilePath -> StaticFileMapping a -> Action [FilePath]
staticTargets dir sfm = (map $ staticInputToOutput sfm) <$> staticInputs dir sfm
rules :: FilePath -> StaticFileMapping a -> Rules ()
rules dir sfm#(StaticFileMapping _ _ process) = join $ mconcat . (map buildInputRule) <$> staticInputs dir sfm
where buildInputRule :: FilePath -> Rules ()
buildInputRule input = (staticInputToOutput sfm input) %> (process input)
That way I can define a mapping for each input type (.coffee -> .js, .svg -> .png) and so on, with only a tiny amount of code implementing the transformation for each. And it almost works.
But it seems impossible to go from (Action a) to Rules _ without throwing the value inside the Action away first, as far as I can tell.
Is there a function with type (Action a) -> (a -> Rules ()) -> Rules () or (Action a) -> (Rules a)? Can I implement either one myself, or do I need to modify the library's code?
Or is this entire approach hare-brained and I should take some other route?
First off, using priority would not work, as that picks a rule statically then runs it - it doesn't backtrack. It's also important that Shake doesn't run any Action operations to produce Rules (as per the two functions you propose) since the Action might call need on a Rule that it itself defines, or is defined by another action rule, thus making the ordering of those Action calls visible. You could add IO (Rules ()) -> Rules (), which might be enough for what you are thinking of (directory listing), but it isn't currently exposed (I have an internal function that does exactly that).
To give a few example approaches it's useful to define plausible commands to convert .js/.coffee files:
cmdCoffee :: FilePath -> FilePath -> Action ()
cmdCoffee src out = do
need [src]
cmd "coffee-script-convertor" [src] [out]
cmdJavascript :: FilePath -> FilePath -> Action ()
cmdJavascript = copyFile'
Approach 1: Use doesFileExist
This would be my standard approach, writing something like:
"build/*.js" %> \out -> do
let srcJs = "static" </> dropDirectory1 out
let srcCf = srcJs -<.> "coffee"
b <- doesFileExist srcCf
if b then cmdCoffee srcCf out else cmdJavascript srcJs out
This accurately captures the dependency that if the user adds a .coffee file in the directory then the rule should be rerun. You could imagine sugaring up the doesFileExist if this is a common pattern for you. You could even drive it from you list of StaticFileMapping structures (do a group on the oExt field to add one rule per oExt than calls doesFileExists on each iExt in turn). An advantage of this approach is that if you do shake build/out.js it doesn't need to do a directory listing, although likely that cost is negligible.
Approach 2: List the files before calling shake
Instead of writing main = shakeArgs ... do:
import System.Directory.Extra(listFilesRecursive) -- from the "extra" package
main = do
files <- listFilesRecursive "static"
shakeArgs shakeOptions $ do
forM_ files $ \src -> case takeExtension src of
".js" -> do
let out = "build" </> takeDirectory1 src
want [out]
out %> \_ -> cmdJavascript src out
-- rules for all other types you care about
_ -> return ()
Here you operate in IO to get the list of files, then can add rules by referring to that previously captured value. Adding rulesIO :: IO (Rules ()) -> Rules () would allow you to list the files inside shakeArgs.
Approach 3: List the files inside the rules
You can define a mapping between file names and outputs, driven from the directory listing:
buildJs :: Action (Map FilePath (Action ()))
buildJs = do
js <- getDirectoryFiles "static" ["*.js"]
cf <- getDirectoryFiles "static" ["*.coffee"]
return $ Map.fromList $
[("build" </> j, cmdJavascript ("static" </> j) ("build" </> j)) | j <- js] ++
[("build" </> c, cmdCoffee ("static" </> c) ("")) | c <- cf]
Then lift that into a set of rules:
action $ do
mpJs <- buildJs
need $ Map.keys mpJs
"//*.js" %> \out -> do
mpJs <- buildJs
mpJs Map.! out
However, that recomputes the directory listing for every file we build, so we should cache it and make sure it's only computed once:
mpJs <- newCache $ \() -> buildJs
action $ do
mpJs <- mpJs ()
need $ Map.keys mpJs
"//*.js" %> \out -> do
mpJs <- mpJs ()
mpJs Map.! out
This solution is probably closest to your original approach, but I find it the most complex.