I am beginning to think my search skills are lacking.
I trying to find any articles on how with Groovy, to open an interactive process, read its output and then write to the process depending on the output text. All I can find is how printing, reading and writing with files. Nothing about how to Write to a interactive process.
The process is asking for a password
Write the password to process
Something like this if possible:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
if (line.contains("enter password")) {
process.out.write("myPassword")
}
}
This here works reading from the process output:
def process = "some-command.sh".execute()
process.in.eachLine { line ->
println line
}
Though it stops when the process is asking for input. It does not print out the line with the question.
Edit: Found out why it did not print the line with the ask password. It was not a new line. The question was a simple print (not println). How do I read when there is not yet a new line?
I have been told expect can be used, but I am looking for a solution which does not require a dependency.
1.bat
#echo off
echo gogogo
set /P V=input me:
echo V=%V%
this script waits for input just after :
gogogo
input me:
this means that eachLine not triggered for input me because no new line after it
however the previous line gogogo could be caught
and following script works for gogogo but does not work for input me
groovy
def process = "1.bat".execute()
process.in.eachLine { line ->
if (line.contains("gogogo")) {
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}
groovy2
probably this could be optimized.. following script works without new line:
def process = "1.bat".execute()
def pout = new ByteArrayOutputStream()
def perr = new ByteArrayOutputStream()
process.consumeProcessOutput(pout, perr) //starts listening threads and returns immediately
while(process.isAlive()){
Thread.sleep(1234)
if(pout.toString("UTF-8").endsWith("input me:")){
process.out.write("myPassword\n".getBytes("UTF-8"))
process.out.flush()
}
}
I have input data of type
abc 12d
uy 76d
ce 12a
with the lines being separated by \n and the values by \t.
The data comes from a shell command:
brlist = 'mycommand'.execute().text
Then I want to get this into a map:
brmap = brlist.split("\n").collectEntries {
tkns = it.tokenize("\t")
[ (tkns[0]): tkns[1] ]
}
I also tried
brmap = brlist.split("\n").collectEntries {
it.tokenize("\t").with { [ (it[0]): it[1] ] }
}
Both ways gave the same result, which is a map with a single entry:
brmap.toString()
# prints "[abc:12d]"
Why does only the first line of the input data end up being in the map?
Your code works, which means the input String brlist isn't what you say it is...
Are you sure that's what you have? Try printing brlist, and then it inside collectEntries
As an aside, this does the same thing as your code:
brlist.split('\n')*.split('\t')*.toList().collectEntries()
Or you could try (incase it's spaces not tabs, this will expect both)
brlist.split('\n')*.split(/\s+/)*.toList().collectEntries()
This code works
// I use 4 spaces as tab.
def text = 'sh abc.sh'.execute().text.replaceAll(" " * 4, "\t")
brmap = text.split("\n").collectEntries {
tkns = it.tokenize("\t")
[(tkns[0]) : tkns[1]]
}
assert[abc:"12d", uy:"76d", ce:"12a"] == brmap
abc.sh
#!/bin/sh
echo "abc 12d"
echo "uy 76d"
echo "ce 12a
Also, I think your groovy code is correct. maybe your mycommand has some problem.
Ok, thanks for the hints, it is a bug in Jenkins: https://issues.jenkins-ci.org/browse/JENKINS-26481.
And it has been mentioned here before: Groovy .each only iterates one time
How can I get all the keys set in my memcached instance(s)?
I tried googling, but didn't find much except that PHP supports a getAllKeys method, which means it is actually possible to do this somehow. How can I get the same within a telnet session?
I have tried out all the retrieval related options mentioned in memcached cheat sheet and Memcached telnet command summary, but none of them work and I am at a loss to find the correct way to do this.
Note: I am currently doing this in development, so it can be assumed that there will be no issues due to new keys being set or other such race conditions happening, and the number of keys will also be limited.
Found a way, thanks to the link here (with the original google group discussion here)
First, Telnet to your server:
telnet 127.0.0.1 11211
Next, list the items to get the slab ids:
stats items
STAT items:3:number 1
STAT items:3:age 498
STAT items:22:number 1
STAT items:22:age 498
END
The first number after ‘items’ is the slab id. Request a cache dump for each slab id, with a limit for the max number of keys to dump:
stats cachedump 3 100
ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s]
END
stats cachedump 22 100
ITEM views.decorators.cache.cache_page..8427e [7736 b; 1256056128 s]
END
memdump
There is a memcdump (sometimes memdump) command for that (part of libmemcached-tools), e.g.:
memcdump --servers=localhost
which will return all the keys.
memcached-tool
In the recent version of memcached there is also memcached-tool command, e.g.
memcached-tool localhost:11211 dump | less
which dumps all keys and values.
See also:
What's the simplest way to get a dump of all memcached keys into a file?
How do I view the data in memcache?
Base on #mu 無 answer here. I've written a cache dump script.
The script dumps all the content of a memcached server. It's tested with Ubuntu 12.04 and a localhost memcached, so your milage may vary.
#!/usr/bin/env bash
echo 'stats items' \
| nc localhost 11211 \
| grep -oe ':[0-9]*:' \
| grep -oe '[0-9]*' \
| sort \
| uniq \
| xargs -L1 -I{} bash -c 'echo "stats cachedump {} 1000" | nc localhost 11211'
What it does, it goes through all the cache slabs and print 1000 entries of each.
Please be aware of certain limits of this script i.e. it may not scale for a 5GB cache server for example. But it's useful for debugging purposes on a local machine.
If you have PHP & PHP-memcached installed, you can run
$ php -r '$c = new Memcached(); $c->addServer("localhost", 11211); var_dump( $c->getAllKeys() );'
Bash
To get list of keys in Bash, follow the these steps.
First, define the following wrapper function to make it simple to use (copy and paste into shell):
function memcmd() {
exec {memcache}<>/dev/tcp/localhost/11211
printf "%s\n%s\n" "$*" quit >&${memcache}
cat <&${memcache}
}
Memcached 1.4.31 and above
You can use lru_crawler metadump all command to dump (most of) the metadata for (all of) the items in the cache.
As opposed to cachedump, it does not cause severe performance problems and has no limits on the amount of keys that can be dumped.
Example command by using the previously defined function:
memcmd lru_crawler metadump all
See: ReleaseNotes1431.
Memcached 1.4.30 and below
Get list of slabs by using items statistics command, e.g.:
memcmd stats items
For each slub class, you can get list of items by specifying slub id along with limit number (0 - unlimited):
memcmd stats cachedump 1 0
memcmd stats cachedump 2 0
memcmd stats cachedump 3 0
memcmd stats cachedump 4 0
...
Note: You need to do this for each memcached server.
To list all the keys from all stubs, here is the one-liner (per one server):
for id in $(memcmd stats items | grep -o ":[0-9]\+:" | tr -d : | sort -nu); do
memcmd stats cachedump $id 0
done
Note: The above command could cause severe performance problems while accessing the items, so it's not advised to run on live.
Notes:
stats cachedump only dumps the HOT_LRU (IIRC?), which is managed by a background thread as activity happens. This means under a new enough version which the 2Q algo enabled, you'll get snapshot views of what's in just one of the LRU's.
If you want to view everything, lru_crawler metadump 1 (or lru_crawler metadump all) is the new mostly-officially-supported method that will asynchronously dump as many keys as you want. you'll get them out of order but it hits all LRU's, and unless you're deleting/replacing items multiple runs should yield the same results.
Source: GH-405.
Related:
List all objects in memcached
Writing a Redis client in pure bash (it's Redis, but very similar approach)
Check other available commands at https://memcached.org/wiki
Check out the protocol.txt docs file.
The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats
The keys() method should get you going.
Example -
from memcached_stats import MemcachedStats
mem = MemcachedStats()
mem.keys()
['key-1',
'key-2',
'key-3',
... ]
I was using Java's spyMemcached, and used this code. It is based on Anshul Goyal's answer
#Autowired
#Qualifier("initMemcachedClient")
private MemcachedClient memcachedClient;
public List<String> getCachedKeys(){
Set<Integer> slabIds = new HashSet<>();
Map<SocketAddress, Map<String, String>> stats;
List<String> keyNames = new ArrayList<>();
// Gets all the slab IDs
stats = memcachedClient.getStats("items");
stats.forEach((socketAddress, value) -> {
System.out.println("Socket address: "+socketAddress.toString());
value.forEach((propertyName, propertyValue) -> {
slabIds.add(Integer.parseInt(propertyName.split(":")[1]));
});
});
// Gets all keys in each slab ID and adds in List keyNames
slabIds.forEach(slabId -> {
Map<SocketAddress, Map<String, String>> keyStats = memcachedClient.getStats("cachedump "+slabId+" 0");
keyStats.forEach((socketAddress, value) -> {
value.forEach((propertyName, propertyValue) -> {
keyNames.add(propertyName);
});
});
});
System.out.println("number of keys: "+keyNames.size());
return keyNames;
}
Java Solution:
Thanks! #Satvik Nema
Your solution helped me to find the approach, but it doesn't work for memcached 2.4.6 version. (implementation 'com.googlecode.xmemcached:xmemcached:2.4.6')
Not sure when did new method getStatsByItem included.
I figured out required changes using documentation and below code worked for me.
// Gets all the slab IDs
Set<Integer> slabIds = new HashSet<>();
Map<InetSocketAddress, Map<String, String>> itemsMap = null;
try {
itemsMap = this.memcachedClient.getStatsByItem("items");
} catch (Exception e) {
log.error("Failed while pulling 'items'. ERROR", e);
}
if (Objects.nonNull(itemsMap)) {
itemsMap.forEach((key, value) -> {
log.info("itemsMap {} : {}", key, value);
value.forEach((k, v) -> {
slabIds.add(Integer.parseInt(k.split(":")[1]));
});
});
}
// Gets all keys in each slab ID and adds in List keyNames
slabIds.forEach(slabId -> {
Map<InetSocketAddress, Map<String, String>> keyStats = null;
try {
keyStats = this.memcachedClient.getStatsByItem("cachedump " + slabId + " 0");
} catch (Exception e) {
log.error("Failed while pulling 'cachedump' for slabId: {}. ERROR", slabId, e);
}
if (Objects.nonNull(keyStats)) {
keyStats.forEach((socketAddress, value) -> {
value.forEach((propertyName, propertyValue) -> {
//keyNames.add(propertyName);
log.info("keyName: {} Value: {}", propertyName, propertyValue);
});
});
}
});
I'll apologise in advance, I'm new to groovy. The problem I have is I have 3 groovy scripts which perform different functionality, and I need to call them from my main groovy script, using the output from script 1 as input for script 2 and script 2's output as input for script 3.
I've tried the following code:
script = new GroovyShell(binding)
script.run(new File("script1.groovy"), "--p", "$var" ) | script.run(new File("script2.groovy"), "<", "$var" )
When I run the above code the first script runs successfully but the 2nd doesn't run at all.
Script 1 takes an int as a parameter using the "--p", "$var" code. This runs successfully in the main script using: script.run(new File("script1.groovy"), "--p", "$var" ) - Script 1's output is an xml file.
When I run script.run(new File("script2.groovy"), "<", "$var" ) on its own in the main groovy script nothing happens and the system hangs.
I can run script 2 from the command line using groovy script2.groovy < input_file and it works fine.
Any help would be greatly appreciated.
You cannot pass the < as an argument to the script as redirection is handled by the Shell when you run things from the command line...
Redirecting output from Scripts into other scripts is notoriously difficult, and basically relies on you changing System.out for the duration of each script (and hoping that nothing else in the JVM prints and messes up your data)
Better to use java processes like the following:
Given these 3 scripts:
script1.groovy
// For each argument
args.each {
// Wrap it in xml and write it out
println "<woo>$it</woo>"
}
linelength.groovy
// read input
System.in.eachLine { line ->
// Write out the number of chars in each line
println line.length()
}
pretty.groovy
// For each line print out a nice report
int index = 1
System.in.eachLine { line ->
println "Line $index contains $line chars (including the <woo></woo> bit)"
index++
}
We can then write something like this to get a new groovy process to run each in turn, and pipe the outputs into each other (using the overloaded or operator on Process):
def s1 = 'groovy script1.groovy arg1 andarg2'.execute()
def s2 = 'groovy linelength.groovy'.execute()
def s3 = 'groovy pretty.groovy'.execute()
// pipe the output of process1 to process2, and the output
// of process2 to process3
s1 | s2 | s3
s3.waitForProcessOutput( System.out, System.err )
Which prints out:
Line 1 contains 15 chars (including the <woo></woo> bit)
Line 2 contains 18 chars (including the <woo></woo> bit)
//store standard I/O
PrintStream systemOut = System.out
InputStream systemIn = System.in
//Buffer for exchanging data between scripts
ByteArrayOutputStream buffer = new ByteArrayOutputStream()
PrintStream out = new PrintStream(buffer)
//Redirecting "out" of 1st stream to buffer
System.out = out
//RUN 1st script
evaluate("println 'hello'")
out.flush()
//Redirecting buffer to "in" of 2nd script
System.in = new ByteArrayInputStream(buffer.toByteArray())
//set standard "out"
System.out = systemOut
//RUN 2nd script
evaluate("println 'message from the first script: ' + new Scanner(System.in).next()")
//set standard "in"
System.in = systemIn
result is: 'message from the first script: hello'
I have one groovy script which print some statistics: println: "..."
now I have another groovy script which needs this data. Is possible somehow run first script from second and save this data as paramater and then work with them from second script ? I just know how to run script: with GroovyShell() and then run(...) but this doesnt return output of first script
A few options:
If you're calling it from a script, redefine stdout.
Fix the first script so it prints data retrieved from a class, and re-write the calling script to use that class instead of relying on the printed output from the first. Long-term may be the best option.
Use a pipe on the command line: groovy s1.groovy | groovy s2.groovy
Personally, when composing things that do stuff with stdin/stdio, I prefer the last method. Example:
s1.groovy
5.times { println it }
s2.groovy
r = new BufferedReader(new InputStreamReader(System.in))
while (l = r.readLine()) { println((l as Integer) * 2) }
Output
$ groovy s1.groovy
0
1
2
3
4
$ groovy s1.groovy | groovy s2.groovy
0
2
4
6
8
One way to do this would be to set the out parameter in the binding when calling the first script:
So given a script s1.groovy:
//Print the letters of 'tim_yates', one per line
'tim_yates'.each this.&println
We can do (in s2.groovy)
// Create a StringWriter that will capture output
String output = new StringWriter().with { sw ->
// And make a Binding for our script
new Binding().with { b ->
// Set 'out' in the Binding to be our StringWriter
b[ 'out' ] = sw
// evaluate the file with the GroovyShell (using the binding)
new GroovyShell( b ).evaluate( new File( 's1.groovy' ) )
}
// And return the String captured in our writer
sw.toString()
}
println output
And then run it with groovy s2.groovy
Edit
I think this is option #1 in Dave's answer...