I have a mission to make a "stamp creator" for Powder Toy. The "stamp" looks like this: http://i.imgur.com/3Cac3.jpg . I'd like to use the bytes' values for my creator. For that I need to read the unencoded content of the file. Any help? Also, I'll need to create such content as well. (Maybe I'm better off using a byte[] array?(but how to enter the special control characters?))
You can easily use File.ReadAllBytes to read the file, and File.WriteAllBytes to write it back.
If you need more control instead of reading an entire file into memory, you can use File.OpenRead and File.OpenWrite to read and write using a Stream. If you do that make sure you dispose of it - using the using keyword is encouraged.
Here are a couple of examples
public void CopyFileByReadingItAllToMemory() {
byte[] data = File.ReadAllBytes(#"C:\Temp\input");
File.WriteAllBytes(#"C:\temp\output", data);
}
public void CopyFileWithoutReadingItAllToMemory() {
using (Stream input = File.OpenRead(#"C:\Temp\input"))
using (Stream output = File.OpenWrite(#"C:\Temp\output"))
{
while (true) {
int value = input.ReadByte();
if (value == -1) {
break;
}
output.WriteByte((byte)value);
}
}
}
Related
Have been stringbuilding CSV files for ages on MVC applications just fine, until now.
One mistake made me generate a CSV string bigger then the system can handle in memory, so i have been searching the web for any solution on minifing a string that could be reconstructed back on client.
So far i have been doing this:
StringBUilder sb = new StringBuilder();
foreach(stuff in manyEnumerableStuff)
sb.Append(stuff);
return csv.ToString().ToBase64();
public static string ToBase64(this string value) => Convert.ToBase64String(Encoding.Default.GetBytes(value));
The application can handle .ToString() in this HUGE case just "fine", but it fails without creating excpetions at .ToBase64String(Encoding.Default.GetBytes(value));
This only happens on huge strings because from what i know, base64 will make the string 33% bigger.
Compressed json can't solve this problem, since this happens on server side.
So I have gonne on search to minify or compress this string, but it still need to be a string and can be converted on client site Angular application.
I have found this:
public static string compress(this string big) {
var bytes = Encoding.Default.GetBytes(big);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream()) {
using (var gs = new GZipStream(mso, CompressionMode.Compress)) {
//msi.CopyTo(gs);
CopyTo(msi, gs);
}
return mso.ToArray().ToString();
}
}
private static void CopyTo(Stream src, Stream dest) {
byte[] bytes = new byte[4096];
int cnt;
while ((cnt = src.Read(bytes, 0, bytes.Length)) != 0) {
dest.Write(bytes, 0, cnt);
}
}
but I think there is no sense at all, because i can't put byte[] on json value as string without converting it back.
Is it possible to compress plain Pipe separated values that represents a .CSV file after getting the string from StringBuilder()?
I have tried GC.collect() right after parsing SB to string but still broke the application.
I'm on .Net Core 2.1, Linux server.
I have a custom rest control written in Java. Everything works fine for string fields, but not for multi-value string fields. My code returns
"[Value1, Value2, Value3...]". Note there are no commas around the values. On the web page the output looks like this:
If I can get commas around the values, the front-end framework can easily parse it.
I have tried to parse the value from the field and to get it formatted correctly, but cannot seem to get it right.
The first set of code works for a string. The second is an attempt to work for a multi-value field.
Any help would be greatly appreciated.
writer.startProperty("chgTitle");
writer.outStringLiteral(String.valueOf(columnValues.get(0)));
writer.endProperty();
writer.startProperty("plpAffected");
Object tmpObj = columnValues.get(5);
if (tmpObj instanceof Vector) {
String[] copyArr = new String[((Vector) tmpObj).size()];
((Vector) tmpObj).copyInto(copyArr);
writer.outStringLiteral(String.valueOf(copyArr));
} else {
}
writer.endProperty();
I would recommend a slightly different approach. I like to force my JSON structures to use an array anywhere a multiple value situation is possible. It also looks like you're using Philippe Riand's specialized JSON writer, so I'll assume that.
Here's about how I would attack it:
writer.startProperty("chgTitle");
writer.outStringLiteral(String.valueOf(columnValues.get(0)));
writer.endProperty();
writer.startProperty("plpAffected");
Vector<?> tmpVec = Util.getValueAsVector(columnValues.get(5));
writer.startArray();
for ( Object ob : tmpVec ) {
writer.startArrayItem();
// assuming String contents
writer.outStringLiteral(String.valueOf(ob));
writer.endArrayItem();
}
writer.endArray();
writer.endProperty();
To wrap the returning columnValues, I'm using a Java equivalent of my SSJS getValueAsVector helper function. It checks for Vector or ArrayList, which I happen to use almost exclusively; if it's not, it shoves it into a new Vector. Here's what that method looks like,
public static Vector<?> getValueAsVector(Object obj) {
if(obj instanceof java.util.Vector){
return (Vector<?>) obj;
}else if( obj instanceof ArrayList ) {
List<?> o = (List<?>) obj;
Vector<Object> tmpVec = new Vector<Object>();
for(int i=0;i<o.size();i++){
tmpVec.add(o.get(i));
}
return tmpVec;
}else {
Vector<Object> tmpVec = new Vector<Object>();
tmpVec.add(obj);
return tmpVec;
}
}
I'm exporting MIME eMails with the following code:
public String fromRawMime(final Session s, final Document doc) throws NotesException {
final Stream notesStream = s.createStream();
final MIMEEntity rootMime = doc.getMIMEEntity();
// check if it is multi-part or single
if (rootMime.getContentType().equals("multipart")) {
this.printMIME(rootMime, notesStream);
} else {
// We can just write the content into the
// Notes stream to get the bytes
rootMime.getEntityAsText(notesStream);
}
// Write it out
notesStream.setPosition(0);
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.append(notesStream.read());
notesStream.close();
notesStream.recycle();
rootMime.recycle();
return out.toString();
}
// Write out a mime entry to a Stream object, includes sub entries
private void printMIME(final MIMEEntity mimeRoot, final Stream out) throws NotesException {
if (mimeRoot == null) {
return;
}
// Encode binary as base64
if (mimeRoot.getEncoding() == MIMEEntity.ENC_IDENTITY_BINARY) {
mimeRoot.decodeContent();
mimeRoot.encodeContent(MIMEEntity.ENC_BASE64);
}
out.writeText(mimeRoot.getBoundaryStart(), Stream.EOL_NONE);
mimeRoot.getEntityAsText(out);
out.writeText(mimeRoot.getBoundaryEnd(), Stream.EOL_NONE);
if (mimeRoot.getContentType().equalsIgnoreCase("multipart")) {
// Print preamble if it isn't empty
final String preamble = mimeRoot.getPreamble();
if (!preamble.isEmpty()) {
out.writeText(preamble, Stream.EOL_NONE);
}
// Print content of each child entity - recursive calls
// Include recycle of mime elements
MIMEEntity mimeChild = mimeRoot.getFirstChildEntity();
while (mimeChild != null) {
this.printMIME(mimeChild, out);
final MIMEEntity mimeNext = mimeChild.getNextSibling();
// Recycle to ensure we don't bleed memory
mimeChild.recyle();
mimeChild = mimeNext;
}
}
}
The result contains one empty line for each line. Including the content that gets added using getEntityAsText. What am I missing to get rid of the extra lines?
The email RFCs require the use of CRLF to terminate text lines.
You are using EOL_NONE, so the writeText method isn't adding anything to the text, but apparently both the CR and LF are being treated as newlines in your output. You may want to try using out.writeText with EOL_PLATFORM instead.
The devils is in the details...
the printMIME function works just fine. Changing the EOL didn't have an impact. However I added EOL_PLATFORM later on for the final result to separate the headers from the content.
The offending code is this:
notesStream.setPosition(0);
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.append(notesStream.read());
notesStream.close();
Turns out that it seems to interpret whatever was in the MIME as 2 line feeds. So the code needs to be changed to:
notesStream.setPosition(0);
String out = notesStream.readText();
notesStream.close();
so instead of a OutputStream I needed a String and instead of read() I needed readText(). Now working happily in my "project castle"
I would like to get a list of most possible list of tokens for a given location in the text (line and column number) to determine what has to be populated for auto code completion. Can this be easily achieved using ANTLR 4 API.
I want to get the possible list of tokens for a given location because the user might be writing/editing somewhere in the middle of the text which still guarantees the possible list of tokens.
Please give me some guidelines because I was unable to find an online resource on this topic.
One way to get tokens by line number is to create a ParseTreeListener for your grammar, use it to walk a given ParseTree and index TerminalNodes by line number. I don't know C#, but here is how I've done it in Java. Logic should be similar.
public class MyLineIndexer extends MyGrammarParserBaseListener {
protected MultiMap<Integer, TerminalNode> filelineTokenIndex;
#Override
public void visitTerminal(#NotNull TerminalNode node) {
// map every token to its file line for searching later...
if ( node.getSymbol() != null ) {
List<TerminalNode> tokens;
Integer line = node.getSymbol().getLine();
if (!filelineTokenIndex.containsKey(line)) {
tokens = new ArrayList<>();
filelineTokenIndex.put(line, tokens);
} else {
tokens = filelineTokenIndex.get(line);
}
tokens.add(node);
}
super.visitTerminal(node);
}
}
then walk the parse tree the usual way...
ParseTree parseTree = ... ; // parse it how you want to
MyLineIndexer indexer = new MyLineIndexer();
ParseTreeWalker walker = new ParseTreeWalker();
walker.walk(indexer, parseTree);
Getting the token at a line and range is now reasonably straight forward and efficient assuming you have a reasonable number of tokens on a line. For example you can add another method to the Listener like this:
public TerminalNode findTerminalNodeAtCaret(int caretPos, int caretLine) {
if (caretPos <= 0) return null;
if (this.filelineTokenIndex.containsKey(caretLine)) {
List<TerminalNode> nodes = filelineTokenIndex.get(caretLine);
if (nodes.size() == 0) return null;
int tokenEndPos, tokenStartPos;
for (TerminalNode n : nodes) {
if (n.getSymbol() != null) {
tokenEndPos = n.getSymbol().getCharPositionInLine() + n.getText().length();
tokenStartPos = n.getSymbol().getCharPositionInLine();
// If the caret is within this token, return this token
if (caretPos >= tokenStartPos && caretPos <= tokenEndPos) {
return n;
}
}
}
}
return null;
}
You will also need to ensure your parser allows for 'loose' parsing. While a language construct is being typed, it is likely not to be valid. Your Parser rules should allow for this.
The below simple java code getting Fortify Path Manipulation error. Please help me to resolve this. I am struggling from long time.
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
}
}
Try to normalize the URL before using it
https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#normalize()
Path path = Paths.get("/foo/../bar/../baz").normalize();
or use normalize from org.apache.commons.io.FilenameUtils
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html#normalize(java.lang.String)
Stirng path = FilenameUtils.normalize("/foo/../bar/../baz");
For both the result will be \baz
Looking at the OWASP page for Path Manipulation, it says
An attacker can specify a path used in an operation on the filesystem
You are opening a file as defined by a user-given input. Your code is almost a perfect example of the vulnerability! Either
Don't use the above code (don't let the user specify the input file as an argument)
Let the user choose from a list of files that you supply (an array of files with an integer choice)
Don't let the user supply the filename at all, remove the configurability
Accept the vulnerability but protect against it by checking the filename (although this is the worst thing to do - someone may get round it anyway).
Or re-think your application's design.
Fortify will flag the code even if the path/file doesn't come from user input like a property file. The best way to handle these is to canonicalize the path first, then validate it against a white list of allowed paths.
Bad:
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
}
}
Good:
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
if (!isInSecureDir(file)) {
throw new IllegalArgumentException();
}
String canonicalPath = file.getCanonicalPath();
if (!canonicalPath.equals("/img/java/file1.txt") &&
!canonicalPath.equals("/img/java/file2.txt")) {
// Invalid file; handle error
}
FileInputStream fis = new FileInputStream(f);
}
Source: https://www.securecoding.cert.org/confluence/display/java/FIO16-J.+Canonicalize+path+names+before+validating+them
Only allow alnum and a period in input. That means you filter out the control chars, "..", "/", "\" which would make your files vulnerable. For example, one should not be able to enter /path/password.txt.
Once done, rescan and then run Fortify AWB.
I have a solution to the Fortify Path Manipulation issues.
What it is complaining about is that if you take data from an external source, then an attacker can use that source to manipulate your path. Thus, enabling the attacker do delete files or otherwise compromise your system.
The suggested remedy to this problem is to use a whitelist of trusted directories as valid inputs; and, reject everything else.
This solution is not always viable in a production environment. So, I suggest an alternative solution. Parse the input for a whitelist of acceptable characters. Reject from the input, any character you don't want in the path. It could be either removed or replaced.
Below is an example. This does pass the Fortify review. It is important to remember here to return the literal and not the char being checked. Fortify keeps track of the parts that came from the original input. If you use any of the original input, you may still get the error.
public class CleanPath {
public static String cleanString(String aString) {
if (aString == null) return null;
String cleanString = "";
for (int i = 0; i < aString.length(); ++i) {
cleanString += cleanChar(aString.charAt(i));
}
return cleanString;
}
private static char cleanChar(char aChar) {
// 0 - 9
for (int i = 48; i < 58; ++i) {
if (aChar == i) return (char) i;
}
// 'A' - 'Z'
for (int i = 65; i < 91; ++i) {
if (aChar == i) return (char) i;
}
// 'a' - 'z'
for (int i = 97; i < 123; ++i) {
if (aChar == i) return (char) i;
}
// other valid characters
switch (aChar) {
case '/':
return '/';
case '.':
return '.';
case '-':
return '-';
case '_':
return '_';
case ' ':
return ' ';
}
return '%';
}
}
Assuming you're running Fortify against a web application, during your triage of Fortify vulnerabilities that would likely get marked as "Not an issue". Reasoning being A) obviously this is test code and B) unless you have multiple personality disorder you're not going to be doing a path manipulation exploit against your self when you run that test app.
If very common to see little test utilities committed to a repository which produces this style of false positive.
As for your compilation errors, that generally comes down to classpath issues.
We have code like below which was raising Path Manipulation high category issue in fortify .
String.join(delimeter,string1,string2,string2,string4);
Our program is to deal with AWS S3 bucket so, we changed as below and it worked .
com.amazonaws.util.StringUtils.join(delimeter,string1,string2,string2,string4);
Using the Tika library FilenameUtils.normalize solves the fortify issue.
import org.apache.tika.io.FilenameUtils;
public class Test {
public static void main(String[] args) {
String filePath = FilenameUtils.normalize(args[0]); //This line solve issue.
File file=new File(filePath);
}
}
Try this for replacing FileInputStream. You will need to close your project and open again to accurately see whether changes worked.
File to byte[] in Java
Use Normalize() function in C# and it resolved the fortify vulnerability in next scan.
string s = #:c:\temp\scan.log".Normalize();
Use regex to validate the file path and file name
fileName = args[0];
final String regularExpression = "([\\w\\:\\\\w ./-]+\\w+(\\.)?\\w+)";
Pattern pattern = Pattern.compile(regularExpression);
boolean isMatched = pattern.matcher(fileName).matches();