I'm making a service that among other has the "photo albums" feature that serve photos to users. User has to be "allowed" to see the photo from the album. So sending the direct link to other person shouldn't allow to view photo.
Photos are stored in the folder outside of the context.
What I need to do is to perform some checks when user requests the photo and then if checks are OK - serve the file. I want to avoid making a wheel and just let tomcat serve the image as it usually does for static files. Can you give some advice on that?
Ok, guys.
After struggling hard with this question I think I've finally found out what to do to solve it. First of all it looks like the question actually decomposes into two independent tasks. One of them is securing access to some resources and second one is feeding resources from the folder outside of the context.
First task is trivial and can be solved by writing a simple filter hanged to "/".
Second task is much less trivial but fortunately also can be resolved. Tomcat uses the implementation of javax.naming.directory.DirContext to load all resources of the given web application including class files. It also allows you to provide the custom implementation of this interface and configure it in the context.xml file. The default implementation is org.apache.naming.resources.FileDirContext. Details here: http://tomcat.apache.org/tomcat-6.0-doc/config/resources.html
I've created my own implementation of DirContext by simply extending FileDirContext. Luckily enough there was a single method that had to be overwritten in order to "hook up" file discovery. The method is called file().
I'm posting my test code here. It is far from perfect and does not take into account the corner cases like renaming files but I don't think that these are needed under a normal run of the server.
The basic idea under this code is to check if the path starts with "virtual directory" prefix and if it is - search for file in the other place in the filesystem (I know there is some duplicate code there but I hope you're not that lazy to remove it if you ever want to use it :-). setVirtualName and setVirtualBase are called automatically to inject the configuration params.
/**
* TODO: add javadocs
*
* #author Juriy Bura
*/
public class VirtualFolderDirContext extends FileDirContext {
private String virtualName;
private String realName;
private File virtualBase;
private String absoluteVirtualBase;
public VirtualFolderDirContext() {
super();
}
public VirtualFolderDirContext(Hashtable env) {
super(env);
}
public void setVirtualName(String path) {
virtualName = path;
}
public void setVirtualBase(String base) {
this.realName = base;
virtualBase = new File(realName);
try {
virtualBase = virtualBase.getCanonicalFile();
} catch (IOException e) {
// Ignore
}
this.absoluteVirtualBase = virtualBase.getAbsolutePath();
}
protected File file(String name) {
File file = null;
boolean virtualFile = name.startsWith(virtualName + "/");
if (virtualFile) {
file = new File(virtualBase, name.substring(virtualName.length()));
} else {
file = new File(base, name);
}
if (file.exists() && file.canRead()) {
if (allowLinking)
return file;
// Check that this file belongs to our root path
String canPath = null;
try {
canPath = file.getCanonicalPath();
} catch (IOException e) {
}
if (canPath == null)
return null;
// Check to see if going outside of the web application root
if (!canPath.startsWith(absoluteBase) && !canPath.startsWith(absoluteVirtualBase)) {
return null;
}
// Case sensitivity check
if (caseSensitive) {
String fileAbsPath = file.getAbsolutePath();
if (fileAbsPath.endsWith("."))
fileAbsPath = fileAbsPath + "/";
String absPath = normalize(fileAbsPath);
if (canPath != null)
canPath = normalize(canPath);
if (virtualFile) {
if ((absoluteVirtualBase.length() < absPath.length())
&& (absoluteVirtualBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteVirtualBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteVirtualBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
} else {
if ((absoluteBase.length() < absPath.length())
&& (absoluteBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
}
}
} else {
return null;
}
return file;
}
}
After you have this class in place you have to jar it and put that jar into the Tomcat lib folder. For obvious reasons it cannot go together with war file. In your context.xml you should add a config lines like these:
<?xml version="1.0" encoding="UTF-8"?>
<Context antiResourceLocking="true" antiJARLocking="true">
<Resources
className="com.juriy.tomcat.virtualdir.VirtualFolderDirContext"
virtualName="/upload"
virtualBase="c:/temp/up">
</Resources>
...
...
Now any time user asks for /upload/ it will be resolved to c:\temp. With this technique you can implement loading resources from virtually any location: http, shared folder, database, even version control system. So it is pretty cool.
P.S. I've killed the whole day to make this all work together so don't hesitate to give me your vote if you like the answer :-))
Cheers
Juriy
Related
In the below code test.txt exists before it is run and test2.txt does not. When run destFile.Exists returns null after the file is copied to destFile's location. What is causing this? I cannot find any information in msdn that supports what is happening.
var origFile = new FileInfo(#"C:\Users\user\Desktop\CopyTest\test.txt");
var destFile = new FileInfo(#"C:\Users\user\Desktop\CopyTest\test2.txt");
if (!destFile.Exists && origFile.Exists)
origFile.CopyTo(destFile.FullName);
if (destFile.Exists)
Console.WriteLine("The file was found");
Console.ReadLine();
Try to use destFile.Refresh(); before you access the property
destFile.Refresh();
if (destFile.Exists)
Console.WriteLine("The file was found");
or use the static method File.Exists:
if (File.Exists(#"C:\Users\user\Desktop\CopyTest\test2.txt"))
Console.WriteLine("The file was found");
The FileInfo provides a lot of informations, but this is a snapshot which will be initialized the first time you access it and will not be updated later. So use it only if you need current state and if you need multiple informations. Otherwise use the static methods in System.IO.File.
Here you can see the current implementation of the Exists property. You see that it's initializing it the first time you access it, later the old state will be returned:
public override bool Exists {
[System.Security.SecuritySafeCritical] // auto-generated
get {
try {
if (_dataInitialised == -1)
Refresh();
if (_dataInitialised != 0) {
// Refresh was unable to initialise the data.
// We should normally be throwing an exception here,
// but Exists is supposed to return true or false.
return false;
}
return (_data.fileAttributes & Win32Native.FILE_ATTRIBUTE_DIRECTORY) == 0;
}
catch
{
return false;
}
}
In one of my Azure Web App Web API application, I am creating temp files using this code in a Get method
string path = Path.GetTempFileName();
// do some writing on this file. then read
var fileStream = File.OpenRead(path);
// then returning this stream as a HttpResponseMessage response
My question is, in a managed environment like this (not in VM), do I need to clear those temporary files by myself?
Shouldn't Azure itself supposed to clear those temp files?
Those files only get cleaned when your site is restarted.
If your site is running in Free or Shared mode, it only gets 300MB for temp files, so you could run out if you don't clean up.
If your site is in Basic or Standard mode, then there is significantly more space (around 200GB!). So you could probably get away with not cleaning up without running into the limit. Eventually, your site will get restarted (e.g. during platform upgrade), so things will get cleaned up.
See this page for some additional detail on this topic.
Maybey if you extend FileStream you can override dispose and remove it when disposed is called? That is how i'm resolving it for now. If i'm wrong let me know.
/// <summary>
/// Create a temporary file and removes its when the stream is closed.
/// </summary>
internal class TemporaryFileStream : FileStream
{
public TemporaryFileStream() : base(Path.GetTempFileName(), FileMode.Open)
{
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
// After the stream is closed, remove the file.
File.Delete(Name);
}
}
Following sample demonstrate how to save temp file in azure, both Path and Bolb.
Doc is here:https://code.msdn.microsoft.com/How-to-store-temp-files-in-d33bbb10
Code click here:https://github.com/Azure-Samples/storage-blob-dotnet-store-temp-files/archive/master.zip
Under part is the core logic of bolb code:
private long TotalLimitSizeOfTempFiles = 100 * 1024 * 1024;
private async Task SaveTempFile(string fileName, long contentLenght, Stream inputStream)
{
try
{
await container.CreateIfNotExistsAsync();
CloudBlockBlob tempFileBlob = container.GetBlockBlobReference(fileName);
tempFileBlob.DeleteIfExists();
await CleanStorageIfReachLimit(contentLenght);
tempFileBlob.UploadFromStream(inputStream);
}
catch (Exception ex)
{
if (ex.InnerException != null)
{
throw ex.InnerException;
}
else
{
throw ex;
}
}
}
private async Task CleanStorageIfReachLimit(long newFileLength)
{
List<CloudBlob> blobs = container.ListBlobs()
.OfType<CloudBlob>()
.OrderBy(m => m.Properties.LastModified)
.ToList();
long totalSize = blobs.Sum(m => m.Properties.Length);
long realLimetSize = TotalLimitSizeOfTempFiles - newFileLength;
foreach (CloudBlob item in blobs)
{
if (totalSize <= realLimetSize)
{
break;
}
await item.DeleteIfExistsAsync();
totalSize -= item.Properties.Length;
}
}
I want to download the files from a sharepoint document library through code as there are thousand of files in the document library.
I am thinking of creating console application, which I will run on sharepoint server and download files. Is this approach correct or, there is some other efficient way to do this.
Any help with code will be highly appreciated.
Like SigarDave said, it's perfectly possible to achieve this without writing a single line of code. But if you really want to code the solution for this, it's something like:
static void Main(string[] args)
{
// Change to the URL of your site
using (var site = new SPSite("http://MySite"))
using (var web = site.OpenWeb())
{
var list = web.Lists["MyDocumentLibrary"]; // Get the library
foreach (SPListItem item in list.Items)
{
if (item.File != null)
{
// Concat strings to get the absolute URL
// to pass to an WebClient object.
var fileUrl = string.Format("{0}/{1}", site.Url, item.File.Url);
var result = DownloadFile(fileUrl, "C:\\FilesFromMyLibrary\\", item.File.Name);
Console.WriteLine(result ? "Downloaded \"{0}\"" : "Error on \"{0}\"", item.File.Name);
}
}
}
Console.ReadKey();
}
private static bool DownloadFile(string url, string dest, string fileName)
{
var client = new WebClient();
// Change the credentials to the user that has the necessary permissions on the
// library
client.Credentials = new NetworkCredential("Username", "Password", "Domain");
var bytes = client.DownloadData(url);
try
{
using (var file = File.Create(dest + fileName))
{
file.Write(bytes, 0, bytes.Length); // Write file to disk
return true;
}
}
catch (Exception)
{
return false;
}
}
another way without using any scripts is by opening the document library using IE then in the ribbon you can click on Open in File Explorer where you can then drag and drop the files right on your desktop!
The below simple java code getting Fortify Path Manipulation error. Please help me to resolve this. I am struggling from long time.
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
}
}
Try to normalize the URL before using it
https://docs.oracle.com/javase/7/docs/api/java/net/URI.html#normalize()
Path path = Paths.get("/foo/../bar/../baz").normalize();
or use normalize from org.apache.commons.io.FilenameUtils
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html#normalize(java.lang.String)
Stirng path = FilenameUtils.normalize("/foo/../bar/../baz");
For both the result will be \baz
Looking at the OWASP page for Path Manipulation, it says
An attacker can specify a path used in an operation on the filesystem
You are opening a file as defined by a user-given input. Your code is almost a perfect example of the vulnerability! Either
Don't use the above code (don't let the user specify the input file as an argument)
Let the user choose from a list of files that you supply (an array of files with an integer choice)
Don't let the user supply the filename at all, remove the configurability
Accept the vulnerability but protect against it by checking the filename (although this is the worst thing to do - someone may get round it anyway).
Or re-think your application's design.
Fortify will flag the code even if the path/file doesn't come from user input like a property file. The best way to handle these is to canonicalize the path first, then validate it against a white list of allowed paths.
Bad:
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
}
}
Good:
public class Test {
public static void main(String[] args) {
File file=new File(args[0]);
if (!isInSecureDir(file)) {
throw new IllegalArgumentException();
}
String canonicalPath = file.getCanonicalPath();
if (!canonicalPath.equals("/img/java/file1.txt") &&
!canonicalPath.equals("/img/java/file2.txt")) {
// Invalid file; handle error
}
FileInputStream fis = new FileInputStream(f);
}
Source: https://www.securecoding.cert.org/confluence/display/java/FIO16-J.+Canonicalize+path+names+before+validating+them
Only allow alnum and a period in input. That means you filter out the control chars, "..", "/", "\" which would make your files vulnerable. For example, one should not be able to enter /path/password.txt.
Once done, rescan and then run Fortify AWB.
I have a solution to the Fortify Path Manipulation issues.
What it is complaining about is that if you take data from an external source, then an attacker can use that source to manipulate your path. Thus, enabling the attacker do delete files or otherwise compromise your system.
The suggested remedy to this problem is to use a whitelist of trusted directories as valid inputs; and, reject everything else.
This solution is not always viable in a production environment. So, I suggest an alternative solution. Parse the input for a whitelist of acceptable characters. Reject from the input, any character you don't want in the path. It could be either removed or replaced.
Below is an example. This does pass the Fortify review. It is important to remember here to return the literal and not the char being checked. Fortify keeps track of the parts that came from the original input. If you use any of the original input, you may still get the error.
public class CleanPath {
public static String cleanString(String aString) {
if (aString == null) return null;
String cleanString = "";
for (int i = 0; i < aString.length(); ++i) {
cleanString += cleanChar(aString.charAt(i));
}
return cleanString;
}
private static char cleanChar(char aChar) {
// 0 - 9
for (int i = 48; i < 58; ++i) {
if (aChar == i) return (char) i;
}
// 'A' - 'Z'
for (int i = 65; i < 91; ++i) {
if (aChar == i) return (char) i;
}
// 'a' - 'z'
for (int i = 97; i < 123; ++i) {
if (aChar == i) return (char) i;
}
// other valid characters
switch (aChar) {
case '/':
return '/';
case '.':
return '.';
case '-':
return '-';
case '_':
return '_';
case ' ':
return ' ';
}
return '%';
}
}
Assuming you're running Fortify against a web application, during your triage of Fortify vulnerabilities that would likely get marked as "Not an issue". Reasoning being A) obviously this is test code and B) unless you have multiple personality disorder you're not going to be doing a path manipulation exploit against your self when you run that test app.
If very common to see little test utilities committed to a repository which produces this style of false positive.
As for your compilation errors, that generally comes down to classpath issues.
We have code like below which was raising Path Manipulation high category issue in fortify .
String.join(delimeter,string1,string2,string2,string4);
Our program is to deal with AWS S3 bucket so, we changed as below and it worked .
com.amazonaws.util.StringUtils.join(delimeter,string1,string2,string2,string4);
Using the Tika library FilenameUtils.normalize solves the fortify issue.
import org.apache.tika.io.FilenameUtils;
public class Test {
public static void main(String[] args) {
String filePath = FilenameUtils.normalize(args[0]); //This line solve issue.
File file=new File(filePath);
}
}
Try this for replacing FileInputStream. You will need to close your project and open again to accurately see whether changes worked.
File to byte[] in Java
Use Normalize() function in C# and it resolved the fortify vulnerability in next scan.
string s = #:c:\temp\scan.log".Normalize();
Use regex to validate the file path and file name
fileName = args[0];
final String regularExpression = "([\\w\\:\\\\w ./-]+\\w+(\\.)?\\w+)";
Pattern pattern = Pattern.compile(regularExpression);
boolean isMatched = pattern.matcher(fileName).matches();
I want to make the application where I can get all the images no matter whether it is in phone or in external memory. I want to import all that images in my application. How can it be possible? I came to know that it is possible through file connection. But not getting exact idea.
Get all the file system roots using FileSystemRegistry.listRoots()
Open connection to each root in turn using FileConnection fconn = (FileConnection)Connector.open(root)
List the folder using fconn.list().
For each entry in the list, if it ends with an image extension (file.getName().endsWith(".png") etc), then it's an image.
If the entry is a folder (file.isDirectory() returns true) then use fconn.setFileConnection(folder) to traverse into that directory/
Do the same recursively for all folders in all roots.
Here is a code snippet I once used for my application. It more or less does the same in funkybros steps.
protected void showFiles() {
if (path == null) {
Enumeration e = FileSystemRegistry.listRoots();
path = DATAHEAD; //DATAHEAD = file:///
setTitle(path);
while (e.hasMoreElements()) {
String root = (String) e.nextElement();
append(root, null);
}
myForm.getDisplay().setCurrent(this);
} else {
//this if-else just sets the title of the Listing Form
if (selectedItem != null) {
setTitle(path + selectedItem);
}
else {
setTitle(path);
}
try {
// works when users opens a directory, creates a connection to that directory
if (selectedItem != null) {
fileConncetion = (FileConnection) Connector.open(path + selectedItem, Connector.READ);
} else // works when presses 'Back' to go one level above/up
{
fileConncetion = (FileConnection) Connector.open(path, Connector.READ);
}
// Check if the selected item is a directory
if (fileConncetion.isDirectory()) {
if (selectedItem != null) {
path = path + selectedItem;
selectedItem = null;
}
//gathers the directory elements
Enumeration files = fileConncetion.list();
while (files.hasMoreElements()) {
String file = (String) files.nextElement();
append(file, null);
}
//
myForm.getDisplay().setCurrent(this);
try {
if (fileConncetion != null) {
fileConncetion.close();
fileConncetion = null;
}
} catch (IOException ex) {
ex.printStackTrace();
}
}//if (fileConncetion.isDirectory())
else {
System.out.println(path);
//if it gets a file then calls the publishToServer() method
myForm.publishToServer();
}