I have a folder with around 200,000 jpeg images. Below are the format of fileNames I can find
BATCHID_GROUPID ex: 501234_20123.jpg
BATCHID_GROUPID ex. 501235_20124_1.jpg, 501235_20124_2.jpg, 501235_20124_3.jpg. Each of this type images will have max of 10 images of BATCHID_GROUPID. What I mean is for this set the max will be 501235_20124_10.jpg
I need to take all the images that don't end with _x or _xx i.e _1.jpg or _2.jpg or _10.jpg and pick the BATCHID and copy it and FTP it to a different location
For the ones that end with _x or _xx I need to pick BATCHID and create a folder with the name as batchID and move all the files that end with _X to _xx into the folder.
Thanks
Try using String.Split on your files names, check for length to determine where you are going to copy it and take the first index as your BATCHID:
Something like this:
DirectoryInfo di = new DirectoryInfo("SourcePath");
IEnumerable<FileInfo> fileinfo = di.EnumerateFiles();
foreach(FileInfo fi in fileinfo)
{
string[] tmp = fi.Name.Split('_');
if (tmp.Length == 3)
{
if (!Directory.Exists("YourPath"))
{
Directory.CreateDirectory("YourPath" + tmp[0].ToString());
fi.MoveTo("YourPath" + tmp[0].ToString() + #"\" + fi.Name);
}
else
fi.MoveTo("YourPath" + + tmp[0].ToString() + #"\" + fi.Name);
}
else if (tmp.Length == 2)
{
//Copy Batch Id and Ftp logic
}
}
Related
I am trying to copy data from gen2 ADLS into another ADLS using data factory pipeline.
This pipeline runs daily and copies data only for that particular day. This has been done by providing start and end time in the copy activity.
Somedays the files in the source ADLS will be delayed so that the pipeline will run, but no data will be copied.
In order to track this we have planned to keep an acknowledgment file after data copy into source ADLS, so that before copying we could check for the ack file and proceed data copy only if ack file is present.
So the check should happen every 10 mins If ack file is not present, this check should run after 10 mins and this should continue for 2 hrs.
Within this 2 hrs, if file is present then the data copy should proceed and check task also should be stopped.
If there is no data after 2 hrs then the job should fail.
I was trying with validation task in ADF. But one issue is with the folder name since my folder will be named with data and timestamp of creation (for eg: 2021-03-30-02-19-33).
I have to exclude the timestamp part of folder while providing the folder name.
How is it possible. Is wildcard path accepted for validation activity?
Any leads how to implement this?
Is there any way to implement continuous check after 10 mins for 2 hrs in the get matadata task? Can we implement above scenario with get metadata task?
If we do have a requirement for using wildcard path, we have to write Scala/Python.... script on a notebook file and to execute from ADF.
I have used below scala script which takes input parameters from ADF.
import java.io.File
import java.util.Calendar
dbutils.widgets.text("mainFolderPath", "","")
dbutils.widgets.text("finalFolderStartName", "","")
dbutils.widgets.text("fileName", "","")
dbutils.widgets.text("noOfTry", "1","")
val mainFolderPath = dbutils.widgets.get("mainFolderPath")
val finalFolderStartName = dbutils.widgets.get("finalFolderStartName")
val fileName = dbutils.widgets.get("fileName")
val noOfTry = (dbutils.widgets.get("noOfTry")).toInt
println("Main folder path : " + mainFolderPath)
println("Final folder start name : " + finalFolderStartName)
println("File name to be checked : " + fileName)
println("Number of tries with a gap of 1 mins : " + noOfTry)
if(mainFolderPath == "" || finalFolderStartName == "" || fileName == ""){
dbutils.notebook.exit("Please pass input parameters and rerun!")
}
def getListOfSubDirectories(directoryName: String): Array[String] = {
(new File(directoryName))
.listFiles
.filter(_.isDirectory)
.map(_.getName)
}
var counter = 0
var folderFound = false
var fileFound = false
try{
while (counter < noOfTry && !fileFound) {
val folders = getListOfSubDirectories(mainFolderPath)
if(folders.exists(firstName => firstName.startsWith(finalFolderStartName))){
folders.foreach(fol => {
if(fol.startsWith(finalFolderStartName)){
val finalPath = mainFolderPath + "/" + fol + "/" + fileName
println("Final file path : " + finalPath)
folderFound = true
if(new File(finalPath).exists) {
fileFound = true
}else{
println("found the final folder but no file found!")
println("waiting for 10 mins! " + Calendar.getInstance().getTime())
counter = counter+1
Thread.sleep(1*60*1000)
}
}
})
}else{
println("folder does not exists with name : " + mainFolderPath + "/" + finalFolderStartName + "*")
println("waiting for 10 mins! " + Calendar.getInstance().getTime())
counter = counter+1
Thread.sleep(1*60*1000)
}
}
}catch{
case e : Throwable => throw e;
}
if(folderFound && fileFound){
println("File Exists!")
}else{
throw new Exception("File does not exists!");
}
As far as I know, ADF validation and metadata activities does not support wildcard path for the folder/file path.
So, I've run into an issue here that has me sort of perplexed. To give you an idea of what I'm trying to accomplish, I'm executing this from After Effects, to get a path for images in a directory, and then write a text file that holds the path for each image as a new line. Currently everything works almost exactly as I want it, and the issue is coming down to how AE lists a file path. I feel like I'm missing something simple, but here's the chunk of code I'm having an issue with:
var saveTextFile = File(savePath + "images.txt");
if(saveTextFile.exists)
saveTextFile.remove();
saveTextFile.encoding = "UTF8";
saveTextFile.open("e", "TEXT", "????");
var files = Folder (savePath).getFiles("*.PNG");
if (files.length == 0) return;
for each (var file in files){
//var drive = '/x';
//var fixName = fileName.replace(drive, 'X:');
//name = fixName.toString();
//$.writeln(name)
saveTextFile.writeln(('file ' + "'" + file.toString() + "'"));
}
saveTextFile.close();
The issue exists in the for each (var file in files) section. If I run it as is, I end up with a path similar to what's listed here:
file '/x/_CURRENT_/sequence_PNG_00000.png'
file '/x/_CURRENT_/sequence_PNG_00001.png'
file '/x/_CURRENT_/sequence_PNG_00002.png'
file '/x/_CURRENT_/sequence_PNG_00003.png'
file '/x/_CURRENT_/sequence_PNG_00004.png'
file '/x/_CURRENT_/sequence_PNG_00005.png'
Now this is great, except for the fact that it's reading the drive letter as "/x". This is problematic, so If I uncomment the variables in the for each loop, I end up with something similar to this:
file 'X:/_CURRENT_/sequence_PNG_00000.png'
file 'X:/_CURRENT_/sequence_PNG_00000.png'
file 'X:/_CURRENT_/sequence_PNG_00000.png'
file 'X:/_CURRENT_/sequence_PNG_00000.png'
file 'X:/_CURRENT_/sequence_PNG_00000.png'
file 'X:/_CURRENT_/sequence_PNG_00000.png'
And so that's great because it formats the X drive properly in the string.... But alas, it renames the incremental number in the string to 00000.png for every image.
Can anyone spot what I might be overlooking?
for each (var file in files){
does not look like valid JS syntax. Also Extendscript is ES3 so there is no [1,2,3].forEach(function(ele,i,arr){}) either.
Try it with
for(var i = 0; i < files.length;i++){
var file = files[i];
// ...and so on
}
I'm not sure if my code is the most efficient here, but it's effective, and at the moment that's all I can ask for.
This is the chunk that ended up working for me.
var saveTextFile = File(savePath + "images.txt");
if(saveTextFile.exists)
saveTextFile.remove();
saveTextFile.encoding = "UTF8";
saveTextFile.open("e", "TEXT", "????");
var files = Folder (savePath).getFiles("*.PNG");
if (files.length == 0) return;
for(var i = 0; i < files.length;i++){
var file = files[i];
var drive = '/x';
var fileIt = file.toString();
var fixName = fileIt.replace(drive, 'X:');
$.writeln(fixName);
saveTextFile.writeln(('file ' + "'" + fixName + "'"));
}
saveTextFile.close();
};
Artifactory Version: 5.8.4
In Artifactory, files are stored in the internal database via file's checksum (SHA1) and for retrieval purposes, SHA-256 is useful (for verifying if file is intact).
Read this first: https://www.jfrog.com/confluence/display/RTF/Checksum-Based+Storage
Let's say there are 2 Jenkins jobs, which creates few artifacts/file (rpm/jar/etc). In my case, I'll take a simple .txt file which stores date in MM/DD/YYYY format and some other jobA/B specific build result files (jars/rpms etc).
If we focus only on the text file (as I mentioned above), then:
Jenkins_jobA > generates jobA.date_mm_dd_yy.txt
Jenkins_jobA > generates jobB.date_mm_dd_yy.txt
Jenkins jobA and jobB run multiple times per day in no given run order. Sometime jobA runs first and sometime jobB.
As the content of the file are mostly same for both jobs (per day), the SHA-1 value on jobA's .txt file and jobB.txt file will be same i.e. in Artifactory, both files will be stored in the first 2 character based directry folder structure (as per the Check-sum based storage mechanism).
Basically running sha1sum and sha256sum on both files in Linux, would return the exact same output.
Over the time, these artifacts (.txt, etc) gets promoted from one repository to another (promotion process i.e. from snapshot -> stage -> release repo) so my current logic written in Groovy is to find the URI of the artifact sitting behind a "VIRTUAL" repository (which contains a set of physical local repositories in some order) is listed below:
// Groovy code
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
jsonSlurper = new JsonSlurper()
// The following function will take artifact.SHA_256 as it's parameter to find URI of the artifact
def checkSumBasedSearch(artifactSha) {
virt_repo = "jar-repo" // this virtual may have many physical repos release/stage/snapshot for jar(maven) or it can be a YUM repo for (rpm) or generic repo for (.txt file)
// Note: Virtual repos don't span different repo types (i.e. a virtual repository in Artifactory for "Maven" artifacts (jar/war/etc) can NOT see YUM/PyPi/Generic physical Repos).
// Run aqlCmd on Linux, requires "...", "..", "..." for every distinctive words / characters in the cmd line.
checkSum_URL = artifactoryURL + "/api/search/checksum?sha256="
aqlCmd = ["curl", "-u", username + ":" + password, "${checkSum_URL}" + artifactSha + "&repos=" + virt_repo]
}
def procedure = aqlCmd.execute()
def standardOut = new StringBuilder(), standardErr = new StringBuilder()
procedure.waitForProcessOutput(standardOut, standardErr)
// Fail early
if (! standardErr ) {
println "\n\n-- checkSumBasedSearch() - standardErr exists ---\n" + standardErr +"\n\n-- Exiting with error 12!!!\n\n"
System.exit(12)
}
def obj = jsonSlurper.parseText(standardOut.toString())
def results = obj.results
def uri = results[0].uri // This would work, if a file's sha-1 /256 is always different or sits in different repo at least.
return uri
// to get the URL, I can use:
//aqlCmd = ["curl", "-u", username + ":" + password, "${uri}"]
//def procedure = aqlCmd.execute()
//def obj = jsonSlurper.parseText(standardOut.toString())
//def url = obj.downloadUri
//return url
//aqlCmd = [ "curl", "-u", username + ":" + password, "${url}", "-o", somedirectory + "/" + variableContainingSomeArtifactFilenameThatIWant ]
//
// def procedure = aqlCmd.execute()
//def standardOut = new StringBuilder(), standardErr = new StringBuilder()
//procedure.waitForProcessOutput(standardOut, standardErr)
// Now, I'll get the artifact downloaded in some Directory as some Filename.
}
My concern is, as both files (even though different name -or file-<versioned-timestamp>.txt) have same content in them and generated multiple times per day, how can I get a specific versioned file downloaded for jobA or jobB?
In Artifactory, the SHA_256 property for all files containing same content will be same!! (Artifactory will use SHA-1 for storing these files efficiently to save space, new uploads will be just minimal database level transactions transparent to the user).
Questions:
Will the above logic return jobA's file or jobB's .txt file or any Job's .txt file which uploaded it's file first or latest/acc. to LastModified -aka- last upload time?
How can I get jobA's .txt file and jobB's .txt file downloaded for a given timestamp?
Do I need to add more properties during my rest Api call?
If I was just concerned for the file content, then it doesn't matter much (sha-1/256 dependent) whether it's coming from JobA .txt or job's .txt file, but in a complex case, one may have file name containing meaningful info that they'd like to know to find which file was download (A / B)!
You can use AQL (Artifactory Query Langueage)
https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language
curl -u<username>:<password> -XPOST https://repo.jfrog.io/artifactory/api/search/aql -H "Content-Type: text/plain" -T ./search
The content of the file named search is:
items.find(
{
"artifact.module.build.name":{"$eq":"<build name>"},
"artifact.sha1":"<sha1>"
}
)
The above logic (in the original question) will return one of them arbitrary, since you are taking the first result returned and there is no guarantee on the order.
Since your text file contains the timestamp in the name, then you can add the name to the aql given above, it will also filter by the name.
AQL search API is more flexible than the checksum search, use it and customise your query according to the parameters you need.
So, I ended up doing this instead of just returning [0]th element from array in every case.
// Do NOT return [0] first element as yet as Artifactory uses SHA-1/256 so return [Nth].uri where artifact's full name matches with the sha256
// def uri = results[0].uri
def nThIndex=0
def foundFlag = 'false'
for (r in results) {
println "> " + r.uri + " < " + r.uri.toString() + " artifact: " + artFullName
if ( r.uri.toString().contains(artFullName) ) {
foundFlag = 'true'
println "- OK - Found artifact: " + artFullName + " at results[" + nThIndex + "] index."
break; // i.e. a match for the artifact name with SHA-256 we want - has been found.
} else {
nThIndex++;
}
}
if ( foundFlag == 'true' ) {
def uri = results[nThIndex].uri
return uri
} else {
// Fail early if results were found based on SHA256 but not for the artifact but for some other filename with same SHA256
if (! standardErr ) {
println "\n\n\n\n-- [Cool] -- checkSum_Search() - SHA-256 unwanted situation occurred !!! -- results Array was set with some values BUT it didn't contain the artifact (" + artFullName + ") that we were looking for \n\n\n-- !!! Artifact NOT FOUND in the results array during checkSum_Search()---\n\n\n-- Exiting with error 17!!!\n\n\n\n"
System.exit(17) // Nooka
}
}
I setup a simple test to stream text files from S3 and got it to work when I tried something like
val input = ssc.textFileStream("s3n://mybucket/2015/04/03/")
and in the bucket I would have log files go in there and everything would work fine.
But if their was a subfolder, it would not find any files that got put into the subfolder (and yes, I am aware that hdfs doesn't actually use a folder structure)
val input = ssc.textFileStream("s3n://mybucket/2015/04/")
So, I tried to simply do wildcards like I have done before with a standard spark application
val input = ssc.textFileStream("s3n://mybucket/2015/04/*")
But when I try this it throws an error
java.io.FileNotFoundException: File s3n://mybucket/2015/04/* does not exist.
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:506)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1483)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1523)
at org.apache.spark.streaming.dstream.FileInputDStream.findNewFiles(FileInputDStream.scala:176)
at org.apache.spark.streaming.dstream.FileInputDStream.compute(FileInputDStream.scala:134)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
at scala.Option.orElse(Option.scala:257)
.....
I know for a fact that you can use wildcards when reading fileInput for a standard spark applications but it appears that when doing streaming input, it doesn't do that nor does it automatically process files in subfolders. Is there something I'm missing here??
Ultimately what I need is a streaming job to be running 24/7 that will be monitoring an S3 bucket that has logs placed in it by date
So something like
s3n://mybucket/<YEAR>/<MONTH>/<DAY>/<LogfileName>
Is there any way to hand it the top most folder and it automatically read files that show up in any folder (cause obviously the date will increase every day)?
EDIT
So upon digging into the documentation at http://spark.apache.org/docs/latest/streaming-programming-guide.html#basic-sources it states that nested directories are not supported.
Can anyone shed some light as to why this is the case?
Also, since my files will be nested based upon their date, what would be a good way of solving this problem in my streaming application? It's a little complicated since the logs take a few minutes to get written to S3 and so the last file being written for the day could be written in the previous day's folder even though we're a few minutes into the new day.
Some "ugly but working solution" can be created by extending FileInputDStream.
Writing sc.textFileStream(d) is equivalent to
new FileInputDStream[LongWritable, Text, TextInputFormat](streamingContext, d).map(_._2.toString)
You can create CustomFileInputDStream that will extend FileInputDStream. The custom class will copy the compute method from the FileInputDStream class and adjust the findNewFiles method to your needs.
changing findNewFiles method from:
private def findNewFiles(currentTime: Long): Array[String] = {
try {
lastNewFileFindingTime = clock.getTimeMillis()
// Calculate ignore threshold
val modTimeIgnoreThreshold = math.max(
initialModTimeIgnoreThreshold, // initial threshold based on newFilesOnly setting
currentTime - durationToRemember.milliseconds // trailing end of the remember window
)
logDebug(s"Getting new files for time $currentTime, " +
s"ignoring files older than $modTimeIgnoreThreshold")
val filter = new PathFilter {
def accept(path: Path): Boolean = isNewFile(path, currentTime, modTimeIgnoreThreshold)
}
val newFiles = fs.listStatus(directoryPath, filter).map(_.getPath.toString)
val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
logInfo("Finding new files took " + timeTaken + " ms")
logDebug("# cached file times = " + fileToModTime.size)
if (timeTaken > slideDuration.milliseconds) {
logWarning(
"Time taken to find new files exceeds the batch size. " +
"Consider increasing the batch size or reducing the number of " +
"files in the monitored directory."
)
}
newFiles
} catch {
case e: Exception =>
logWarning("Error finding new files", e)
reset()
Array.empty
}
}
to:
private def findNewFiles(currentTime: Long): Array[String] = {
try {
lastNewFileFindingTime = clock.getTimeMillis()
// Calculate ignore threshold
val modTimeIgnoreThreshold = math.max(
initialModTimeIgnoreThreshold, // initial threshold based on newFilesOnly setting
currentTime - durationToRemember.milliseconds // trailing end of the remember window
)
logDebug(s"Getting new files for time $currentTime, " +
s"ignoring files older than $modTimeIgnoreThreshold")
val filter = new PathFilter {
def accept(path: Path): Boolean = isNewFile(path, currentTime, modTimeIgnoreThreshold)
}
val directories = fs.listStatus(directoryPath).filter(_.isDirectory)
val newFiles = ArrayBuffer[FileStatus]()
directories.foreach(directory => newFiles.append(fs.listStatus(directory.getPath, filter) : _*))
val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
logInfo("Finding new files took " + timeTaken + " ms")
logDebug("# cached file times = " + fileToModTime.size)
if (timeTaken > slideDuration.milliseconds) {
logWarning(
"Time taken to find new files exceeds the batch size. " +
"Consider increasing the batch size or reducing the number of " +
"files in the monitored directory."
)
}
newFiles.map(_.getPath.toString).toArray
} catch {
case e: Exception =>
logWarning("Error finding new files", e)
reset()
Array.empty
}
}
will check for files in all first degree sub folders, you can adjust it to use the batch timestamp in order to access the relevant "subdirectories".
I created the CustomFileInputDStream as I mentioned and activated it by calling:
new CustomFileInputDStream[LongWritable, Text, TextInputFormat](streamingContext, d).map(_._2.toString)
It seems to behave us expected.
When I write solution like this I must add some points for consideration:
You are breaking Spark encapsulation and creating a custom class that you would have to support solely as time pass.
I believe that solution like this is the last resort. If your use case can be implemented by different way, it is usually better to avoid solution like this.
If you will have a lot of "subdirectories" on S3 and would check each one of them it will cost you.
It will be very interesting to understand if Databricks doesn't support nested files just because of possible performance penalty or not, maybe there is a deeper reason I haven't thought about.
we had same problem. we joined sub folder names with comma.
List<String> paths = new ArrayList<>();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd");
try {
Date start = sdf.parse("2015/02/01");
Date end = sdf.parse("2015/04/01");
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (calendar.getTime().before(end)) {
paths.add("s3n://mybucket/" + sdf.format(calendar.getTime()));
calendar.add(Calendar.DATE, 1);
}
} catch (ParseException e) {
e.printStackTrace();
}
String joinedPaths = StringUtils.join(",", paths.toArray(new String[paths.size()]));
val input = ssc.textFileStream(joinedPaths);
I hope that in this way your problem is solved.
I have to zip multiple files together using 7zip.exe. I have paths of two files say file1 and file2. I append the two paths using the following.
string filetozip = file1+ "\"" + file2+" "; and do the below
Process proc = new Process();
proc.StartInfo.FileName = #"C:\Freedom\7-Zip\7z.exe";
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.RedirectStandardError = true;
proc.StartInfo.RedirectStandardInput = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.StartInfo.Arguments = string.Format(" a -tzip \"{0}\" \"{1}\" -mx=9 -mem=AES256 -p\"{2}\" ", destZipFile, filetozip , zipPassword);
proc.Start();
proc.WaitForExit();
if (proc.ExitCode != 0)
{
throw new Exception("Error Zipping Data File : " + proc.StandardError.ReadToEnd());
}
filetozip is passed as an argument above. The above code does not work properly. I am getting proc.ExitCode=1. Which is the right way to append the file paths.Is string filetozip = file1+ "\"" + file2+" "; the right way? I can have one or more files. What is the separator used?
The command line that you want to create looks like
plus the required switches (arguments quoted and space delimited).
String.Join or StringBuilder are some coding things that may be helpful