Copy files from gen2 ADLS based on acknowledgement file - azure

I am trying to copy data from gen2 ADLS into another ADLS using data factory pipeline.
This pipeline runs daily and copies data only for that particular day. This has been done by providing start and end time in the copy activity.
Somedays the files in the source ADLS will be delayed so that the pipeline will run, but no data will be copied.
In order to track this we have planned to keep an acknowledgment file after data copy into source ADLS, so that before copying we could check for the ack file and proceed data copy only if ack file is present.
So the check should happen every 10 mins If ack file is not present, this check should run after 10 mins and this should continue for 2 hrs.
Within this 2 hrs, if file is present then the data copy should proceed and check task also should be stopped.
If there is no data after 2 hrs then the job should fail.
I was trying with validation task in ADF. But one issue is with the folder name since my folder will be named with data and timestamp of creation (for eg: 2021-03-30-02-19-33).
I have to exclude the timestamp part of folder while providing the folder name.
How is it possible. Is wildcard path accepted for validation activity?
Any leads how to implement this?
Is there any way to implement continuous check after 10 mins for 2 hrs in the get matadata task? Can we implement above scenario with get metadata task?

If we do have a requirement for using wildcard path, we have to write Scala/Python.... script on a notebook file and to execute from ADF.
I have used below scala script which takes input parameters from ADF.
import java.io.File
import java.util.Calendar
dbutils.widgets.text("mainFolderPath", "","")
dbutils.widgets.text("finalFolderStartName", "","")
dbutils.widgets.text("fileName", "","")
dbutils.widgets.text("noOfTry", "1","")
val mainFolderPath = dbutils.widgets.get("mainFolderPath")
val finalFolderStartName = dbutils.widgets.get("finalFolderStartName")
val fileName = dbutils.widgets.get("fileName")
val noOfTry = (dbutils.widgets.get("noOfTry")).toInt
println("Main folder path : " + mainFolderPath)
println("Final folder start name : " + finalFolderStartName)
println("File name to be checked : " + fileName)
println("Number of tries with a gap of 1 mins : " + noOfTry)
if(mainFolderPath == "" || finalFolderStartName == "" || fileName == ""){
dbutils.notebook.exit("Please pass input parameters and rerun!")
}
def getListOfSubDirectories(directoryName: String): Array[String] = {
(new File(directoryName))
.listFiles
.filter(_.isDirectory)
.map(_.getName)
}
var counter = 0
var folderFound = false
var fileFound = false
try{
while (counter < noOfTry && !fileFound) {
val folders = getListOfSubDirectories(mainFolderPath)
if(folders.exists(firstName => firstName.startsWith(finalFolderStartName))){
folders.foreach(fol => {
if(fol.startsWith(finalFolderStartName)){
val finalPath = mainFolderPath + "/" + fol + "/" + fileName
println("Final file path : " + finalPath)
folderFound = true
if(new File(finalPath).exists) {
fileFound = true
}else{
println("found the final folder but no file found!")
println("waiting for 10 mins! " + Calendar.getInstance().getTime())
counter = counter+1
Thread.sleep(1*60*1000)
}
}
})
}else{
println("folder does not exists with name : " + mainFolderPath + "/" + finalFolderStartName + "*")
println("waiting for 10 mins! " + Calendar.getInstance().getTime())
counter = counter+1
Thread.sleep(1*60*1000)
}
}
}catch{
case e : Throwable => throw e;
}
if(folderFound && fileFound){
println("File Exists!")
}else{
throw new Exception("File does not exists!");
}

As far as I know, ADF validation and metadata activities does not support wildcard path for the folder/file path.

Related

How to get artifact file's URI using Artifactory's checksum API where multiple artifacts have same SHA-1 / SHA-256 values aka file's content

Artifactory Version: 5.8.4
In Artifactory, files are stored in the internal database via file's checksum (SHA1) and for retrieval purposes, SHA-256 is useful (for verifying if file is intact).
Read this first: https://www.jfrog.com/confluence/display/RTF/Checksum-Based+Storage
Let's say there are 2 Jenkins jobs, which creates few artifacts/file (rpm/jar/etc). In my case, I'll take a simple .txt file which stores date in MM/DD/YYYY format and some other jobA/B specific build result files (jars/rpms etc).
If we focus only on the text file (as I mentioned above), then:
Jenkins_jobA > generates jobA.date_mm_dd_yy.txt
Jenkins_jobA > generates jobB.date_mm_dd_yy.txt
Jenkins jobA and jobB run multiple times per day in no given run order. Sometime jobA runs first and sometime jobB.
As the content of the file are mostly same for both jobs (per day), the SHA-1 value on jobA's .txt file and jobB.txt file will be same i.e. in Artifactory, both files will be stored in the first 2 character based directry folder structure (as per the Check-sum based storage mechanism).
Basically running sha1sum and sha256sum on both files in Linux, would return the exact same output.
Over the time, these artifacts (.txt, etc) gets promoted from one repository to another (promotion process i.e. from snapshot -> stage -> release repo) so my current logic written in Groovy is to find the URI of the artifact sitting behind a "VIRTUAL" repository (which contains a set of physical local repositories in some order) is listed below:
// Groovy code
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
jsonSlurper = new JsonSlurper()
// The following function will take artifact.SHA_256 as it's parameter to find URI of the artifact
def checkSumBasedSearch(artifactSha) {
virt_repo = "jar-repo" // this virtual may have many physical repos release/stage/snapshot for jar(maven) or it can be a YUM repo for (rpm) or generic repo for (.txt file)
// Note: Virtual repos don't span different repo types (i.e. a virtual repository in Artifactory for "Maven" artifacts (jar/war/etc) can NOT see YUM/PyPi/Generic physical Repos).
// Run aqlCmd on Linux, requires "...", "..", "..." for every distinctive words / characters in the cmd line.
checkSum_URL = artifactoryURL + "/api/search/checksum?sha256="
aqlCmd = ["curl", "-u", username + ":" + password, "${checkSum_URL}" + artifactSha + "&repos=" + virt_repo]
}
def procedure = aqlCmd.execute()
def standardOut = new StringBuilder(), standardErr = new StringBuilder()
procedure.waitForProcessOutput(standardOut, standardErr)
// Fail early
if (! standardErr ) {
println "\n\n-- checkSumBasedSearch() - standardErr exists ---\n" + standardErr +"\n\n-- Exiting with error 12!!!\n\n"
System.exit(12)
}
def obj = jsonSlurper.parseText(standardOut.toString())
def results = obj.results
def uri = results[0].uri // This would work, if a file's sha-1 /256 is always different or sits in different repo at least.
return uri
// to get the URL, I can use:
//aqlCmd = ["curl", "-u", username + ":" + password, "${uri}"]
//def procedure = aqlCmd.execute()
//def obj = jsonSlurper.parseText(standardOut.toString())
//def url = obj.downloadUri
//return url
//aqlCmd = [ "curl", "-u", username + ":" + password, "${url}", "-o", somedirectory + "/" + variableContainingSomeArtifactFilenameThatIWant ]
//
// def procedure = aqlCmd.execute()
//def standardOut = new StringBuilder(), standardErr = new StringBuilder()
//procedure.waitForProcessOutput(standardOut, standardErr)
// Now, I'll get the artifact downloaded in some Directory as some Filename.
}
My concern is, as both files (even though different name -or file-<versioned-timestamp>.txt) have same content in them and generated multiple times per day, how can I get a specific versioned file downloaded for jobA or jobB?
In Artifactory, the SHA_256 property for all files containing same content will be same!! (Artifactory will use SHA-1 for storing these files efficiently to save space, new uploads will be just minimal database level transactions transparent to the user).
Questions:
Will the above logic return jobA's file or jobB's .txt file or any Job's .txt file which uploaded it's file first or latest/acc. to LastModified -aka- last upload time?
How can I get jobA's .txt file and jobB's .txt file downloaded for a given timestamp?
Do I need to add more properties during my rest Api call?
If I was just concerned for the file content, then it doesn't matter much (sha-1/256 dependent) whether it's coming from JobA .txt or job's .txt file, but in a complex case, one may have file name containing meaningful info that they'd like to know to find which file was download (A / B)!
You can use AQL (Artifactory Query Langueage)
https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language
curl -u<username>:<password> -XPOST https://repo.jfrog.io/artifactory/api/search/aql -H "Content-Type: text/plain" -T ./search
The content of the file named search is:
items.find(
{
"artifact.module.build.name":{"$eq":"<build name>"},
"artifact.sha1":"<sha1>"
}
)
The above logic (in the original question) will return one of them arbitrary, since you are taking the first result returned and there is no guarantee on the order.
Since your text file contains the timestamp in the name, then you can add the name to the aql given above, it will also filter by the name.
AQL search API is more flexible than the checksum search, use it and customise your query according to the parameters you need.
So, I ended up doing this instead of just returning [0]th element from array in every case.
// Do NOT return [0] first element as yet as Artifactory uses SHA-1/256 so return [Nth].uri where artifact's full name matches with the sha256
// def uri = results[0].uri
def nThIndex=0
def foundFlag = 'false'
for (r in results) {
println "> " + r.uri + " < " + r.uri.toString() + " artifact: " + artFullName
if ( r.uri.toString().contains(artFullName) ) {
foundFlag = 'true'
println "- OK - Found artifact: " + artFullName + " at results[" + nThIndex + "] index."
break; // i.e. a match for the artifact name with SHA-256 we want - has been found.
} else {
nThIndex++;
}
}
if ( foundFlag == 'true' ) {
def uri = results[nThIndex].uri
return uri
} else {
// Fail early if results were found based on SHA256 but not for the artifact but for some other filename with same SHA256
if (! standardErr ) {
println "\n\n\n\n-- [Cool] -- checkSum_Search() - SHA-256 unwanted situation occurred !!! -- results Array was set with some values BUT it didn't contain the artifact (" + artFullName + ") that we were looking for \n\n\n-- !!! Artifact NOT FOUND in the results array during checkSum_Search()---\n\n\n-- Exiting with error 17!!!\n\n\n\n"
System.exit(17) // Nooka
}
}

S3 doesObjectExist API is not recognizing path with equal symbol

I have a Apache Spark application which is writing to S3 folder, since the Spark application is partitioning the data while writing to S3 it has adding EQUAL symbol like below.
s3://biops/XXX/YYY/entryDateYear=2018/entryDateMonth=07
I totally understand that S3 does not allow to create bucket_name with "=" , but spark streaming creates partition with field_name followed by "=" and then value.
Please advise on how to access S3 folder with equal sign.
// Actual paths in S3 is --> biops/XXX/YYY/royalty_raw_json/entryDateYear=2018/
String bucket = "biops";
String without_equal = "XXX/YYY/royalty_raw_json/";
String with_equal = "XXX/YYY/royalty_raw_json/entryDateYear=2018";
String with_equal_encoding = "XXX/YYY/royalty_raw_json/entryDateYear%3D2018";
AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().
withCredentials(getCredentialsProvider(credentials))
.withRegion("us-east-1")
.build();
amazonS3.doesObjectExist(bucket, without_equal); // Works
amazonS3.doesObjectExist(bucket, with_equal); // Not works
amazonS3.doesObjectExist(bucket, with_equal_encoding); // Not works 2
UPDATE
I managed with a work around to list the objects as below to check if a folder is present are not
ListObjectsRequest listObjectsRequest = new ListObjectsRequest().withBucketName(bucket).withPrefix(with_equal);
ObjectListing bucketListing = amazonS3.listObjects(listObjectsRequest);
if (bucketListing != null && bucketListing.getObjectSummaries() != null && bucketListing.getObjectSummaries().size() > 0)
System.out.println("Folder present with files");
else
System.out.println("Folder present with zero files or Folder not present");

Spark Streaming textFileStream not supporting wildcards

I setup a simple test to stream text files from S3 and got it to work when I tried something like
val input = ssc.textFileStream("s3n://mybucket/2015/04/03/")
and in the bucket I would have log files go in there and everything would work fine.
But if their was a subfolder, it would not find any files that got put into the subfolder (and yes, I am aware that hdfs doesn't actually use a folder structure)
val input = ssc.textFileStream("s3n://mybucket/2015/04/")
So, I tried to simply do wildcards like I have done before with a standard spark application
val input = ssc.textFileStream("s3n://mybucket/2015/04/*")
But when I try this it throws an error
java.io.FileNotFoundException: File s3n://mybucket/2015/04/* does not exist.
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:506)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1483)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1523)
at org.apache.spark.streaming.dstream.FileInputDStream.findNewFiles(FileInputDStream.scala:176)
at org.apache.spark.streaming.dstream.FileInputDStream.compute(FileInputDStream.scala:134)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1$$anonfun$1.apply(DStream.scala:300)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:299)
at org.apache.spark.streaming.dstream.DStream$$anonfun$getOrCompute$1.apply(DStream.scala:287)
at scala.Option.orElse(Option.scala:257)
.....
I know for a fact that you can use wildcards when reading fileInput for a standard spark applications but it appears that when doing streaming input, it doesn't do that nor does it automatically process files in subfolders. Is there something I'm missing here??
Ultimately what I need is a streaming job to be running 24/7 that will be monitoring an S3 bucket that has logs placed in it by date
So something like
s3n://mybucket/<YEAR>/<MONTH>/<DAY>/<LogfileName>
Is there any way to hand it the top most folder and it automatically read files that show up in any folder (cause obviously the date will increase every day)?
EDIT
So upon digging into the documentation at http://spark.apache.org/docs/latest/streaming-programming-guide.html#basic-sources it states that nested directories are not supported.
Can anyone shed some light as to why this is the case?
Also, since my files will be nested based upon their date, what would be a good way of solving this problem in my streaming application? It's a little complicated since the logs take a few minutes to get written to S3 and so the last file being written for the day could be written in the previous day's folder even though we're a few minutes into the new day.
Some "ugly but working solution" can be created by extending FileInputDStream.
Writing sc.textFileStream(d) is equivalent to
new FileInputDStream[LongWritable, Text, TextInputFormat](streamingContext, d).map(_._2.toString)
You can create CustomFileInputDStream that will extend FileInputDStream. The custom class will copy the compute method from the FileInputDStream class and adjust the findNewFiles method to your needs.
changing findNewFiles method from:
private def findNewFiles(currentTime: Long): Array[String] = {
try {
lastNewFileFindingTime = clock.getTimeMillis()
// Calculate ignore threshold
val modTimeIgnoreThreshold = math.max(
initialModTimeIgnoreThreshold, // initial threshold based on newFilesOnly setting
currentTime - durationToRemember.milliseconds // trailing end of the remember window
)
logDebug(s"Getting new files for time $currentTime, " +
s"ignoring files older than $modTimeIgnoreThreshold")
val filter = new PathFilter {
def accept(path: Path): Boolean = isNewFile(path, currentTime, modTimeIgnoreThreshold)
}
val newFiles = fs.listStatus(directoryPath, filter).map(_.getPath.toString)
val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
logInfo("Finding new files took " + timeTaken + " ms")
logDebug("# cached file times = " + fileToModTime.size)
if (timeTaken > slideDuration.milliseconds) {
logWarning(
"Time taken to find new files exceeds the batch size. " +
"Consider increasing the batch size or reducing the number of " +
"files in the monitored directory."
)
}
newFiles
} catch {
case e: Exception =>
logWarning("Error finding new files", e)
reset()
Array.empty
}
}
to:
private def findNewFiles(currentTime: Long): Array[String] = {
try {
lastNewFileFindingTime = clock.getTimeMillis()
// Calculate ignore threshold
val modTimeIgnoreThreshold = math.max(
initialModTimeIgnoreThreshold, // initial threshold based on newFilesOnly setting
currentTime - durationToRemember.milliseconds // trailing end of the remember window
)
logDebug(s"Getting new files for time $currentTime, " +
s"ignoring files older than $modTimeIgnoreThreshold")
val filter = new PathFilter {
def accept(path: Path): Boolean = isNewFile(path, currentTime, modTimeIgnoreThreshold)
}
val directories = fs.listStatus(directoryPath).filter(_.isDirectory)
val newFiles = ArrayBuffer[FileStatus]()
directories.foreach(directory => newFiles.append(fs.listStatus(directory.getPath, filter) : _*))
val timeTaken = clock.getTimeMillis() - lastNewFileFindingTime
logInfo("Finding new files took " + timeTaken + " ms")
logDebug("# cached file times = " + fileToModTime.size)
if (timeTaken > slideDuration.milliseconds) {
logWarning(
"Time taken to find new files exceeds the batch size. " +
"Consider increasing the batch size or reducing the number of " +
"files in the monitored directory."
)
}
newFiles.map(_.getPath.toString).toArray
} catch {
case e: Exception =>
logWarning("Error finding new files", e)
reset()
Array.empty
}
}
will check for files in all first degree sub folders, you can adjust it to use the batch timestamp in order to access the relevant "subdirectories".
I created the CustomFileInputDStream as I mentioned and activated it by calling:
new CustomFileInputDStream[LongWritable, Text, TextInputFormat](streamingContext, d).map(_._2.toString)
It seems to behave us expected.
When I write solution like this I must add some points for consideration:
You are breaking Spark encapsulation and creating a custom class that you would have to support solely as time pass.
I believe that solution like this is the last resort. If your use case can be implemented by different way, it is usually better to avoid solution like this.
If you will have a lot of "subdirectories" on S3 and would check each one of them it will cost you.
It will be very interesting to understand if Databricks doesn't support nested files just because of possible performance penalty or not, maybe there is a deeper reason I haven't thought about.
we had same problem. we joined sub folder names with comma.
List<String> paths = new ArrayList<>();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd");
try {
Date start = sdf.parse("2015/02/01");
Date end = sdf.parse("2015/04/01");
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (calendar.getTime().before(end)) {
paths.add("s3n://mybucket/" + sdf.format(calendar.getTime()));
calendar.add(Calendar.DATE, 1);
}
} catch (ParseException e) {
e.printStackTrace();
}
String joinedPaths = StringUtils.join(",", paths.toArray(new String[paths.size()]));
val input = ssc.textFileStream(joinedPaths);
I hope that in this way your problem is solved.

Groovy script to read multiple xml files one at a time from 2 folders and apply xmlunit comparison methods

I have series of xml files placed in 2 separate folders as below. My objective is to read each file one at a time from both folders and apply xmlunit comaprison methods.
Folder1 : actual1.xml
actual2.xml
actual3.xml
Folder2 : compare1.xml
compare2.xml
compare3.xml
Part1: Am reading each file at a time from both folders by using below script. I welcome suggestions if there are more simpler methods to do this
log.info "**********Read files from Folder1************"
def xml_file1 = []
new File("D:\\GroovyTest\\Folder1").eachFile{ f ->
f (f.isFile()&& f.name.contains('.xml'))
{
def filename = f.name[0..-1]
xml_file1.add(filename)
log.info filename
}
}
if (xml_file1.size() <1)
{
testRunner.fail("No request files found")
}
log.info "**********Read files from Folder2************"
def xml_file2 = []
new File("D:\\GroovyTest\\Folder2").eachFile{ f ->
if (f.isFile()&& f.name.contains('.xml'))
{
def filename = f.name[0..-1]
xml_file2.add(filename)
log.info filename
}
}
if (xml_file2.size() <1)
{
testRunner.fail("No request files found")
}
Part2: Script to perform comparison for each combination of xml files contained in array xml_file1 and xml_file2.
Am actually stuck at this part as the below script works for single files if each xml file is kept in a string, but i have to pass an array as arguments since i have series of xml files to be compared. I get a run time error - groovy.lang.GroovyRuntimeException: Could not find matching constructor for: java.io.FileInputStream(java.util.ArrayList) error at line: 60
InputStream xml_stream1 = new FileInputStream(xml_file1)
String xml1 = getStringFromInputStream(xml_stream1)
InputStream xml_stream2 = new FileInputStream(xml_file2)
String xml2 = getStringFromInputStream(xml_stream2)
def factory = TransformerFactory.newInstance()
def transformer = factory.newTransformer(new StreamSource(new StringReader(xslt)))
StreamResult result_xml1 = new StreamResult(new StringWriter());
transformer.transform(new StreamSource(new StringReader(xml1)), result_xml1)
xml1 = result_xml1.getWriter().toString()
StreamResult result_xml2 = new StreamResult(newStringWriter());
transformer.transform(new StreamSource(new StringReader(xml2)), result_xml2)
xml2 = result_xml2.getWriter().toString()
XMLUnit.setIgnoreComments(true)
DifferenceListener differenceListener = newIgnoreTextAndAttributeValuesDifferenceListener();
DetailedDiff myDiff = new DetailedDiff(new Diff(xml1, xml2));
myDiff.overrideDifferenceListener(differenceListener);
myDiff.overrideElementQualifier(new RecursiveElementNameAndTextQualifier());
log.info "similar ? " + myDiff.similar()
log.info "identical ? " + myDiff.identical()
List allDifferences = myDiff.getAllDifferences();
for (Object object : allDifferences) {
Difference difference = (Difference)object;
log.info(difference);
}
Could someone also help me with methods to ignore empty tags during comparison?
Thanks

Pick files with certain file names in c#

I have a folder with around 200,000 jpeg images. Below are the format of fileNames I can find
BATCHID_GROUPID ex: 501234_20123.jpg
BATCHID_GROUPID ex. 501235_20124_1.jpg, 501235_20124_2.jpg, 501235_20124_3.jpg. Each of this type images will have max of 10 images of BATCHID_GROUPID. What I mean is for this set the max will be 501235_20124_10.jpg
I need to take all the images that don't end with _x or _xx i.e _1.jpg or _2.jpg or _10.jpg and pick the BATCHID and copy it and FTP it to a different location
For the ones that end with _x or _xx I need to pick BATCHID and create a folder with the name as batchID and move all the files that end with _X to _xx into the folder.
Thanks
Try using String.Split on your files names, check for length to determine where you are going to copy it and take the first index as your BATCHID:
Something like this:
DirectoryInfo di = new DirectoryInfo("SourcePath");
IEnumerable<FileInfo> fileinfo = di.EnumerateFiles();
foreach(FileInfo fi in fileinfo)
{
string[] tmp = fi.Name.Split('_');
if (tmp.Length == 3)
{
if (!Directory.Exists("YourPath"))
{
Directory.CreateDirectory("YourPath" + tmp[0].ToString());
fi.MoveTo("YourPath" + tmp[0].ToString() + #"\" + fi.Name);
}
else
fi.MoveTo("YourPath" + + tmp[0].ToString() + #"\" + fi.Name);
}
else if (tmp.Length == 2)
{
//Copy Batch Id and Ftp logic
}
}

Resources