I would like to know how I can load some files in a specific order. For instance, I would like to load my files according to their timestamp, in order to make sure that subsequent data updates are replayed in the proper order.
Lets say I have 2 types of files : deal info files and risk files.
I would like to load T1_Info.csv, then T1_Risk.csv, T2_Info.csv, T2_Risk.csv...
I have tried to implement a comparator, as it is said on Confluence, but it seems that the loadInstructions file has the priority. It will order the Info files and the risk files independently. (loading T1_Info.csv, T2_Info.csv and then T1_Risk.csv, T2_Risk.csv..)
Do I have to implement a custom file loader, or is it possible using an AP configuration ?
The loading of the files based on load instructions is done in
com.quartetfs.tech.store.csv.impl.CSVDataModelFactory.load(List<FileLoadDescriptor>). The FileLoadDescriptor list you receive is created directly from the load instructions files.
What you can do is create a simple instructions files with 2 entries, one for deal info and one for risk. So your custom implementation of CSVDataModelFactory will be called with a list of two items. In your custom implementation you scan the directory where the files are, sort them in the order you want them to be parsed and call the super.load() with the list of FileLoadDescriptor you created from the directory scanning.
If you want to also load files that are place in the future in this folder you have to add to your load instructions a line that will match all files and that will make the super.load() implementation to create a directory watcher for that (you should then maybe override createDirectoryWatcher() to not watch the files already present in the folder when load is called).
Related
Typically, we have a depot root for every different product that we work on. For e.g.:
//products/productX
/productY
As the common files in the 2 products increase, I would like to put them into a top level folder of it's own
//products/productX
/productY
/common
Now to ensure that this works for all the users who have existing workspace, we would need to update all their workspaces. Is there an alternative? Can we put some markers in the depot to create a link it to a different folder? Any other option?
What you're describing is essentially the reason that streams were created -- the idea of a stream is that you definition the structure of a codeline in one place (e.g. "product X lives in //products/productX"), multiple people base their workspaces on that, and when you change it (e.g. "product X lives in //products/productX + //products/common), every workspace based on that definition updates automatically.
So if you're using streams, all you need to do is update the stream definitions that need to include the new //products/common directory. Easy!
If you're using "classic" workspaces, users who are using the default //products/... mapping will get the common directory automatically regardless. For users with custom views, my suggestion would be to alert them of the refactor and then let them make their own adjustments as appropriate; if they're familiar enough with Perforce to have built a custom client view, they may not appreciate having it changed underneath them.
Previously I had a problem on how to merge several JSON files into one single file,
which I was able to resolve it with the answer of this question.
At first, I tried with just some files by using wild cards in the file name in the connection section of the input dataset. But when I remove the file name, theory tells me that all of the files in all folders would be loaded recursively as I checked the copy recursively option, in the source section of the copy activity.
The problem is that when I manually trigger the pipeline after removing the file name from the input of the data set, only some of the files get loaded and the task ends successfully but only loading around 400+ files, each folder has 1M+ files, I want to create BIG csv files by merging all the small JSON files of the source (I already was able to create csv file by mapping the schemas in the copy activity).
It is probably stopping due to a timeout or out of memory exception.
One solution is to loop over the contents of the directory using
Directory.EnumerateFiles(searchDir)
This way you can process all the files without having the list / contents of all files in memory at the same time.
I'm working on a project and we want to handle our logging using log4j. I am running into some issues that I am not able to easily resolve looking at the log4j docs, or other documentation online.
I get the basic idea of putting logging code throughout the codebase and then having the properties file assort the logged data into a hierarchy of appenders and how to write out to a file. That's fine. This basically allows me to create greppable log files in one hard coded folder, such as this:
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log
But I have two basic questions: I want to have the log location be dynamic, such as:
log4j.appender.R.File={$processDir}/example.log
Also, every time the user runs this app, a folder will be created with the output files. I would like to have the log file be placed there, and I'm not sure how to accomplish that.
The other issue (although I think this will be a lot easier once the first issue is addressed...) is about creating a formatted log that does not necessarily reflect the process of how the app ran...for example, a title, followed by a list of all input files, a list of all output files, any warnings encountered.
I think for that I would create an object that implemented ObjectRenderer and write a doRender method that gave me the info I wanted.
Does that sound correct?
Thanks!
You can use variable with this syntax
log4j.appender.R.File=${processDir}/example.log
You must define the variables as system properties (es. -DprocessDir=...) or manually (after creating folder) with
System.setProperty("processDir",logDir);
Is there a possibility to add directory (with sub directories) to SupportFiles section in installshield. I have dynamically created content (list of files changes constantly) that I use to support my installation.
Is there any other method to add dynamically whole directory to installation package. I need files from this directory only during installation process. At the end of the installation I want IS to remove these files automatically ?
The ISSetupFile table ( which drives the Support Files pattern ) only supports extracting files to a single directory. There is no way in the UI to tell it to be dynamic although you could have a build automation step that reflects the contents of a directory and wires it up to the table using the automation interface. This still couldn't do sub directories though.
What you probably want to do is creating a self-extracting zip of the files you need as part of your build and add that EXE to Support Files. Then write custom actions that call the EXE to extract the payload to Support Files and another CA to clean it up. Look for the custom action ISSetupFilesExtract and ISSetupFilesCleanup to figure out the best way to do this.
Although I'm late to the party: #BuvinJ mentioned in a comment that you can add directories to "Advanced Files" under "Disk1." In this case, they do appear in the temporary directory (support directory), eg. SUPPORTDIR\Disk1
SUPPORTDIR is a temporary location where the installer dumps files, and cleans them up afterward. An example is C:\Users\<your user name>\AppData\Local\Temp\2\{F6B9B2D6-2A5A-4146-9297-E80A199CB0CB}.
This could be a quicker/cheaper/faster solution to writing custom actions and/or packaging up files by hand in, say, a self-extracting zip file.
When generating my DAL files with SubSonic, I'd like the names of the files to be .gen.cs. The main reason for this is that the files are partial classes, and I would like to add some additional implementation details into another source file for the table called .cs. This is somewhat the standard pattern for generated source files , and I'm wondering if its possible with SubSonic? I'm using SubSonic 2.2.
I thought you might be able to do this by using a set of custom templates, but the CS_ClassTemplate.aspx (or VB_ClassTemplate.aspx) doesn't control the file name of the class.
I don't think this is possible.
As an alternative, you can do what I do. I have a "generated" directory, such as \database\generated and then I put my partial classes at \database\custom. As long as the namespaces of the files in the two different directories match (like .database or whatever), then it works fine. By using two different directories, it's easier to find your custom files without looking at the generated ones.