I have written a groovy script to read a huge file line by line.
I currently am using boilerplate code as follows
File hugeFile = new File(filePath)
if (hugeFile.exists()) {
hugeFile.eachLine {line ->
//some process
}
}
My question is how do I find out if the "eachLine" is using a BufferedReader to be memory efficient ?
Give it a shot to see if it can handle big files.
And check the source at github:
public static <T> T eachLine(Reader self, int firstLine, #ClosureParams(value=FromString.class,options={"String","String,Integer"}) Closure<T> closure) throws IOException {
BufferedReader br;
int count = firstLine;
T result = null;
if (self instanceof BufferedReader)
br = (BufferedReader) self;
else
br = new BufferedReader(self);
...
}
Related
I have been trying to make a string validator but some characters in the input have some swedish characters which I haven't been able to parse. I have been going mad about this. Tried everything I found on the internet. Can anybody please help me out?
This is the logic for reading the file. I have been trying to parse it.
I haven't been able to specifically parse this word: leverantör. It's parsed as leverant�r.
public String processFile(MultipartFile file) throws IOException, FormatNotFoundException {
// TODO Auto-generated method stub
BufferedReader r = new BufferedReader(new InputStreamReader(file.getInputStream()));
String output;
while((output = r.readLine())!=null ) {
logger.debug(output);
InputStream is= new ByteArrayInputStream(output.getBytes(StandardCharsets.UTF_8));
StringBuilder textBuilder = new StringBuilder();
try (
Reader reader = new BufferedReader(new InputStreamReader
(is, Charset.forName(StandardCharsets.UTF_8.name())))) {
int c = 0;
while ((c = reader.read()) != -1) {
textBuilder.append((char) c);
}
}
logger.debug(textBuilder.toString());
System.out.println(textBuilder.toString());
}
return "File read.";
}
I have Loader class where I load txt file into BufferedReader from resources and return this field. I use this method but it acts really strange(for me). When I don't put
String str = bufferReader.readLine(); after
bufferReader = new BufferedReader(fileReader);
(in Loader class) than bufferReader in another class is empty, and readLine() returns null. When I write that piece of code in Loader class, I can read each line from txt, except the 1. one which is read in Loader class. Also, I can't read last line if I dont put enter at the end.
public BufferedReader loadFromFileToBufferReader(String fileName) {
ClassLoader classLoader = getClass().getClassLoader();
System.out.print(getClass().getClassLoader().getResource("resources/" + fileName));
File file = new File(classLoader.getResource("resources/" + fileName).getFile());
BufferedReader bufferReader = null;
try (FileReader fileReader = new FileReader(file)) {
bufferReader = new BufferedReader(fileReader);
String str = bufferReader.readLine();
} catch (IOException e) {
System.err.println("Something went terribly wrong with file reading");
}
return bufferReader;
}
and usage:
public Database() {
productsInDatabse = new ArrayList<>();
codesList = new ArrayList<>();
loader = new LoadFromFile();
BufferedReader output = loader.loadFromFileToBufferReader("database.txt");
Product product;
String line;
String[] array;
try {
line = output.readLine();
while (line != null) {
You should paste your code here because it's hard to deduce all the possible causes of this without seeing the code on 100% but I am guessing you have it the same file open at the same time from multiple sources without closing it before from one? Could be literally millions of little things, just telling you how the same error happened to me.
private List<T> ReadCurrentFile(string currentExtractedFile, PurgingDetails purgingParams)
{
List<T> thinLogDoList = new List<T>();
using (StreamReader sr = new StreamReader(currentExtractedFile))
{
string currentLine = string.Empty;
Dictionary<string, string> ColumnNamesDictionary = null;
while ((currentLine = sr.ReadLine()) != null)
{
if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
{
string[] columnNames = currentLine.Split(new char[] { 'Æ' });
ColumnNamesDictionary = FillColumnNameDictionary(columnNames);
if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
{
thinLogDoList.Add(FillThinLogDO(ColumnNamesDictionary));
}
}
}
}
return thinLogDoList;
}
(Above code is for Reading a File and adding data to the List by filling the object.)
The function is reading file of size 10 MB which is inside a zip file, first I am extracting the zip files, then reading the data, using this function and storing it into List and then deleting the extracted zip files. It is working for approximately 6L(6,00,000) Data but above that data it throws exception.
I want to read More data 10L(10,00,000) how should I do that ?
Do not return a list. Instead, use yield return to just run through the data:
private IEnumerable<i1LogThinDO> ReadCurrentFile(string currentExtractedFile,
PurgingDetails purgingParams)
{
using (StreamReader sr = new StreamReader(currentExtractedFile))
{
string currentLine = string.Empty;
Dictionary<string, string> ColumnNamesDictionary = null;
while ((currentLine = sr.ReadLine()) != null)
{
if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
{
string[] columnNames = currentLine.Split(new char[] { 'Æ' });
ColumnNamesDictionary = FillColumnNameDictionary(columnNames);
if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
{
yield return FillThinLogDO(ColumnNamesDictionary);
}
}
}
}
}
This way, the ball is in the caller's yard. The caller must be able to process the data returned from this method without keeping them all in memory. This could mean that you have to redesign the calling methods as well, but it would bring a huge cut down in memory footprint of the application if you could do all the processing without keeping the data in memory.
I have my uni assignment and it's only very basic coding but I have to do
A user shall be able to store records to a file. On start-up a user shall be able to select a file of records and load these into the program.
I am having trouble with this as it will save but once I close the program and re-open it they are gone, any ones help is appreciated.
This is what I have so far:
private void Save_Click(object sender, EventArgs e)
{
SaveToFile(records, file);
}
private void SaveToFile(List<Client> records, string file)
{
//File.WriteAllText(file, String.Empty);
StreamWriter writer = new StreamWriter(file);
StreamReader reader = new StreamReader(file);
try
{
AddMember();
for (int i = 0; i < records.Count; i++)
{
writer.WriteLine(records[i].WriteToFile());
}
writer.Close();
}
catch (IOException z)
{
MessageBox.Show("Error" + z);
}
}
Before closing the StreamWriter you should call Flush() method. Flush() Clears all buffers for the current writer and causes any buffered data to be written to the underlying stream.
reader.clos();
you forgot this
This is just a guess, but it sounds like you might be overwriting the file when you start the program.
In your SaveToFile method, you have the following two lines at the start:
StreamWriter writer = new StreamWriter(file);
StreamReader reader = new StreamReader(file);
The first one will create a new file with the name in file. The second one isn't needed if you're not doing any reading from the file.
Now, if you have a similar block of code in somewhere else in your program that is executed before the SaveToFile method is called, it will overwrite the file (and since you most likely don't write anything in that earlier part of the code, you're left with a blank file).
To prevent this, there are two suggestions I'll offer:
Only create a StreamWriter when you are going to be writing to the file. Any other times you create a StreamWriter, you will be overwriting the existing file.
If you don't want to overwrite the file, you can use the overload of the constructor that takes a second parameter (a boolean) to indicate whether or not to append new data to the file:
StreamWriter writer = new StreamWriter(file, true);
Additionally, I'd suggest getting to know and use using blocks, as this will take care of closing and flushing the StreamWriter (and its underlying stream) for you, even if an exception occurs and is unhandled:
private void SaveToFile(List<Client> records, string file)
{
try
{
AddMember();
// If you don't want to append but overwrite, use:
// using (StreamWriter writer = new StreamWriter(file))
using (StreamWriter writer = new StreamWriter(file, append))
{
for (int i = 0; i < records.Count; i++)
{
writer.WriteLine(records[i].WriteToFile());
}
}
}
catch (IOException z)
{
MessageBox.Show("Error" + z);
}
}
I'm trying to make a Java program more "Groovy". The java code reads an InputStream like so:
static int myFunction(InputStream is) throws IOException {
int b=is.read();
if (b==0) return b;
StringBuffer sb=new StringBuffer();
int c;
boolean done = false;
while(!done) {
c=is.read();
sb.append((char)c);
if(c == '\n') {
done=true;
}
}
System.out.println(sb.toString());
if (b == 1) throw new IOException("blah");
return b;
}
My Groovy version looks like this:
def myFunction(InputStream is) throws IOException {
int b=is.read()
if (b==0) return b
def reader = new BufferedReader(new InputStreamReader(is))
reader.eachLine { println(it) }
println("DONE")
if (b == 1) throw new IOException("blah")
return b
}
It prints the contents of the stream and then just hangs as if it's trying to read more. It never prints "DONE" (added for debugging). Next I tried it using is.eachByte and passing a closure with an explicit "if (c == '\n') return" but I found that return inside a closure acts more like a continue and doesn't actually break out of the closure. Any idea what I'm doing wrong?
Instead of
reader.eachLine { println(it) }
Can you try
println reader.readLine()