I am using MapDB for key-value database for the better performace.After the map.clear() my files are not deleted in disk.I need to reclaim the diskspace after the remove() or clear() method.Below are the my code sippet.
public static void mapDBPerformanceonFileDBStack() throws Exception{
String fileName ="testdatabase.db";
DB db = DBMaker
.fileDB(fileName)
.closeOnJvmShutdown()
.make();
Map myMap = db.hashMap("testmap").keySerializer(Serializer.INTEGER).valueSerializer(Serializer.STRING).createOrOpen();
System.out.println("Intial mapsize :" +myMap.size());
for(int i=0;i<100000;i++){
myMap.put(i,"key added" );
}
db.commit();
System.out.println("mapsize after the addition :"+myMap.size());
myMap.clear();
System.out.println("mapsize after clear :"+myMap.size());
db.commit();
db.close();
}
My testdatabase file size not reclaimed after the map.clear() - Same size after the clear().
...
Store[] stores = (HtreeMap)myMap.getStores();
for (Store store : stores) {
store.compact();
}
...
db.commit();
db.close();
Related
I am trying to print all records of a data-view into a file using a for loop in my customization in Acumatica. Unfortunately I am ending up with printing the first record everytime resulting into duplication of records, Unable to track where I am going wrong....Please Assist
Here Goes my Code......
public class MayBankGIROProcess : PXGraph<MayBankGIROProcess>
{
public PXSelect<MayBankGIRO> Document; //This is my Data View
public PXAction<MayBankGiroFilter> createTextFile;
[PXUIField(DisplayName = "Create Text File")]
[PXButton()]
public virtual IEnumerable CreateTextFile(PXAdapter adapter)
{
List<string> myList = new List<string> { };
foreach (MayBankGIRO dacRecord in this.Document.Select()) //this is the loop which is taking the data records.
{
myList.Add(dacRecord.ReordType+ "|"+ dacRecord.CustomerReferenceNumber+ "|"+ dacRecord.ClientBatchID+ "|");
// The above line is printing only the first record of the data view everytime .
}
string filename = "DAWN" + ".txt";
Download(myList, filename);
return adapter.Get();
}
public static void Download(List<string> lines, string name) //method generating file
{
var bytes = default(byte[]);
using (MemoryStream stream = new MemoryStream())
{
StreamWriter sw = new StreamWriter(stream);
foreach (string line in lines)
{
sw.WriteLine(line);
}
stream.Position = 0;
bytes = stream.ToArray();
sw.Close();
};
PX.SM.FileInfo textDoc = new PX.SM.FileInfo(name, null, bytes);
if (textDoc != null)
{
throw new PXRedirectToFileException(textDoc, true);
}
else
{
PXTrace.WriteInformation("Could not generate file");
}
}
}
[Generated Text File with all duplicate Record][1]
[1]: https://i.stack.imgur.com/Kllmk.png
[Original Record from database][2]
[2]: https://i.stack.imgur.com/Rbr9k.png
This usually happens when the report is pulling from a SQL View DAC which doesn't have unique key defined. Add IsKey=True on DAC fields until the SQL view is pulling unique record and the error should go away.
I am trying to update an item in was dynamoDB using nodes, db.updateItem(query).
I am getting the following error :
Invalid UpdateExpression: Expression size has exceeded the maximum allowed size dynamodb
On reading few posts, I realised that dynamoDB allows itemSize to be 400KB and that might be a problem here. But if that is the problem, why did it allow to insert the item in the first place.
I am not sure what exactly the issue. Any help would be appreciated.
Please let me know if I missed any required information
You are probably hitting Expression Parameters limits. Please refer to:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-expression-parameters
If you are getting this exception
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Item size has exceeded the maximum allowed size
This exception is due to AWS Dynamodb limits mentioned here
in my case, I compressed the record using gzip and stored binary zipped data, and uncompressed it back after reading that record.
please see below sample code to compress and decompress (I am using enhanced dynamodb client library)
public CompletableFuture<Boolean> storeItem(MyBeanClass object) {
CompletableFuture<Boolean> completableFuture = CompletableFuture.supplyAsync(() -> false);
try {
byte[] serialized = objectMapper.writeValueAsString(object.getLargeData()).getBytes(UTF_8);
if (serialized.length >= 10000) { //large record, gzip it
try (ByteArrayOutputStream bos = new ByteArrayOutputStream(serialized.length);
GZIPOutputStream gzip = new GZIPOutputStream(bos)) {
gzip.write(serialized);
gzip.close();
MyBeanClass newObject = new MyBeanClass();
newObject.setPrimaryId(object.getPrimaryId());
newObject.setGzData(SdkBytes.fromByteArray(bos.toByteArray()));
completableFuture = enhancedDynamoDbTable.putItem(newObject)
.thenApply(res -> true)
.exceptionally(th -> {
th.printStackTrace();
return false;
});
}
} else { //no compression required
completableFuture = enhancedDynamoDbTable.putItem(object).thenApply(res -> true)
.exceptionally(th -> {
th.printStackTrace();
return false;
});
}
} catch (IOException e) {
e.printStackTrace();
}
To fetch record and unzip
public CompletableFuture<MyBeanClass> getItem(String id) {
return enhancedDynamoDbTable
.getItem(Key.builder().partitionValue(id).build())
.thenApply(record -> {
if (record.getGzData() != null) {
try (ByteArrayInputStream arrayInputStream = new ByteArrayInputStream(record.getGzData().asByteArray());
GZIPInputStream inputStream = new GZIPInputStream(arrayInputStream);
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
byteArrayOutputStream.write(buffer, 0, length);
}
record = objectMapper.readValue(byteArrayOutputStream.toString(UTF_8), MyBeanClass.class);
} catch (IOException e) {
e.printStackTrace();
}
}
return record;
});
}
Hope that helps.
I've added storage file to my codenameone application. In some event I wanna delete specific storage file and recreate it with some old filtered data and new data. It doesn't work well on deletion.
First I create method for clear storage file in StorageManager class:
public void clearData(String pStorageTable){
Storage.getInstance().deleteStorageFile(pStorageTable);
}
In other class I use this method like this:
// load all data of specific storage file
// ...
new DBManager().clearData(ThreeTrans.DB_NAME);
// write old data with filtering of specific ID and new data
// ...
here is method of write data:
public void write(ThreeTrans pTT){
if(store == null) {
store = Storage.getInstance();
}
DB_NAME = "TT";
if(!store.exists(DB_NAME)) {
Hashtable depHash = new Hashtable();
String k = "1" + pTT.getNumber();
depHash.put(k, pTT.toString());
store.writeObject(DB_NAME, depHash);
}
else {
Hashtable depHash = (Hashtable)store.readObject(DB_NAME);
if (!depHash.containsValue(pTT.getNumber())) {
String k = String.valueOf(getLastKeyNumber());
depHash.put(k, pTT.toString());
store.writeObject(DB_NAME, depHash);
}
}
}
at first I was using this method for delete storage file:
public void clearData(String pStorageTable){
if(store == null) {
store = Storage.getInstance();
}
for (String str : store.listEntries()) {
if(str.toLowerCase().startsWith(pStorageTable)) {
store.deleteStorageFile(str);
}
}
}
and after this problem this method changed to this;
public void clearData(String pStorageTable){
Storage.getInstance().deleteStorageFile(pStorageTable);
}
I'm assuming that you didn't invoke close() on the output stream or the input stream working with the file and there is still a lock on the file. This is usually the case for such issues.
I am developing a Location-based J2ME app & in that I'm using RMS to store data.
In RecordStore when I delete any record, the underlying records doesn't get re-indexed. For example, if I have 5 records & I delete record no.2 then record ids will be {1, 3, 4, 5}. But I want record ids after deletion to be {1, 2, 3, 4}. How should I do this??? Because recordId is playing an important role in my app to retrieve & update the record.
You need to change your application logic. ID is just for identification, and not for sorting. Because it is for identification, it must remains the same.
Very often the easiest thing to do is to read and write the whole recordstore at once.
So, since you've said that your record store is basically small (not that much data), I would recommend simply adding your own custom id field to each record. As Meier said, the RMS record id is not really meant to be recalculated, and changed, once a record has been created. So, I would use your own.
If each of your records contain:
boolean isMale
int age
String firstName
then, I would simply add another field at the start of each record:
int id
It makes your records a little bigger, but not much (4 bytes/record). If you'll have less than 64k records, then you could also use a short for the id, and save a couple bytes.
Here's an example (adapted from this IBM tutorial), of reading, writing, and deleting with this kind of record:
private RecordStore _rs;
// these next two methods are just small optimizations, to allow reading and
// updating the ID field in a record without the overhead of creating a new
// stream to call readInt() on. this assumes the id is a 4 byte int, written
// as the first field in each record.
/** Update one record with a new id field */
private static final void putIdIntoRecord(int id, byte[] record) {
// we assume the first 4 bytes are the id (int)
record[0] = (byte)(id >> 24);
record[1] = (byte)(id >> 16);
record[2] = (byte)(id >> 8);
record[3] = (byte)id;
}
/** Get the id field from one record */
private static final int getIdFromRecord(byte[] record) {
// we assume the first 4 bytes are the id (int)
return ((0xFF & record[0]) << 24) |
((0xFF & record[1]) << 16) |
((0xFF & record[2]) << 8) |
(0xFF & record[3]);
}
/** delete a record with the given (custom) id, re-indexing records afterwards */
private void delete(int idToDelete) {
try {
RecordEnumeration enumerator = _rs.enumerateRecords(new IdEqualToFilter(idToDelete),
null, false);
_rs.deleteRecord(enumerator.nextRecordId());
// now, re-index records after 'idToDelete'
enumerator = _rs.enumerateRecords(new IdGreaterThanFilter(idToDelete), null, true);
while (enumerator.hasNextElement()) {
int recordIdToUpdate = enumerator.nextRecordId();
byte[] record = _rs.getRecord(recordIdToUpdate);
// decrement the id by 1
int newId = getIdFromRecord(record) - 1;
// copy the new id back into the record
putIdIntoRecord(newId, record);
// update the record, which now has a lower id, in the store
_rs.setRecord(recordIdToUpdate, record, 0, record.length);
}
} catch (RecordStoreNotOpenException e) {
e.printStackTrace();
} catch (InvalidRecordIDException e) {
e.printStackTrace();
} catch (RecordStoreException e) {
e.printStackTrace();
}
}
/** generate some record store data ... example of writing to store */
public void writeTestData()
{
// just put 20 random records into the record store
boolean[] booleans = new boolean[20];
int[] integers = new int[20];
String[] strings = new String[20];
for (int i = 0; i < 20; i++) {
booleans[i] = (i % 2 == 1);
integers[i] = i * 2;
strings[i] = "string-" + i;
}
writeRecords(booleans, integers, strings);
}
/** take the supplied arrays of data, and save a record for each array index */
public void writeRecords(boolean[] bData, int[] iData, String[] sData)
{
try
{
// Write data into an internal byte array
ByteArrayOutputStream strmBytes = new ByteArrayOutputStream();
// Write Java data types into the above byte array
DataOutputStream strmDataType = new DataOutputStream(strmBytes);
byte[] record;
for (int i = 0; i < sData.length; i++)
{
// Write Java data types
strmDataType.writeInt(i); // this will be the ID field!
strmDataType.writeBoolean(bData[i]);
strmDataType.writeInt(iData[i]);
strmDataType.writeUTF(sData[i]);
// Clear any buffered data
strmDataType.flush();
// Get stream data into byte array and write record
record = strmBytes.toByteArray();
_rs.addRecord(record, 0, record.length);
// Toss any data in the internal array so writes
// starts at beginning (of the internal array)
strmBytes.reset();
}
strmBytes.close();
strmDataType.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
/** read in all the records, and print them out */
public void readRecords()
{
try
{
RecordEnumeration re = _rs.enumerateRecords(null, null, false);
while (re.hasNextElement())
{
// Get next record
byte[] recData = re.nextRecord();
// Read from the specified byte array
ByteArrayInputStream strmBytes = new ByteArrayInputStream(recData);
// Read Java data types from the above byte array
DataInputStream strmDataType = new DataInputStream(strmBytes);
// Read back the data types
System.out.println("Record ID=" + strmDataType.readInt());
System.out.println("Boolean: " + strmDataType.readBoolean());
System.out.println("Integer: " + strmDataType.readInt());
System.out.println("String: " + strmDataType.readUTF());
System.out.println("--------------------");
strmBytes.close();
strmDataType.close();
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
Here, I make use of a couple small RecordFilter classes, to use when searching the record store:
/** helps filter out records greater than a certain id */
private class IdGreaterThanFilter implements RecordFilter {
private int _minimumId;
public IdGreaterThanFilter(int value) {
_minimumId = value;
}
public boolean matches(byte[] candidate) {
// return true if candidate record's id is greater than minimum value
return (getIdFromRecord(candidate) > _minimumId);
}
}
/** helps filter out records by id field (not "recordId"!) */
private class IdEqualToFilter implements RecordFilter {
private int _id;
public IdEqualToFilter(int value) {
_id = value;
}
public boolean matches(byte[] candidate) {
// return true if candidate record's id matches
return (getIdFromRecord(candidate) == _id);
}
}
I am creating one J2ME application which read/write RMS record. I able to read and write
record in RMS but now problem is that I want to delete record by accepting some value
like accountNumber.
Format of RMS record.
101,ABC,12345,12345
and String str contain following data.
String str=accountSrNumber +","+ name +","+ balance +","+ TextDeposit;
deleteRecStore(str,accountSrNumber);
And I need to accept accountNumber(101) from user and need to delete this record.
Here is my Delete method.
public void deleteRecStore(String str, String accNumber121) //
{
int s=str.indexOf(accNumber121);
System.out.println("index in delete function"+s);
if(s==0)
{
try{
rs.deleteRecord(s);
// RecordStore.deleteRecordStore(REC_STORE);
System.out.println("record delete successfully");
}
catch (Exception e)
{}
}
}
I tried to use both of method rs.deleteRecord(s) and RecordStore.deleteRecordStore(REC_STORE);.
But none helps.
You always delete record 0 which is the first record, which is a bad idea.
For example, if you add two records, and delete them, and than add another record, it will be indexed as 2, so you will have to call deleteRecord(2) to remove it.
Method deleteRecordStore() removes entire recordStore (which contains records) - after that, if you create one, the next added record will be indexed as zero.
If I got the idea, you want to delete a record by it's acoountNumber.
If i'm right, you need to find the recordID by it's contents. The code will probably look like this (may have mistakes, did not test it, but the idea is important):
public void deleteRecStore(String accNumber121) {
RecordEnumeration e = rs.enumerateRecords();
int found = -1;
while (e.hasMoreElements()) {
int id = e.nextRecordId();
String next = new String(e.nextRecord());
if (next.startsWidth(accNumber121)) {
found = id;
}
}
if (found == -1) {
System.out.println("not found!");
} else {
rs.deleteRecord(found);
}
}