I am new to Protocol Buffers and seeing this as good approach to go.
I created a proto file, and using compiler, I generated Java beans.
Using this Java Beans, I initialize the object, and try to write it to file.
The purpose is just to see how big the file is.
I don't have client/server test ready at this moment to test via HTTP.
I am just trying to show my team at this point, how the request/response look like using protocol buffers.
Code I have is something like this:
=== Proto file ===
Profile {
optional string name=1
optional string id=2
message DocGuids {
required string docguids=3
}
repeated DocGuids docguids=4
}
=== Sample code ===
ProfileRequest.Builder profile = ProfileRequest.newBuilder();
profile.setName("John");
profile.setId("123");
for (int i=0;i<10;i++) {
ProfileRequest.DocGuids.Builder docGuids = ProfileRequest.DocGuids.newBuilder();
docGuids.setDocguid(GUID.guid());
profile.addDocguids(docGuids);
}
//write to disk
try {
// Write the new address book back to disk.
FileOutputStream output = new FileOutputStream("c:\\testProto.txt");
DataOutputStream dos = new DataOutputStream(output);
dos.write(profile.build().toByteArray());
dos.close();
output.close();
} catch (Exception e) {
}
When I check testProto.txt, I saw the file was written as text file, not binary file, eventhough I use toByteArray.
Anyone could help?
Thanks
By the way, this is the code to read this file:
// Read from disk
FileInputStream input = new FileInputStream("c:\\testProto.txt");
DataInputStream dis = new DataInputStream(input);
profileBuild.mergeFrom(dis);
dis.close();
input.close()
I am able to read the object and get the value just fine, but just wondering if this is the correct approach to do?
I'm not sure why you're creating the DataOutputStream and calling dos.write in the first place...
I would normally use this:
profile.build().writeTo(output);
However, I would still have expected the file to be a binary file really. Sure, it would include the text "123" and "John" as those are just UTF-8 strings... but there should have been non-text in there as well. Are you sure it's just text? Could you post the output?
Related
The Problem:
I want a high quality visualization of a simulation that I've built and runs on another application. Unreal Engine 4 has models that work, and renders nicely enough to fill my needs. The only problem, is I need to tell a running instance of an Unreal Engine project some of the information that my simulation is creating, like moving objects.
Potential solutions:
rpc plugin?
I'm reasonably familiar with grpc, and my simulation is set up so send and receive grpc messages, but I don't know how to implement grpc through unreal_engine. A couple others on the interweb have written rpc plugins, which might work for my needs. For example...
https://github.com/PaddleCreekGames/Proto3RPC_UE4
However, no idea how to take that pile of work done and actually use what I want. Needless to say, the documentation of that specific project is unfriendly to a user who didn't write it.
Something else?
If you have any ideas, or if you've historically passed information to unreal engine a specific way, any pointers in the right direction/code snippets/links to stuff I may have missed would be greatly appreciated.
I had the same question with theodox,
if the input doesn't have to happen instantly/in real-time.
You may for example update a parameter file for you simulation and read the parameters in your unreal engine application.
Here is an example to create the readFile bluemix function to read and parse an external JSON file:
in /Source/[Project name]/MyBlueprintFunctionLibrary.h, define output struct and function header
USTRUCT(BlueprintType)
struct FResultStruct
{
GENERATED_USTRUCT_BODY()
UPROPERTY(EditAnywhere,BlueprintReadWrite,Category = "Result Struct")
float fieldname1;
UPROPERTY(EditAnywhere,BlueprintReadWrite,Category = "Result Struct")
FString fieldname2;
//Constructor
FResultStruct()
{
fieldname1 = 0;
fieldname2 = "string";
}
};
UCLASS()
class EXPERIMENTALPROJECT_API UMyBlueprintFunctionLibrary : public UBlueprintFunctionLibrary
{
GENERATED_BODY()
public:
UFUNCTION(BlueprintCallable, Category = "MyBPLibrary")
static FResultStruct ReadFile(FString fileName);
};
in /Source/[Project name]/MyBlueprintFunctionLibrary.cpp implement your parameter file parser, for example:
// Read the results file
FResultStruct UMyBlueprintFunctionLibrary::ReadFile(FString fileName)
{
FString saveFilePath = FPaths::ConvertRelativePathToFull(FPaths::GameDir());
fileName = saveFilePath + fileName;
FResultStruct Result;
FString jsonString;
FFileHelper::LoadFileToString(jsonString, *fileName);
TSharedPtr<FJsonObject> JsonParsed;
TSharedRef<TJsonReader<TCHAR>> JsonReader = TJsonReaderFactory<TCHAR>::Create(jsonString);
if (FJsonSerializer::Deserialize(JsonReader, JsonParsed))
{
Result.fieldname1 = JsonParsed->GetNumberField("fieldname1");
Result.fieldname2 = JsonParsed->GetStringField("fieldname2");
}
return Result;
}
And in your unreal engine blueprint you can call this file reading function and use its parsed content in your application:
example
If you need live input, you may need to create HUD in UE4 and create input fields for all your parameters, that's more labor intensive imo.
I'm currently doing a FTP download using MFC. Is a very simple program which takes 2 inputs from user and click a download button in order to download from server. Everything is fine and im able to download it from. But i realized this program can only be executed once. Either successful or fail user has to open the .exe again to download another file. I'm a beginner in C&C++ with a simple knowledge i put OnInitDialog() at the last line of the download function hopping it will loop back and initialize again. Of course it doesn't work. Below are my current codes for the download button
BOOL CFTPDOWNLOADDlg::Log_In(char* path, char* ID, char* password {
m_pFtpConnection = NULL;
try{
// path
// ID
// password
m_pFtpConnection = m_Session.GetFtpConnection(path,
ID,password,INTERNET_INVALID_PORT_NUMBER);
}
catch(CInternetException *pEx){
pEx->ReportError(MB_ICONEXCLAMATION);
m_pFtpConnection = NULL;
pEx->Delete();
return FALSE;
}
return TRUE;
}
BOOL CFTPDOWNLOADDlg::Download(){
m_Edit3.SetWindowText("Downloading..");
m_Session.EnableStatusCallback(TRUE);
if(m_pFtpConnection->GetFile(serv_Loc,host_Loc,
FALSE,FILE_ATTRIBUTE_NORMAL,FTP_TRANSFER_TYPE_BINARY,1) != 0){
MessageBox("Download Complete");
m_Edit3.SetWindowText("");}
else{
MessageBox("Download Fail");
return FALSE;
}
// Log_out Session
m_Session.Close();
m_pFtpConnection->Close();
if(m_pFtpConnection!=NULL) delete m_pFtpConnection;
else MessageBox("Download Complete");
return TRUE;
}
BOOL CFTPDOWNLOADDlg::get_Path(){
...
...
...
sprintf(serv_Loc,"soft\\%s\\%d\\%s.zip",s_No,r_Number,r_No);
sprintf(host_Loc,"%s\\%s.zip",buff2,r_No);
return TRUE;
}
void CFTPDOWNLOADDlg::OnCancel() {
// Log_out Session
m_Session.Close();
m_pFtpConnection->Close();
if(m_pFtpConnection!=NULL)
delete m_pFtpConnection;
CDialog::OnCancel();
}
void CFTPDOWNLOADDlg::OnDLButton() {
//get path from user input
get_Path();
// start download
Download();
}
I've tried to search online, i couldn't find anything which is close. Sorry for my poor explanation.
Thank you in advance for your kindness in replying
Here is what you need to do:
You should make CInternetSession m_Session; a member of your CWinApp-derived class.
You should call m_Session.Close() in ExitInstance() method of your CWinApp-derived class.
In your CDialog-derived class you should only deal with CFtpConnection related stuff. So when user clicks on Download button you should call GetFtpConnection() and initialize your m_pFtpConnection and do the rest. When download/upload is done call m_pFtpConnection->Close(); and delete m_pFtpConnection;
Please also use CString instead of char*. There are lots of benefits like automatic UNICODE support, etc.
Please also consider using CString::Format() method instead of sprintf().
You should also consider using threads to perform upload/download tasks in a separate worker thread. Use AfxBeginThread() to start the thread. This way you'll not affect Windows message pump that is a part of main application (GUI) thread. So your GUI wont lock up while you uploading/downloading files.
I've a (fairly large) Azure application that uploads (fairly large) files in parallel to Azure blob storage.
In a few percent of uploads I get an exception:
The specified block list is invalid.
System.Net.WebException: The remote server returned an error: (400) Bad Request.
This is when we run a fairly innocuous looking bit of code to upload a blob in parallel to Azure storage:
public static void UploadBlobBlocksInParallel(this CloudBlockBlob blob, FileInfo file)
{
blob.DeleteIfExists();
blob.Properties.ContentType = file.GetContentType();
blob.Metadata["Extension"] = file.Extension;
byte[] data = File.ReadAllBytes(file.FullName);
int numberOfBlocks = (data.Length / BlockLength) + 1;
string[] blockIds = new string[numberOfBlocks];
Parallel.For(
0,
numberOfBlocks,
x =>
{
string blockId = Convert.ToBase64String(Guid.NewGuid().ToByteArray());
int currentLength = Math.Min(BlockLength, data.Length - (x * BlockLength));
using (var memStream = new MemoryStream(data, x * BlockLength, currentLength))
{
var blockData = memStream.ToArray();
var md5Check = System.Security.Cryptography.MD5.Create();
var md5Hash = md5Check.ComputeHash(blockData, 0, blockData.Length);
blob.PutBlock(blockId, memStream, Convert.ToBase64String(md5Hash));
}
blockIds[x] = blockId;
});
byte[] fileHash = _md5Check.ComputeHash(data, 0, data.Length);
blob.Metadata["Checksum"] = BitConverter.ToString(fileHash).Replace("-", string.Empty);
blob.Properties.ContentMD5 = Convert.ToBase64String(fileHash);
data = null;
blob.PutBlockList(blockIds);
blob.SetMetadata();
blob.SetProperties();
}
All very mysterious; I'd think the algorithm we're using to calculate the block list should produce strings that are all the same length...
We ran into a similar issue, however we were not specifying any block ID or even using the block ID anywhere. In our case, we were using:
using (CloudBlobStream stream = blob.OpenWrite(condition))
{
//// [write data to stream]
stream.Flush();
stream.Commit();
}
This would cause The specified block list is invalid. errors under parallelized load. Switching this code to use the UploadFromStream(…) method while buffering the data into memory fixed the issue:
using (MemoryStream stream = new MemoryStream())
{
//// [write data to stream]
stream.Seek(0, SeekOrigin.Begin);
blob.UploadFromStream(stream, condition);
}
Obviously this could have negative memory ramifications if too much data is buffered into memory, but this is a simplification. One thing to note is that UploadFromStream(...) uses Commit() in some cases, but checks additional conditions to determine the best method to use.
This exception can happen also when multiple threads open stream into a blob with the same file name and try to write into this blob simultaneously.
NOTE: this solution is based on Azure JDK code, but I think we can safely assume that pure REST version will have the very same effect (as any other language actually).
Since I have spent entire work day fighting this issue, even if this is actually a corner case, I'll leave a note here, maybe it will be of help to someone.
I did everything right. I had block IDs in the right order, I had block IDs of the same length, I had a clean container with no leftovers of some previous blocks (these three reasons are the only ones I was able to find via Google).
There was one catch: I've been building my block list for commit via
CloudBlockBlob.commitBlockList(Iterable<BlockEntry> blockList)
with use of this constructor:
BlockEntry(String id, BlockSearchMode searchMode)
passing
BlockSearchMode.COMMITTED
in the second argument. And THAT proved to be the root cause. Once I changed it to
BlockSearchMode.UNCOMMITTED
and eventually landed on the one-parameter constructor
BlockEntry(String id)
which uses UNCOMMITED by default, commiting the block list worked and blob was successfuly persisted.
I am trying to extract attachments using EmbeddedObjects, I am able to extract attachments but create em*tm temp files in system temp folder.
EmbeddedObject embeddedObject=document.getAttachment(attachmentName);
InputStream inputStream=embeddedObject.getInputStream();
.....
......
inputStream.close();
embeddedObject..recycle();
document..recycle();
After closing input Stream its not deleting temp file form system temp folder.
Is it any thing wrong in my code or its setting issue with lotus notes.
Can you please help me in this?
Thanks for the help.
This is a common issue, and it relates to the incorrect closure/recycle of objects (either missing or out of sequence). E0*TM files will be created while the objects are alive and cleaned up when recycled.
If they are correct then check to see if any Antivirus software running that is blocking deletion.
The following sample code I used to test this before works, so compare to yours.
try {
System.out.println("Start");
String path = "test.txt";
Session session = getSession();
AgentContext agentContext = session.getAgentContext();
System.out.println("Get DB");
Database db = session.getCurrentDatabase();
System.out.println("View + doc");
View vw = db.getView("main");
Document doc = vw.getFirstDocument();
System.out.println("Embedded object");
EmbeddedObject att = doc.getAttachment(path);
InputStream is = att.getInputStream();
ByteArrayOutputStream fos = new ByteArrayOutputStream();
byte buffer[] = new byte[(int) att.getFileSize()];
int read;
do {
read = is.read(buffer, 0, buffer.length);
if (read > 0) {
fos.write(buffer, 0, read);
}
} while (read > -1);
fos.close();
is.close();
// recycle the domino variables
doc.recycle();
vw.recycle();
db.recycle();
att.recycle();
} catch (Exception e) {
e.printStackTrace();
}
My suggestion would be to first comment out all the code that you represented in your post as
.....
......
Does the temp file still get left behind? If so, it looks like it's a bug in the Notes back end classes for 8.x that needs to be reported to IBM.
If not, then something in the commented-out code is preventing the the close() call from succeeding. InputStream is an abstract class, so perhaps you are binding inputStream to another type of stream object that must be closed in order to prevent the file from staying open.
So I was writing a quick application to sort my wallpapers neatly into folders according to aspect ratio. Everything is going smoothly until I try to actually move the files (using FileInfo.MoveTo()). The application throws an exception:
System.IO.IOException
The process cannot access the file because it is being used by another process.
The only problem is, there is no other process running on my computer that has that particular file open. I thought perhaps that because of the way I was using the file, perhaps some internal system subroutine on a different thread or something has the file open when I try to move it. Sure enough, a few lines above that, I set a property that calls an event that opens the file for reading. I'm assuming at least some of that happens asynchronously. Is there anyway to make it run synchronously? I must change that property or rewrite much of the code.
Here are some relevant bits of code, please forgive the crappy Visual C# default names for things, this isn't really a release quality piece of software yet:
private void button1_Click(object sender, EventArgs e)
{
for (uint i = 0; i < filebox.Items.Count; i++)
{
if (!filebox.GetItemChecked((int)i)) continue;
//This calls the selectedIndexChanged event to change the 'selectedImg' variable
filebox.SelectedIndex = (int)i;
if (selectedImg == null) continue;
Size imgAspect = getImgAspect(selectedImg);
//This is gonna be hella hardcoded for now
//In the future this should be changed to be generic
//and use some kind of setting schema to determine
//the sort/filter results
FileInfo file = ((FileInfo)filebox.SelectedItem);
if (imgAspect.Width == 8 && imgAspect.Height == 5)
{
finalOut = outPath + "\\8x5\\" + file.Name;
}
else if (imgAspect.Width == 5 && imgAspect.Height == 4)
{
finalOut = outPath + "\\5x4\\" + file.Name;
}
else
{
finalOut = outPath + "\\Other\\" + file.Name;
}
//Me trying to tell C# to close the file
selectedImg.Dispose();
previewer.Image = null;
//This is where the exception is thrown
file.MoveTo(finalOut);
}
}
//The suspected event handler
private void filebox_SelectedIndexChanged(object sender, EventArgs e)
{
FileInfo selected;
if (filebox.SelectedIndex >= filebox.Items.Count || filebox.SelectedIndex < 0) return;
selected = (FileInfo)filebox.Items[filebox.SelectedIndex];
try
{
//The suspected line of code
selectedImg = new Bitmap((Stream)selected.OpenRead());
}
catch (Exception) { selectedImg = null; }
if (selectedImg != null)
previewer.Image = ResizeImage(selectedImg, previewer.Size);
else
previewer.Image = null;
}
I have a long-fix in mind (that's probably more efficient anyway) but it presents more problems still :/
Any help would be greatly appreciated.
Since you are using your selectedImg as a Class scoped variable it is keeping a lock on the File while the Bitmap is open. I would use an using statement and then Clone the Bitmap into the variable you are using this will release the lock that Bitmap is keeping on the file.
Something like this.
using ( Bitmap img = new Bitmap((Stream)selected.OpenRead()))
{
selectedImg = (Bitmap)img.Clone();
}
New answer:
I looked at the line where you do an OpenRead(). Clearly, this locks your file. It would be better to provide the file path instead of an stream, because you can't dispose your stream since bitmap would become erroneous.
Another thing I'm looking in your code which could be a bad practice is binding to FileInfo. Better create a data-transfer object/value object and bind to a collection of this type - some object which has the properties you need to show in your control -. That would help in order to avoid file locks.
In the other hand, you can do some trick: why don't you show streched to screen resolution images compressing them so image size would be extremly lower than actual ones and you provide a button called "Show in HQ"? That should solve the problem of preloading HD images. When the user clicks "Show in HQ" button, loads that image in memory, and when this is closed, it gets disposed.
It's ok for you?
If I'm not wrong, FileInfo doesn't block any file. You're not opening it but reading its metadata.
In the other hand, if you application shows images, you should move to memory visible ones and load them to your form from a memory stream.
That's reasonable because you can open a file stream, read its bytes and move them to a memory stream, leaving the lock against that file.
NOTE: This solution is fine for not so large images... Let me know if you're working with HD images.
using(selectedImg = new Bitmap((Stream)selected))
Will that do it?