NiFi - InvokeHTTP not making GET calls while taking parameter from incoming FlowFile - get

I have an InvokeHTTP processor that makes a GET that accepts an author variable from an incoming FlowFile as input. However, InvokeHTTP does not appear to be doing anything with the input: https://www.pastepic.xyz/image/LKc34
I tested the Remote URL in a browser with an author from one of the input flowfiles, and it returns JSON as expected.
Remote URL: https://www.reddit.com/user/${author:urlEncode()}/comments/.json?limit=100
Data Provenance shows only DROP and ATTRIBUTES_MODIFIED events. Shouldn't there be FETCH, CLONE, and/or FORK events?
The author from the incoming FlowFile is in those DROP and ATTRIBUTES_MODIFIED events. The input processor is an EvaluateJsonPath whose only output is the author field, and the property name is author.
Also, I tested hardcoding a username in the Remote URL and disconnecting the input FlowFile. That worked and made the request as expected, returning output. So appears there's something wrong with the interaction between the input FlowFile and InvokeHTTP.
I should be getting back JSON from InvokeHTTP. But it appears that the requests are not being made in the first place, failing silently so to speak. Read/Write and Out are empty, and I'm not getting errors.

Related

How do I use ResponseEntity<Mono<String>> to do a get call via webclient?

Documentaion says:-
ResponseEntity<Mono> or ResponseEntity<Flux> -- this makes the response status and headers known immediately while the body is provided asynchronously at a later point. Whether the body is Mono or Flux depends on how many values the response has.
My usecase is I want the headers right away so that I can do some processes while I wait for the body.
Can anyone help me with a method that would make use of this ?

Java exception: "Can't get a Writer while an OutputStream is already in use" when running xAgent

I am trying to implement Paul Calhoun's Apache FOP solution for creating PDF's from Xpages (from Notes In 9 #102). I am getting the following java exception when trying to run the xAgent that does the processing --> Can't get a Writer while an OutputStream is already in use
The only changes that I have done from Paul's code was to change the package name. I have isolated when the exception happens to the SSJS line: var jce: DominoXMLFO2PDF = new DominoXMLFO2PDF(); All that line does is instantiate the class, there is no custom constructor. I don't believe it is the code itself, but some configuration issue. The SSJS code is in the beforeRenderResponse event where it should be, I haven't changed anything on the xAgent.
I have copied the jar files from Paul's sample database to mine, I have verified that the build paths are the same between the two databases. Everything compiles fine (after I did all this.) This exception appears to be an xpages only exception.
Here's what's really going on with this error:
XPages are essentially servlets... everything that happens in an XPage is just layers on top of a servlet engine. There are basically two types of data that a servlet can send back to whatever is initiating the connection (e.g. a browser): text and binary.
An ordinary XPage sends text -- specifically, HTML. Some xAgents also send text, such as JSON or XML. In any of these scenarios, however, Domino uses a Java Writer to send the response content, because Writers are optimized for sending Character data.
When we need to send binary content, we use an OutputStream instead, because streams are optimized for sending generic byte data. So if we're sending PDF, DOC/XLS/PPT, images, etc., we need to use a stream, because we're sending binary data, not text.
The catch (as you'll soon see, that's a pun) is that we can only use one per response.
Once any HTTP client is told what the content type of a response is, it makes assumptions about how to process that content. So if you tell it to expect application/pdf, it's expecting to only receive binary data. Conversely, if you tell it to expect application/json, it's expecting to only receive character data. If the response includes any data that doesn't match the promised content type, that nearly always invalidates the entire response.
So Domino in its infinite wisdom protects us from making this mistake by only allowing us to send one or the other in a single request, and throws an exception if we disobey that rule.
Unfortunately... if there's any exception in our code when we're trying to send binary content, Domino wants to report that to the consumer... which tries to invoke the output writer to send HTML reporting that something went wrong. Except we already got a handle on the output stream, so Domino isn't allowed to get a handle on the output writer, because that would violate its own rule against only using one per response. This, in turn, throws the exception you reported, masking the exception that actually caused the problem (in your case, probably a ClassNotFoundException).
So how do we make sure that we see the real problem, and not this misdirection? We try:
try {
/*
* Move all your existing code here...
*/
} catch (e) {
print("Error generating dynamic PDF: " + e.toString());
} finally {
facesContext.responseComplete();
}
There are two reasons this is a preferred approach:
If something goes wrong with our code, we don't let Domino throw an exception about it. Instead, we log it (instead of using print to send it to the console and log, you could also toss it to OpenLog, or whatever your preferred logging mechanism happens to be). This means that Domino doesn't try to report the error to the user, because we've promised that we already reported it to ourselves.
By moving the crucial facesContext.responseComplete() call (which is what ultimately tells Domino not to send any content of its own) to the finally block, this ensures it will get executed. If we left it inside the try block, it would get skipped if an exception occurs, because we'd skip straight to the catch... so even though Domino isn't reporting our exception because we caught it, it still tries to invoke the response writer because we didn't tell it not to.
If you follow the above pattern, and something's wrong with your code, then the browser will receive an incomplete or corrupt file, but the log will tell you what went wrong, rather than reporting an error that has nothing to do with the root cause of the problem.
I almost deleted this question, but decided to answer it myself since there is very little out on google when you search for the exception.
The issue was in the xAgent, there is a line importPackage that was incorrect. Fixing this made everything work. The exception verbage: "Can't get a Writer while an OutputStream is already in use" is quite misleading. I don't know what else triggers this exception, but an alternative description would be "Java class ??yourClass?? not found"
If you found this question, then you likely have the same issue. I would ignore what the exception actually says, and check your package statements throughout your application. The java code will error on its own, but your SSJS that references the java will not error until runtime, focus on that code.
Update the response header after the body can solve this kind of problem, example :
HttpServletResponse response = (HttpServletResponse) facesContext.getExternalContext().getResponse();
response.getWriter().write("<html><body>...</body></html>");
response.setContentType("text/html");
response.setHeader("Cache-Control", "no-cache");
response.setCharacterEncoding("UTF-8");

Can MessageChannel overflow

I am working on an AS3 project in FDT6. I am using the lastest FLEX 4.6 and AIR 3.7.
I have a worker.swf file that is embedded into the main application to do threading work with.
I am using the MessageChannel class to pass information between the two.
In my main class I have defined
private var mainToWorker:MessageChannel;
private var workerToMain:MessageChannel;
mainToWorker = Worker.current.createMessageChannel(worker);
workerToMain = worker.createMessageChannel(Worker.current);
on the mainToWorker I only ever send messages. In these messages I send a byte array of information. The information is an object that contains a 'command' property and a 'props' property. Basically acting like a function call. The command is a function name and the props is an object that contains data for that function.
mainToWorkerMutex.lock();
mainToWorker.send(ByteArrayUtils.ObjectToByteArray({command:"DoSomething", props:{propA:1,propB:7}}));
mainToWorkerMutex.unlock();
The same occurs for the workerToMain var except I only send byte data that contains the 'message' and 'props' parameters.
workerToMainMutex.lock();
workerToMain.send(ByteArrayUtils.ObjectToByteArray({command:"complete", props:{return:"result"}}));
workerToMainMutex.unlock();
As a sanity check I make sure that the message channels are getting what they should.
It is working fine when I build it in FDT, however when it is built using an ANT script through flash builder I am sometimes getting the 'command' events coming back through in the workerToMain channel.
I am sending quite a lot of data through the message channel. Is it possible that I am overloading it and causing a buffer overflow into the other message channel somehow? How could that only be happening in FB?
I have checked my code many times and I am sure there is nothing in my own code that is sending that message back.
I had similar issue. When sending many bytearrays using channels sometimes things i received was not things i've actually sended. I had 4 channels (message channel to worker, message channel to main, data channel to worker, data channel to main).
I've noticed that data channel to main was affecting message channel to worker. When i turned off data channel to main - message channel to worker stared working just fine :D...
They have a big issue there with sending byte arrays it seems.
But what helped me was using shareable (at first it was not shareable) bytearray for communication via channels, but only for communication, as soon as i am receiving such bytearray i'm copying it to another byte array and parsing a copy.
This removed the problem (made quite hard stress tests there)...
Cheers
P.S. I'm also using static functions (like your ByteArrayUtils) to create bytearray's used for communication, but it seems fine, even made tests using non static functions.
So, it looks like I have found the issue. Looks like it's the ByteArray that is doing it.
ByteArray.toString() is basically sometimes mangles your data meaning you can't really trust it.
http://www.actionscript.org/forums/showthread.php3?t=155067
If you read the comment by "Jim Freer" he mentions how strings sometimes do this.
My solution was to switch to using a JSON encoded string instead of ByteArray data in the message channel. The reason I was using bytearray data to begin with is because I wanted to preserve class definition information, which JSON doesn't do.

What exactly does "Response already committed" mean? How to handle exceptions then?

I know writing business logic in getters and setters is a very bad programming practice, but is there any way to handle exceptions if the response is already committed?
What exactly is the meaning of "Response already committed" and "Headers are already sent to the client"?
There's no nice way to handle exceptions if the response is already committed. The HTTP response exist basically of a header and a body. The headers basically instruct the client (the webbrowser) how exactly it should deal with the response, e.g. the content type, the content length, the character encoding, the body encoding, the cache instructions, etcetera.
You can see the headers in the HTTP traffic monitor of the webbrowser's developer toolset. Press F12 in Chrome/IE9+/Firefox23+ and check the "Network" tab. The below screenshow is what my Chrome shows on your current question:
(note: the "Response" tab shows the response body)
The response body is the actual content, usually in flavor of a bunch of HTML code. The server has usually a fixed size buffer to write the response to. The buffer size depends on server make/version and configuration and is usually 2KB~10KB. If this buffer overflows, then it will be flushed to the other end of the connection, the client. This is the commit of a response. The client has already obtained the first part of the response, usually already representing the whole bunch of headers and maybe a part of the body.
The commit of a response is a point of no return. The server cannot take the already sent bytes back. It's too late to change the response headers (for example, a redirect is basically instructed by a Location header with therein the new URL), let alone the response body. Best what you can do is to append the error information to the already written response body. But this may end up in some weird looking HTML as it's not known which HTML tags needs to be closed at that point. The browser may fail to present it in a proper manner.
Apart from avoiding business logic in getters so that the exceptions are not thrown while rendering the response, another way to avoid an already committed response is to configure the response buffer size to be as large as the largest page which your webapp can serve. How to do that depends on the server make/version. In Tomcat for example, you can configure it as bufferSize attribute of the <Connector> element. Note that this won't prevent from flushing if your own code is (implicitly) calling flush() on the response output stream.
Good exlanation BalusC and I would add that primefaces has an issue in their exception handler. They try to redirect to error page after request was already committed. And as you said the only solution I found is to add some extra content to the response body. I owerride the handler and add this code
if ( extContext.isResponseCommitted() ) {
PartialResponseWriter writer = context.getPartialViewContext().getPartialResponseWriter();
writer.startElement( "script", null );
writer.write( "window.location.href = '" + errorPageUrl + "';" );
writer.endElement( "script" );
writer.getWrapped().endCDATA();
writer.endElement( "update" );
writer.getWrapped().endDocument();
}
else {
extContext.redirect( errorPageUrl );
context.responseComplete();
}

Netty: Pipe-ing output of one Channel to the input of an other

Netty-Gurus,
I've been wondering if there is a shortcut/Netty-Utility/smart-trick
for connecting the input of one Channel to the output of
an other channel. In more details consider the following:
Set-Up a Netty (http) server
For an incoming MessageEvent get its ChannelBuffer
and pipe its input to a NettyClient-ChannelBuffer
(which is to be set up along the lines of the NettyServer).
I'm interested in how to achieve bullet-point 3. since my first
thoughts along the lines
// mock messageReceived(ChannelHandlerContext ctx, MessageEvent e):
ChannelBuffer bufIn = (ChannelBuffer) e.getMessage();
ChannelBuffer bufOut = getClientChannelBuffer();// Set-up somewhere else
bufOut.write(bufIn);
seem to me awkward because
A. I have to determine for each and every messageReceived-Event
the target ChannelBuffer
B. To much Low-Level tinkering
My wish/vision would be to connect
--> the input of one Channel
--> to the output of an other channel
and let them do their I/O without any additional coding.
Many thanks in advance!,
Traude
P.S: Issue has arisen as I'm trying to dispatch the various HTTP-requests to the
server (one entry point) to several other servers, depending on
the input content (mapping based on the first HTTP request line).
Obviously, I also need to do the inverse trick -- pipeing back client
to server -- but I guess it'll be similar to the solution of
the question before.
Looks like you need to use a multiplexer in you business handler. The business handler could have a map. With key as "first http request line" and value as the output channel for the server. Once you do a lookup you just do a channel.write(channelBuffer);
Also take a look at bruno de carvalho's tcp tunnel, which may give you more ideas on how to deal with these kind of requirements.

Resources