I am "piping" a json feed (in some cases quite big) that is returned from an external service, to hide an access api-key to the client (the access key is the only available authentication system for that service).
I am using Gaelyk and I wrote this groovlet:
try {
feed(params.topic)
} catch(Exception e) {
redirect "/failure"
}
def feed(topic) {
URL url = new URL("https://somewhere.com/$topic/<apikey>/feed")
def restResponse = url.get()
if (restResponse.responseCode == 200) {
response.contentType = 'application/json'
out << restResponse.text
}
}
The only problem is that the "restResponse" is very big and the value returned by the groovlet is truncated. So I will get back a json like this:
[{"item":....},{"item":....},{"item":....},{"ite
How can I return the complete json without any truncation?
Well I found the solution and the problem was at the beginning (the URL content must be read as stream). So the content truncated it was not the output but the input:
def feed(topic) {
URL url = "https://somewhere.com/$topic/<apikey>/feed".toURL()
def restResponse = url.get()
if (restResponse.responseCode == 200) {
response.contentType = 'application/json'
StringBuffer sb = new StringBuffer()
url.eachLine {
sb << it
}
out << sb.toString()
}
}
Related
I am exploring Microsoft Computer Vision's Read API (asyncBatchAnalyze) for extracting text from images. I found some sample code on Microsoft site to extract text from images asynchronously.It works in following way:
1) Submit image to asyncBatchAnalyze API.
2) This API accepts the request and returns a URI.
3) We need to poll this URI to get the extracted data.
Is there any way in which we can trigger some notification (like publishing an notification in AWS SQS or similar service) when asyncBatchAnalyze is done with image analysis?
public class MicrosoftOCRAsyncReadText {
private static final String SUBSCRIPTION_KEY = “key”;
private static final String ENDPOINT = "https://computervision.cognitiveservices.azure.com";
private static final String URI_BASE = ENDPOINT + "/vision/v2.1/read/core/asyncBatchAnalyze";
public static void main(String[] args) {
CloseableHttpClient httpTextClient = HttpClientBuilder.create().build();
CloseableHttpClient httpResultClient = HttpClientBuilder.create().build();;
try {
URIBuilder builder = new URIBuilder(URI_BASE);
URI uri = builder.build();
HttpPost request = new HttpPost(uri);
request.setHeader("Content-Type", "application/octet-stream");
request.setHeader("Ocp-Apim-Subscription-Key", SUBSCRIPTION_KEY);
String image = "/Users/xxxxx/Documents/img1.jpg";
File file = new File(image);
FileEntity reqEntity = new FileEntity(file);
request.setEntity(reqEntity);
HttpResponse response = httpTextClient.execute(request);
if (response.getStatusLine().getStatusCode() != 202) {
HttpEntity entity = response.getEntity();
String jsonString = EntityUtils.toString(entity);
JSONObject json = new JSONObject(jsonString);
System.out.println("Error:\n");
System.out.println(json.toString(2));
return;
}
String operationLocation = null;
Header[] responseHeaders = response.getAllHeaders();
for (Header header : responseHeaders) {
if (header.getName().equals("Operation-Location")) {
operationLocation = header.getValue();
break;
}
}
if (operationLocation == null) {
System.out.println("\nError retrieving Operation-Location.\nExiting.");
System.exit(1);
}
/* Wait for asyncBatchAnalyze to complete. In place of this wait, can we trigger any notification from Computer Vision when the extract text operation is complete?
*/
Thread.sleep(5000);
// Call the second REST API method and get the response.
HttpGet resultRequest = new HttpGet(operationLocation);
resultRequest.setHeader("Ocp-Apim-Subscription-Key", SUBSCRIPTION_KEY);
HttpResponse resultResponse = httpResultClient.execute(resultRequest);
HttpEntity responseEntity = resultResponse.getEntity();
if (responseEntity != null) {
String jsonString = EntityUtils.toString(responseEntity);
JSONObject json = new JSONObject(jsonString);
System.out.println(json.toString(2));
}
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
There is no notification / webhook mechanism on those asynchronous operations.
The only thing that I can see right know is to change the implementation you mentioned by using a while condition which is checking regularly if the result is there or not (and a mechanism to cancel waiting - based on maximum waiting time or number of retries).
See sample in Microsoft docs here, especially this part:
// If the first REST API method completes successfully, the second
// REST API method retrieves the text written in the image.
//
// Note: The response may not be immediately available. Text
// recognition is an asynchronous operation that can take a variable
// amount of time depending on the length of the text.
// You may need to wait or retry this operation.
//
// This example checks once per second for ten seconds.
string contentString;
int i = 0;
do
{
System.Threading.Thread.Sleep(1000);
response = await client.GetAsync(operationLocation);
contentString = await response.Content.ReadAsStringAsync();
++i;
}
while (i < 10 && contentString.IndexOf("\"status\":\"Succeeded\"") == -1);
if (i == 10 && contentString.IndexOf("\"status\":\"Succeeded\"") == -1)
{
Console.WriteLine("\nTimeout error.\n");
return;
}
// Display the JSON response.
Console.WriteLine("\nResponse:\n\n{0}\n",
JToken.Parse(contentString).ToString());
Can someone tell me why I keep getting a read and write timeout on this function? I have this as a code behind function on click even from a button. Everything as far as the data looks good until I get to the stream section and it still steps through, but when I check the Stream object contents after stepping into that object it states Read Timeout/Write Timeout: System invalid Operation Exception.
protected void SubmitToDB_Click(object sender, EventArgs e)
{
if (FileUploader.HasFile)
{
try
{
if (SectionDropDownList.SelectedValue != null)
{
if (TemplateDropDownList.SelectedValue != null)
{
// This gets the full file path on the client's machine ie: c:\test\myfile.txt
string strFilePath = FileUploader.PostedFile.FileName;
//use the System.IO Path.GetFileName method to get specifics about the file without needing to parse the path as a string
string strFileName = Path.GetFileName(strFilePath);
Int32 intFileSize = FileUploader.PostedFile.ContentLength;
string strContentType = FileUploader.PostedFile.ContentType;
//Convert the uploaded file to a byte stream to save to your database. This could be a database table field of type Image in SQL Server
Stream strmStream = FileUploader.PostedFile.InputStream;
Int32 intFileLength = (Int32)strmStream.Length;
byte[] bytUpfile = new byte[intFileLength + 1];
strmStream.Read(bytUpfile, 0, intFileLength);
strmStream.Close();
saveFileToDb(strFileName, intFileSize, strContentType, bytUpfile); // or use FileUploader.SaveAs(Server.MapPath(".") + "filename") to save to the server's filesystem.
lblUploadResult.Text = "Upload Success. File was uploaded and saved to the database.";
}
}
}
catch (Exception err)
{
lblUploadResult.Text = "The file was not updloaded because the following error happened: " + err.ToString();
}
}
else
{
lblUploadResult.Text = "No File Uploaded because none was selected.";
}
}
Try something like this:
using (var fileStream = FileUploader.PostedFile.InputStream)
{
using (var reader = new BinaryReader(fileStream))
{
byte[] bytUpfile = reader.ReadBytes((Int32)fileStream.Length);
// SAVE TO DB...
}
}
I am using the sample code from the Unity website for the WWW class to make an API request but the text response is garbage. It looks like ����. When I log the response headers, I get a 200 response and everything seems ok except that the CONTENT-TYPE is image/jpeg. I have tried several different random .json files to test it out and they all return the same thing. Requesting an image to use as a texture does work.
public class SpeechReq : MonoBehaviour {
//public string url = "https://gist.githubusercontent.com/wethecode/1f79baf168680afb0f2d/raw/755f9fb71dcc34df811b4bc26448d88a0f97f34d/snippets.json";
public string url = "https://gist.githubusercontent.com/damienh/fea91ab710475d499a09/raw/893065428badd8bfdc7b39fe17675b8aa031ac51/gistfile1.json";
IEnumerator Start()
{
WWW www = new WWW(url);
yield return www;
string respText = www.text;
Debug.Log(respText);
//Output: ����
byte[] resp = www.bytes;
var str = System.Text.Encoding.Default.GetString(resp);
Debug.Log(str);
//Output: ÿØÿà
if (www.responseHeaders.Count > 0)
{
foreach (KeyValuePair<string, string> entry in www.responseHeaders)
{
Debug.Log(entry.Value + "=" + entry.Key);
//Output: HTTP/1.0 200 OK=STATUS
//...
//image/jpeg=CONTENT-TYPE
}
}
}
}
The WWW class's .text method returns the UTF8 Byte Order Mark at the beginning of the response by default. See a description of the BOM here
You could try:
string jsonText = "";
if (string.IsNullOrEmpty(www.error))
{
jsonText = System.Text.Encoding.UTF8.GetString(www.bytes, 3, www.bytes.Length - 3); // Skip the UTF8 BOM
JSONObject myObject = new JSONObject(jsonText);
}
Response of QNetworkReply::readAll is empty when QNetworkReply::error() != NoError.
Is this normal?
I've a Node+Express server that always send a detailed description in case of http status different of 200; I cant get this description from my Qt client base on QNAM. Qt version is 5.3, OS Win 7 64b.
This is my code, really I don't think this can help.
PendingRequest *Foo::sendMsg(QStandardItem *requestItem, HTTP_METHOD_ID method, QString path)
{
PendingRequest *pReq = new PendingRequest(method);
QString url = QString("https://%1:%2%3").arg(host, QString::number(port), path);
QNetworkRequest qNetReq = QNetworkRequest(QUrl(url));
//set headears
qNetReq.setHeader(QNetworkRequest::KnownHeaders::UserAgentHeader, HttpUserAgent);
qNetReq.setRawHeader("Connection", "Keep-Alive");
if(!credentials.isEmpty())
{
qNetReq.setRawHeader("Authorization", QByteArray("Basic ")+credentials);
}
if(!sessionId.isEmpty())
{
qNetReq.setRawHeader("Session-Id", sessionId);
}
//send request
QNetworkReply *reply;
if(method == HTTP_METHOD_ID::POST)
{
qNetReq.setHeader(QNetworkRequest::KnownHeaders::ContentTypeHeader, "application/json");
QByteArray data = outHandlerList[outHandlerIndex](requestItem);
reply = netManager.post(qNetReq, data);
}
else
{
reply = netManager.get(qNetReq);
}
connect(reply, SIGNAL(finished()), this, SLOT(handleResponse()));
connect(reply, SIGNAL(sslErrors(QList<QSslError>)), this, SLOT(handleSslErrors(QList<QSslError>)));
return pReq;
}
and this where I handle response:
void Foo::handleResponse()
{
QNetworkReply *reply = qobject_cast<QNetworkReply *>(sender());
if(reply->hasRawHeader("Session-Id"))
sessionId = reply->rawHeader("Session-Id");
PendingRequest *pReq = pendingRequestMap.contains(reply) ? pendingRequestMap.take(reply) : 0;
Q_ASSERT(pReq);
QStandardItem *responseItem = 0;
QString error;
if(reply->error() != QNetworkReply::NoError)
{
qDebug() << "readAll: ", reply->readAll(), "error: ", reply->errorString();
error = reply->errorString();
}
else
{
responseItem = inHandlerList[pReq->inHandlerIndex](reply, error, pReq);
}
reply->deleteLater();
}
Thanks a lot for your help.
From the documentation:
QByteArray QIODevice::readAll()
This is an overloaded function.
Reads all available data from the device, and returns it as a
QByteArray.
This function has no way of reporting errors; returning an empty
QByteArray() can mean either that no data was currently available for
reading, or that an error occurred.
I had the same problem, spent hours debugging and searching online for solutions...
The resolution is to connect QIODevice::readyRead to as per the documentation:
connect(reply, &QIODevice::readyRead, this, [=]() {
QByteArray response = reply->readAll();
qDebug() << response;
});
You now have the server reply regardless of the HTTP error code.
I have the following code in groovy
HTTPBuilder http = new HTTPBuilder("https://ronna-afghan.harmonieweb.org/_layouts/searchrss.aspx")
http.request(Method.GET, groovyx.net.http.ContentType.XML) {
// set username and password for basic authentication
// set username and password for basic auth
//http.auth.basic(ConfigurationHolder.config.passportService.userName,
// ConfigurationHolder.config.passportService.password)
headers.'User-Agent' = 'Mozilla/5.0'
uri.query = [k:'execution']
// response handler for a success response code:
response.success = {resp, xml ->
println resp.statusLine
log.debug "response status: ${resp.statusLine}"
log.debug xml.toString()
}
// handler for any failure status code:
response.failure = {resp ->
log.error " ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
}
}
when I run the code, it doesn't give me the rss feed which I'm suppose to get
When I have the same code in java
try {
// Create a URLConnection object for a URL
URL oracle = new URL(
"https://ronna-afghan.harmonieweb.org/_layouts/srchrss.aspx?k=execution&count=1&format=rss");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
in.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
it returns the xml Rss. I can't figure what the issue might be. Everything looks okay to me in the groovy code and also the Http return code is 200.
The code that you have described in Java is the equivalent of the following code in Groovy:
def oracle = "https://ronna-afghan.harmonieweb.org/_layouts/srchrss.aspx?k=execution&count=1&format=rss".toURL().text