I am working on video streaming using RTSP protocol. Sometimes but not always, I observe streaming failure. When the client sends "DESCRIBE", the server responds with the following message.
>> from server [398] >>
RTSP/1.0 200 OK
CSeq: 2
Content-Type: application/sdp
Content-Base: rtsp://10.10.131.1:8554/video/TmMrb5Pigkhf8Ln/bmagic-3/
Server: GStreamer RTSP server
Date: Tue, 30 Apr 2013 20:52:12 GMT
Content-Length: 180
v=0
o=- 1188340656180883 1 IN IP4 10.10.131.1
s=Session streamed with GStreamer
i=rtsp-server
e=NONE
t=0 0
a=tool:GStreamer
a=type:broadcast
a=control:*
a=range:npt=now-
<<<<<
Normally, client will follow with "SETUP", but for a message like that the client seems to stop responding. I am new to RTSP, so to interpret this is hard for me. What could cause the server to send this message? What type of error does it imply such that the client chooses not to proceed?
Thank you so much!
It may be due to connection timeout. I would try to do this on your socket:
setTcpNoDelay(true);
setSoTimeout(60000);
Related
We have an ISAPI module and filter that inspects and modifies responses. We have this scenario where Firefox with HTTP2 enabled sends a request that fails within IIS, and a second request is immediately re-introduced into the pipeline (perhaps resent from the Firefox client). The two requests are very similar except the first one had TE: trailer header and connection:close. When looking at the Failed Request Trace, we see that the flush on the first request fails with 'the parameter is incorrect' (below). Is there a way to track down more information on why the flush failed? I tried to track it down within the managed pipeline but wasn't able to - it seems it may be occurring within native code, or maybe a communication error back to the client(?). If Firefox has HTTP2 disabled, the flush error doesn't occur. If we don't have the ISAPI module and filter, the first request succeeds.
0 ms
Verbose
GENERAL_RESPONSE_ENTITY_BUFFER
Buffer
HTTP/1.1 302 Found
Content-Length: 192
Content-Type: text/html; charset=utf-8
Location: https://SERVER-NAME/VDIR/PATH/FILE.aspx?url=https%3a%2f%2fSERVER-NAME%2fVDIR
Server: Microsoft-IIS/10.0
request-id: b8945a72-543a-4474-9837-9420b3176c5b
X-Powered-By: ASP.NET
X-X-Server: SERVER-NAME
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
0 ms
Informational
GENERAL_FLUSH_RESPONSE_END
BytesSent
0
ErrorCode
The parameter is incorrect.
(0x80070057)
0 ms
GENERAL_REQUEST_END
BytesSent
0
BytesReceived
733
HttpStatus
302
HttpSubStatus
0 ```
Visitors to a website get 500 Internal Server error after browsing for a bit due to a tracking cookie that pushes the overall cookie size for our domain to over 4kb (it's a page view cookie so it appends the page name each time you visit a new page).
I can reproduce the issue using curl with a very large cookie payload. In doing this I've been able to verify where exactly the 500 is coming from (we go from Cloudflare to Varnish to the backend webserver). I've verified that the requests that fail don't make it to the webserver, so I believe Varnish is the one serving up the 500s. I have also watched the varnishlog and seen the 500s come through.
This is an example response from the varnishlog
-- VCL_return hash
-- VCL_call HASH
-- VCL_return lookup
-- Hit 57254162
-- VCL_call HIT
-- VCL_return deliver
-- RespProtocol HTTP/1.1
-- RespStatus 200
-- RespReason OK
-- RespHeader X-Powered-By: Express
-- RespHeader Date: Thu, 01 Aug 2019 23:05:52 GMT
-- RespHeader Content-Type: application/json; charset=utf-8
-- RespHeader Content-Length: 1174
-- RespHeader X-Varnish: 57156196 57519178
-- RespHeader Age: 86
-- RespHeader Via: 1.1 varnish-v4
-- VCL_call DELIVER
-- RespHeader X-Cache: HIT
-- RespUnset X-Powered-By: Express
-- VCL_return deliver
-- Timestamp Process: 1564700838.564547 0.000354 0.000354
-- RespHeader Accept-Ranges: bytes
-- Debug "RES_MODE 2"
-- RespHeader Connection: keep-alive
-- Error workspace_client overflow
-- RespProtocol HTTP/1.1
-- RespStatus 500
-- RespReason Internal Server Error
-- Timestamp Resp: 1564700838.564580 0.000387 0.000033
-- ReqAcct 10063 0 10063 0 0 0
-- End
Here is what I'd added to the vcl_recv section to remove the offending cookie
set req.http.Cookie = regsuball(req.http.Cookie, "_loc_[^;]+(; )?", "");
I don't understand what the significance is of two RespStatus entries here. Why is it 200, and then 500? I've also noticed that if I use curl, which is using HTTP/1.1 I get the 500, but if I use HTTPie, which uses HTTP/2, I get a 200. Is that expected? Would Varnish handle the cookie size differently depending on the http version?
*Edited: I think I've figured out that the difference in the two response statuses are that one is the delivery of the content to varnish, and the second is the delivery of the content to the client.
As the log says, the workspace is too small to accommodate the transaction (headers, notably), try increasing it:
varnishadm param.set workspace_client 128k
For a long explanation: varnish uses a "worspace" for each transaction. This is a chunk of memory used to allocate data, and the whole chunk is wiped out at the end of the transaction. The headers notably are copied into the workspace, and everytime to add or modify a header, it goes there too.
Ths issue here is that you don't have enough space. Earlier version would just panic, but it's now smarter and just produces a synthetic response with a 500 status. The trick is that it realizes the lack of workspace after the initial response has been copied, so you see both responses in the log.
As of Spark 2.2.0, there's are new endpoints in the API for getting information about streaming jobs.
I run Spark on EMR clusters, using Spark 2.2.0 in cluster mode.
When I hit the endpoint for my streaming jobs, all it gives me is the error message:
no streaming listener attached to <stream name>
I've dug through the Spark codebase a bit, but this feature is not very well documented. So I'm curious if this is a bug? Is there some configuration I need to do to get this endpoint working?
This appears to be an issue specifically when running on the cluster. The same code running on Spark 2.2.0 on my local machine shows the statistics as expected, but gives that error message when run on the cluster.
I'm using the very latest Spark 2.3.0-SNAPSHOT built today from the master so YMMV. It worked fine.
Is there some configuration I need to do to get this endpoint working?
No. It's supposed to work fine with no changes to the default configuration.
Make sure the you use the host and port of the driver (as rumors are that you could also access 18080 of Spark History Server that does show all the same endpoints, and the same jobs running, but no streaming listener attached).
As you can see in the source code where the error message lives it can happen only when ui.getStreamingJobProgressListener has not been registered (that ends up in case None).
So the question now is why would that SparkListener not be registered?
That leads us to the streamingJobProgressListener var that is set using setStreamingJobProgressListener method exclusively while StreamingTab is being instantiated (which was the reason why I asked you if you can see the Streaming tab).
In other words, if you see the Streaming tab in web UI, you have the streaming metric endpoint(s) available. Check the URL to the endpoint which should be in the format:
http://[driverHost]:[port]/api/v1/applications/[appId]/streaming/statistics
I tried to reproduce your case and did the following that led me to a working case.
Started one of the official examples of Spark Streaming applications.
$ ./bin/run-example streaming.StatefulNetworkWordCount localhost 9999
I did run nc -lk 9999 first.
Opened the web UI # http://localhost:4040/streaming to make sure the Streaming tab is there.
Made sure http://localhost:4040/api/v1/applications/ responds with application ids.
$ http http://localhost:4040/api/v1/applications/
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 266
Content-Type: application/json
Date: Wed, 13 Dec 2017 07:58:04 GMT
Server: Jetty(9.3.z-SNAPSHOT)
Vary: Accept-Encoding, User-Agent
[
{
"attempts": [
{
"appSparkVersion": "2.3.0-SNAPSHOT",
"completed": false,
"duration": 0,
"endTime": "1969-12-31T23:59:59.999GMT",
"endTimeEpoch": -1,
"lastUpdated": "2017-12-13T07:53:53.751GMT",
"lastUpdatedEpoch": 1513151633751,
"sparkUser": "jacek",
"startTime": "2017-12-13T07:53:53.751GMT",
"startTimeEpoch": 1513151633751
}
],
"id": "local-1513151634282",
"name": "StatefulNetworkWordCount"
}
]
Accessed the endpoint for the Spark Streaming application # http://localhost:4040/api/v1/applications/local-1513151634282/streaming/statistics.
$ http http://localhost:4040/api/v1/applications/local-1513151634282/streaming/statistics
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 219
Content-Type: application/json
Date: Wed, 13 Dec 2017 08:00:10 GMT
Server: Jetty(9.3.z-SNAPSHOT)
Vary: Accept-Encoding, User-Agent
{
"avgInputRate": 0.0,
"avgProcessingTime": 30,
"avgSchedulingDelay": 0,
"avgTotalDelay": 30,
"batchDuration": 1000,
"numActiveBatches": 0,
"numActiveReceivers": 1,
"numInactiveReceivers": 0,
"numProcessedRecords": 0,
"numReceivedRecords": 0,
"numReceivers": 1,
"numRetainedCompletedBatches": 376,
"numTotalCompletedBatches": 376,
"startTime": "2017-12-13T07:53:54.921GMT"
}
TL;DR
Just go to:
http://localhost:4040/streaming
Had a same issue. I ran spark application from Pycharm Python virtual environment. Spark reported that port 4040 was taken:
Spark context Web UI available at http://192.168.100.221:4042
but I saw no jobs there and Streaming tab missing.
Then I went to
http://localhost:4040/streaming
and behold, everything was there.
If you look at the output of PyCharm in the console window it will show what port it used streaming on. I was assuming it was 4040 but when i checked the output carefully the port was on 4041. Here is the output:
WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Then you can use localhost:4041 on any web browser and you should see the streaming output. Hope this helps!
I am working with Pouchdb and Cloudant, and when my web app starts up it does a replication from Cloudant down to my pouchdb in the browser. I have an idea of how pouchdb works internally, and this is how I believe the process works (high level):
Replication starts
Gets a checkpoint doc from cloudant db (contains latest sequence number retrived from server, if not exists, assumes sequence # is 0, which is my case)
Grabs the changes from the changes freed starting at that sequence number (it grabs up to 25 changes)
Writes(or updates) the checkpoint doc back to cloudant server with the new sequence number (this way if a network error occurs, it can continue where it left off or for the next replication)
Repeats until no changes left
Replication complete
The problem is at step 4, that when pouch tries to write that doc to the cloudant server (for the first time), the server returns a 'case_clause' error. I am thinking the issue might be an invalid id sent to cloudant (cloudant doesn't accept ids of this format), because the id of the doc written to the server is _local/799c37dfaefb3774a04f55c7f8cee947 (or other random numbers and characters at the end). I don't know if that is a valid doc id or not (for cloudant that is, because this is accurate for pouchdb), so I guess I am asking, is that the issue (unacceptable id for cloudant), or is there some other issue based on the error the cloudant server returns.
Here is the doc being written:
{
_id: "_local/799c37dfaefb3774a04f55c7f8cee947",
last_seq: "63"
}
Here is the full error output from Chrome debugger:
{
error: "case_clause"
reason: "{{case_clause,{ok,{error,[{{doc,>,
{338,
[>]},
{[{>,>}]},
[],false,[]},
{error,internal_server_error}}]}}},
[{fabric,update_doc,3},{chttpd_db,'-update_doc/6-fun-0-',3}]}"
stack: Array[4]
0: "chttpd_db:update_doc/6"
1: "chttpd:handle_request/1"
2: "mochiweb_http:headers/5"
3: "proc_lib:init_p_do_apply/3"
length: 4
__proto__: Array[0]
status: 500
}
Note: When I go into cloudant's Futon and manually enter the url for the checkpoint doc using its id, it does not exist.
Thanks
EDIT:
Header Info from the above request using Chrome debugger:
Request URL:http://lessontrek.toddbluhm.c9.io/db/ilintindingreseseldropec/_local%2F799c37dfaefb3774a04f55c7f8cee947
Request Method:PUT
Status Code:500 Internal Server Error
Request Headersview parsed
PUT /db/ilintindingreseseldropec/_local%2F799c37dfaefb3774a04f55c7f8cee947 HTTP/1.1
Host: lessontrek.toddbluhm.c9.io
Connection: keep-alive
Content-Length: 111
Accept: application/json
Origin: http://lessontrek.toddbluhm.c9.io
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36
Content-Type: application/json
Referer: http://lessontrek.toddbluhm.c9.io/app
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: connect.sid=s%3A8MVBFmbizTX4VNOqZNtIuxQI.TZ9yKRqNv0ePbTB%2FmSpJsncYszJ8qBSD5EWHzxQYIbg; AuthSession=(removed for security purposes, but valid); db_name=ilintindingreseseldropec; __utma=200306492.386329876.1368934655.1375164160.1375252679.55; __utmc=200306492; __utmz=200306492.1372711539.22.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); c9.live.proxy=(removed for security purposes, but valid)
Request Payloadview parsed
{"_id":"_local/799c37dfaefb3774a04f55c7f8cee947","last_seq":"63","_rev":"338-7db9750558e43e2076a3aa720a6de47b"}
Response Headersview parsed
HTTP/1.1 500 Internal Server Error
x-powered-by: Express
vary: Accept-Encoding
x-couch-request-id: 7d2ca9fc
server: CouchDB/1.0.2 (Erlang OTP/R14B)
date: Wed, 31 Jul 2013 07:29:23 GMT
content-type: application/json
cache-control: must-revalidate
content-encoding: gzip
transfer-encoding: chunked
via: 1.1 project-livec993c2dc8b8c.rhcloud.com (node-web-proxy/0.4)
X-C9-Server: proxy_subdomain_collab-bus2_01
Cloudant, like CouchDB, expects all _local revs to begin "0-". pouchdb should not be generating rev values of this form. If you try this PUT against CouchDB you get the same stack trace.
We have a Java process that fetches resources over HTTP. I discovered that one resource was not pulling correctly even after a client had modified it. Digging into it I found that on the server the process is running the Last-Modified date of the resource does not match what I see when I view info from a browser. I then tried fetching it from a different server and my laptop and both of those showed the correct date.
I've since patched the process to allow the option to Ignore the header date for cases when it exists (but will be incorrect) but I would really love to know why this happens.
For reference here is a curl response from the Server that returns the incorrect info.
HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Fri, 23 Sep 2011 14:16:57 GMT
Content-length: 132
Content-type: text/plain
Last-modified: Wed, 15 Sep 2010 21:58:20 GMT
Etag: "84-4c91417c"
Accept-ranges: bytes
And then the same request on a different server (also get the same results on my machine)
HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Fri, 23 Sep 2011 14:18:47 GMT
Content-length: 132
Content-type: text/plain
Last-modified: Fri, 23 Sep 2011 01:20:43 GMT
Etag: "84-4e7bdeeb"
Accept-ranges: bytes
Both servers are running on Fedora 10.
Can anyone shed some light on this for me and how I might be able to fix this long term?
So you have 2 servers and both return different results, i.e. inconsistency problem (I can basically see this from Etag header)?
My first guess would be caching. Are the any caches active? Maybe the invaliation of cache doesn't work correctly or ttl (time-to-live) settings are too long.
As test have a try and fresh restart machine with stale data and see whether results change (usually during restart of systems most of the simple cache-setups are flushed).
Which kind of backend does the resource come from initially (database, file-system, 3rd party-service)?