Nutch can anyone explain what are status name indicates in readdb stats.
1.db_redir_perm
2.db_unfetched
3.db_fetched
4.db_Gone
5.db_redir_temp
6.db_duplicate
7.db_notmodified.
Nutch store all the metadata information of URLs in CrawlDatum Object. and it is stored in /crawldb/*/part-*/data location
As per the source code of CrawlDatum
/** Page was not fetched yet. */
db_unfetched --> public static final byte STATUS_DB_UNFETCHED = 0x01;
/** Page was successfully fetched. */
db_fetched --> public static final byte STATUS_DB_FETCHED = 0x02;
/** Page no longer exists. */
db_Gone --> public static final byte STATUS_DB_GONE = 0x03;
/** Page temporarily redirects to other page. */
db_redir_temp --> public static final byte STATUS_DB_REDIR_TEMP = 0x04;
/** Page permanently redirects to other page. */
db_redir_perm --> public static final byte STATUS_DB_REDIR_PERM = 0x05;
/** Page was successfully fetched and found not modified. */
db_notmodified --> public static final byte STATUS_DB_NOTMODIFIED = 0x06;
/** Page was marked as being a duplicate of another page */
db_duplicate --> public static final byte STATUS_DB_DUPLICATE = 0x07;
CrawlDatum private byte status; will take one of the values mentioned above depending on the state of URL. (and there are lot of other flags which i'm not discussing now)
When will status value of CrawlDatum(object) change?
There are a lot of flows where it might take one of the several states mentioned above.I will explain a few flows which I'm well aware of.
when we inject URLs into nutch. crawlDb folder is created with each URL CrawlDatum object with state as (db_unfetched). see below code from Injector class
InjectReducer.reduce method.
for (CrawlDatum val : values) {
if (val.getStatus() == CrawlDatum.STATUS_INJECTED) {
injected.set(val);
injected.setStatus(CrawlDatum.STATUS_DB_UNFETCHED);
injectedSet = true;
} else {
old.set(val);
oldSet = true;
}
}
By setting this flag it will be helpful for the generator phase to pick only unfetched urls.
In Fetcher phase if you open FetcherThread source code. crawlDatum status is changed based on url http stats code. you can refer http codes here. (for better understanding)
case ProtocolStatus.MOVED: // redirect
case ProtocolStatus.TEMP_MOVED:
int code;
boolean temp;
if (status.getCode() == ProtocolStatus.MOVED) {
code = CrawlDatum.STATUS_FETCH_REDIR_PERM;
temp = false;
} else {
code = CrawlDatum.STATUS_FETCH_REDIR_TEMP;
temp = true;
}
output(fit.url, fit.datum, content, status, code);
String newUrl = status.getMessage();
Text redirUrl = handleRedirect(fit, newUrl, temp,
Fetcher.PROTOCOL_REDIR);
if (redirUrl != null) {
fit = queueRedirect(redirUrl, fit);
} else {
// stop redirecting
redirecting = false;
}
break;
case ProtocolStatus.EXCEPTION:
logError(fit.url, status.getMessage());
int killedURLs = ((FetchItemQueues) fetchQueues).checkExceptionThreshold(fit
.getQueueID());
if (killedURLs != 0)
context.getCounter("FetcherStatus",
"AboveExceptionThresholdInQueue").increment(killedURLs);
/* FALLTHROUGH */
case ProtocolStatus.RETRY: // retry
case ProtocolStatus.BLOCKED:
output(fit.url, fit.datum, null, status,
CrawlDatum.STATUS_FETCH_RETRY);
break;
case ProtocolStatus.GONE: // gone
case ProtocolStatus.NOTFOUND:
case ProtocolStatus.ACCESS_DENIED:
case ProtocolStatus.ROBOTS_DENIED:
output(fit.url, fit.datum, null, status,
CrawlDatum.STATUS_FETCH_GONE);
break;
case ProtocolStatus.NOTMODIFIED:
output(fit.url, fit.datum, null, status,
CrawlDatum.STATUS_FETCH_NOTMODIFIED);
break;
default:
if (LOG.isWarnEnabled()) {
LOG.warn("{} {} Unknown ProtocolStatus: {}", getName(),
Thread.currentThread().getId(), status.getCode());
}
output(fit.url, fit.datum, null, status,
CrawlDatum.STATUS_FETCH_RETRY);
if (redirecting && redirectCount > maxRedirect) {
((FetchItemQueues) fetchQueues).finishFetchItem(fit);
if (LOG.isInfoEnabled()) {
LOG.info("{} {} - redirect count exceeded {}", getName(),
Thread.currentThread().getId(), fit.url);
}
output(fit.url, fit.datum, null,
ProtocolStatus.STATUS_REDIR_EXCEEDED,
CrawlDatum.STATUS_FETCH_GONE);
}
In deduplication phase if a URLs is found to be duplicate based on md5 hash then it will mark the status as STATUS_DB_DUPLICATE in the deduplication phase and in the next iteration it will not be picked by the Generator.
Related
I am exploring Microsoft Computer Vision's Read API (asyncBatchAnalyze) for extracting text from images. I found some sample code on Microsoft site to extract text from images asynchronously.It works in following way:
1) Submit image to asyncBatchAnalyze API.
2) This API accepts the request and returns a URI.
3) We need to poll this URI to get the extracted data.
Is there any way in which we can trigger some notification (like publishing an notification in AWS SQS or similar service) when asyncBatchAnalyze is done with image analysis?
public class MicrosoftOCRAsyncReadText {
private static final String SUBSCRIPTION_KEY = “key”;
private static final String ENDPOINT = "https://computervision.cognitiveservices.azure.com";
private static final String URI_BASE = ENDPOINT + "/vision/v2.1/read/core/asyncBatchAnalyze";
public static void main(String[] args) {
CloseableHttpClient httpTextClient = HttpClientBuilder.create().build();
CloseableHttpClient httpResultClient = HttpClientBuilder.create().build();;
try {
URIBuilder builder = new URIBuilder(URI_BASE);
URI uri = builder.build();
HttpPost request = new HttpPost(uri);
request.setHeader("Content-Type", "application/octet-stream");
request.setHeader("Ocp-Apim-Subscription-Key", SUBSCRIPTION_KEY);
String image = "/Users/xxxxx/Documents/img1.jpg";
File file = new File(image);
FileEntity reqEntity = new FileEntity(file);
request.setEntity(reqEntity);
HttpResponse response = httpTextClient.execute(request);
if (response.getStatusLine().getStatusCode() != 202) {
HttpEntity entity = response.getEntity();
String jsonString = EntityUtils.toString(entity);
JSONObject json = new JSONObject(jsonString);
System.out.println("Error:\n");
System.out.println(json.toString(2));
return;
}
String operationLocation = null;
Header[] responseHeaders = response.getAllHeaders();
for (Header header : responseHeaders) {
if (header.getName().equals("Operation-Location")) {
operationLocation = header.getValue();
break;
}
}
if (operationLocation == null) {
System.out.println("\nError retrieving Operation-Location.\nExiting.");
System.exit(1);
}
/* Wait for asyncBatchAnalyze to complete. In place of this wait, can we trigger any notification from Computer Vision when the extract text operation is complete?
*/
Thread.sleep(5000);
// Call the second REST API method and get the response.
HttpGet resultRequest = new HttpGet(operationLocation);
resultRequest.setHeader("Ocp-Apim-Subscription-Key", SUBSCRIPTION_KEY);
HttpResponse resultResponse = httpResultClient.execute(resultRequest);
HttpEntity responseEntity = resultResponse.getEntity();
if (responseEntity != null) {
String jsonString = EntityUtils.toString(responseEntity);
JSONObject json = new JSONObject(jsonString);
System.out.println(json.toString(2));
}
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
There is no notification / webhook mechanism on those asynchronous operations.
The only thing that I can see right know is to change the implementation you mentioned by using a while condition which is checking regularly if the result is there or not (and a mechanism to cancel waiting - based on maximum waiting time or number of retries).
See sample in Microsoft docs here, especially this part:
// If the first REST API method completes successfully, the second
// REST API method retrieves the text written in the image.
//
// Note: The response may not be immediately available. Text
// recognition is an asynchronous operation that can take a variable
// amount of time depending on the length of the text.
// You may need to wait or retry this operation.
//
// This example checks once per second for ten seconds.
string contentString;
int i = 0;
do
{
System.Threading.Thread.Sleep(1000);
response = await client.GetAsync(operationLocation);
contentString = await response.Content.ReadAsStringAsync();
++i;
}
while (i < 10 && contentString.IndexOf("\"status\":\"Succeeded\"") == -1);
if (i == 10 && contentString.IndexOf("\"status\":\"Succeeded\"") == -1)
{
Console.WriteLine("\nTimeout error.\n");
return;
}
// Display the JSON response.
Console.WriteLine("\nResponse:\n\n{0}\n",
JToken.Parse(contentString).ToString());
I am using asp,net core and have used the tutorial to create sorted, paged and search page (Index). Once I edit an item from this page the controller always dumps me back to the default index page. How do I return to the previous URL. Many thanks.
Here is a section of my controller file.
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> Edit(int id, Bind("id,UserPassword,user")] UserProfiles userProfiles)
{
var users = from u in _context.UserProfiles
select u;
if (id != userProfiles.id)
{
return NotFound();
}
if (ModelState.IsValid)
{
try
{
_context.Update(userProfiles);
await _context.SaveChangesAsync();
}
catch (DbUpdateConcurrencyException)
{
if (!UserProfilesExists(userProfiles.id))
{
return NotFound();
}
else
{
throw;
}
}
// ***************
// Redirect to the previous URL,i.e. the Index
return Redirect(TempData["PreviousURL"].ToString()) ;
}
return View(userProfiles);
}
public async Task<IActionResult> Index(string sortOrder, string currentFilter, string searchString, int? page)
{
ViewData["CurrentSort"] = sortOrder;
ViewData["NameSortParm"] = String.IsNullOrEmpty(sortOrder) ? "name_desc" : "";
// paging
if (searchString != null)
{
page = 1;
}
else
{
searchString = currentFilter;
}
// search
ViewData["CurrentFilter"] = searchString;
var users = from u in _context.UserProfiles
select u;
if (!String.IsNullOrEmpty(searchString))
{
users = users.Where(u => u.user.Contains(searchString)
);
}
//sort
switch (sortOrder)
{
case "name_desc":
users = users.OrderByDescending(u => u.user);
break;
default:
users = users.OrderBy(s => s.user);
break;
}
// ***************
// store the current path and query string in TempData["PreviousURL" session variable
TempData["PreviousURL"] = HttpContext.Request.Path.ToString() + HttpContext.Request.QueryString.ToString();
return View(await PaginatedList<UserProfiles>.CreateAsync(users.AsNoTracking(), page ?? 1, pageSize));
}
This is my first MVC project.
It depends on your logic where controller takes you after saving data.
You need to pass search, sort and paging related data to controller when saving data. You can send them as part of extra post data, as query string parameters or as part of the model itself which is being posted.
After saving data retrieve data based on those parameters and populater your view with that paged, filtred and sorted data.
I solved my problem with the use of session variables: ViewData, ViewBag and TempData. The following two pages were very useful:
https://www.codeproject.com/Articles/476967/What-is-ViewData-ViewBag-and-TempData-MVC-Option
http://andrewlock.net/an-introduction-to-session-storage-in-asp-net-core/
Please see edited question above for the solution.
I tried this:
<awe:WebControl x:Name="webBrowser" Cursor="None" Source="http://example.com/"/>
but the cursor still shows.
I figured that I could alter the CSS of the page by adding the following line:
*{
cursor: none;
}
But, is there a solution for when I don't have the access to the actual page that I'm showing?
You can use a ResouceInterceptor and manipulate the page on the fly to insert custom CSS.
EDIT:
The following implementation should do the job. (It assumes there is a text.css file)
class ManipulatingResourceInterceptor : IResourceInterceptor
{
public ResourceResponse OnRequest(ResourceRequest request)
{
Stream stream = null;
//do stream manipulation
if (request.Url.ToString() == "http://your.web.url/test.css")
{
WebRequest myRequest;
myRequest = WebRequest.Create(request.Url);
Stream webStream = myRequest.GetResponse().GetResponseStream();
StreamReader webStreamReader = new StreamReader(webStream);
string webStreamContent = webStreamReader.ReadToEnd();
stream = webStream;
string extraContent = "*{cursor: none;}";
webStreamContent += extraContent;
byte[] responseBuffer = Encoding.UTF8.GetBytes(webStreamContent);
// Initialize unmanaged memory to hold the array.
int responseSize = Marshal.SizeOf(responseBuffer[0]) * responseBuffer.Length;
IntPtr pointer = Marshal.AllocHGlobal(responseSize);
try
{
// Copy the array to unmanaged memory.
Marshal.Copy(responseBuffer, 0, pointer, responseBuffer.Length);
return ResourceResponse.Create((uint)responseBuffer.Length, pointer, "text/css");
}
finally
{
// Data is not owned by the ResourceResponse. A copy is made
// of the supplied buffer. We can safely free the unmanaged memory.
Marshal.FreeHGlobal(pointer);
stream.Close();
}
}
return null;
}
public bool OnFilterNavigation(NavigationRequest request)
{
return false;
}
}
I'm making a service that among other has the "photo albums" feature that serve photos to users. User has to be "allowed" to see the photo from the album. So sending the direct link to other person shouldn't allow to view photo.
Photos are stored in the folder outside of the context.
What I need to do is to perform some checks when user requests the photo and then if checks are OK - serve the file. I want to avoid making a wheel and just let tomcat serve the image as it usually does for static files. Can you give some advice on that?
Ok, guys.
After struggling hard with this question I think I've finally found out what to do to solve it. First of all it looks like the question actually decomposes into two independent tasks. One of them is securing access to some resources and second one is feeding resources from the folder outside of the context.
First task is trivial and can be solved by writing a simple filter hanged to "/".
Second task is much less trivial but fortunately also can be resolved. Tomcat uses the implementation of javax.naming.directory.DirContext to load all resources of the given web application including class files. It also allows you to provide the custom implementation of this interface and configure it in the context.xml file. The default implementation is org.apache.naming.resources.FileDirContext. Details here: http://tomcat.apache.org/tomcat-6.0-doc/config/resources.html
I've created my own implementation of DirContext by simply extending FileDirContext. Luckily enough there was a single method that had to be overwritten in order to "hook up" file discovery. The method is called file().
I'm posting my test code here. It is far from perfect and does not take into account the corner cases like renaming files but I don't think that these are needed under a normal run of the server.
The basic idea under this code is to check if the path starts with "virtual directory" prefix and if it is - search for file in the other place in the filesystem (I know there is some duplicate code there but I hope you're not that lazy to remove it if you ever want to use it :-). setVirtualName and setVirtualBase are called automatically to inject the configuration params.
/**
* TODO: add javadocs
*
* #author Juriy Bura
*/
public class VirtualFolderDirContext extends FileDirContext {
private String virtualName;
private String realName;
private File virtualBase;
private String absoluteVirtualBase;
public VirtualFolderDirContext() {
super();
}
public VirtualFolderDirContext(Hashtable env) {
super(env);
}
public void setVirtualName(String path) {
virtualName = path;
}
public void setVirtualBase(String base) {
this.realName = base;
virtualBase = new File(realName);
try {
virtualBase = virtualBase.getCanonicalFile();
} catch (IOException e) {
// Ignore
}
this.absoluteVirtualBase = virtualBase.getAbsolutePath();
}
protected File file(String name) {
File file = null;
boolean virtualFile = name.startsWith(virtualName + "/");
if (virtualFile) {
file = new File(virtualBase, name.substring(virtualName.length()));
} else {
file = new File(base, name);
}
if (file.exists() && file.canRead()) {
if (allowLinking)
return file;
// Check that this file belongs to our root path
String canPath = null;
try {
canPath = file.getCanonicalPath();
} catch (IOException e) {
}
if (canPath == null)
return null;
// Check to see if going outside of the web application root
if (!canPath.startsWith(absoluteBase) && !canPath.startsWith(absoluteVirtualBase)) {
return null;
}
// Case sensitivity check
if (caseSensitive) {
String fileAbsPath = file.getAbsolutePath();
if (fileAbsPath.endsWith("."))
fileAbsPath = fileAbsPath + "/";
String absPath = normalize(fileAbsPath);
if (canPath != null)
canPath = normalize(canPath);
if (virtualFile) {
if ((absoluteVirtualBase.length() < absPath.length())
&& (absoluteVirtualBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteVirtualBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteVirtualBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
} else {
if ((absoluteBase.length() < absPath.length())
&& (absoluteBase.length() < canPath.length())) {
absPath = absPath.substring(absoluteBase.length() + 1);
if ((canPath == null) || (absPath == null))
return null;
if (absPath.equals(""))
absPath = "/";
canPath = canPath.substring(absoluteBase.length() + 1);
if (canPath.equals(""))
canPath = "/";
if (!canPath.equals(absPath))
return null;
}
}
}
} else {
return null;
}
return file;
}
}
After you have this class in place you have to jar it and put that jar into the Tomcat lib folder. For obvious reasons it cannot go together with war file. In your context.xml you should add a config lines like these:
<?xml version="1.0" encoding="UTF-8"?>
<Context antiResourceLocking="true" antiJARLocking="true">
<Resources
className="com.juriy.tomcat.virtualdir.VirtualFolderDirContext"
virtualName="/upload"
virtualBase="c:/temp/up">
</Resources>
...
...
Now any time user asks for /upload/ it will be resolved to c:\temp. With this technique you can implement loading resources from virtually any location: http, shared folder, database, even version control system. So it is pretty cool.
P.S. I've killed the whole day to make this all work together so don't hesitate to give me your vote if you like the answer :-))
Cheers
Juriy
I must create a custom media player within the application with support for mp3 and wav files. I read in the documentation I can't seek or get the media file duration without a custom datasource.
I checked the demo in the JDE 4.6 but I have still problems... I can't get the duration, it returns much more then expected so I'm sure I screwed up something while I modified the code to read the mp3 file locally from the filesystem.
Can somebody tell me what I did wrong? (I can hear the mp3, so the player plays it correctly from start to end)
I must support OSs >= 4.6.
Here is my modified datasource:
/* LimitedRateStreaminSource.java
*
* Copyright © 1998-2009 Research In Motion Ltd.
*
* Note: For the sake of simplicity, this sample application may not leverage
* resource bundles and resource strings. However, it is STRONGLY recommended
* that application developers make use of the localization features available
* within the BlackBerry development platform to ensure a seamless application
* experience across a variety of languages and geographies.
* For more information on localizing your application, please refer to the
* BlackBerry Java Development Environment Development Guide associated with
* this release.
*/
package com.halcyon.tawkwidget.model;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.microedition.io.Connector;
import javax.microedition.io.file.FileConnection;
import javax.microedition.media.Control;
import javax.microedition.media.protocol.ContentDescriptor;
import javax.microedition.media.protocol.DataSource;
import javax.microedition.media.protocol.SourceStream;
import net.rim.device.api.io.SharedInputStream;
/**
* The data source used by the BufferedPlayback's media player.
*/
public final class LimitedRateStreamingSource extends DataSource
{
/** The max size to be read from the stream at one time. */
private static final int READ_CHUNK = 512; // bytes
/** A reference to the field which displays the load status. */
//private TextField _loadStatusField;
/** A reference to the field which displays the player status. */
//private TextField _playStatusField;
/**
* The minimum number of bytes that must be buffered before the media file
* will begin playing.
*/
private int _startBuffer = 200000;
/** The maximum size (in bytes) of a single read. */
private int _readLimit = 32000;
/**
* The minimum forward byte buffer which must be maintained in order for
* the video to keep playing. If the forward buffer falls below this
* number, the playback will pause until the buffer increases.
*/
private int _pauseBytes = 64000;
/**
* The minimum forward byte buffer required to resume
* playback after a pause.
*/
private int _resumeBytes = 128000;
/** The stream connection over which media content is passed. */
//private ContentConnection _contentConnection;
private FileConnection _fileConnection;
/** An input stream shared between several readers. */
private SharedInputStream _readAhead;
/** A stream to the buffered resource. */
private LimitedRateSourceStream _feedToPlayer;
/** The MIME type of the remote media file. */
private String _forcedContentType;
/** A counter for the total number of buffered bytes */
private volatile int _totalRead;
/** A flag used to tell the connection thread to stop */
private volatile boolean _stop;
/**
* A flag used to indicate that the initial buffering is complete. In
* other words, that the current buffer is larger than the defined start
* buffer size.
*/
private volatile boolean _bufferingComplete;
/** A flag used to indicate that the remote file download is complete. */
private volatile boolean _downloadComplete;
/** The thread which retrieves the remote media file. */
private ConnectionThread _loaderThread;
/** The local save file into which the remote file is written. */
private FileConnection _saveFile;
/** A stream for the local save file. */
private OutputStream _saveStream;
/**
* Constructor.
* #param locator The locator that describes the DataSource.
*/
public LimitedRateStreamingSource(String locator)
{
super(locator);
}
/**
* Open a connection to the locator.
* #throws IOException
*/
public void connect() throws IOException
{
//Open the connection to the remote file.
_fileConnection = (FileConnection)Connector.open(getLocator(),
Connector.READ);
//Cache a reference to the locator.
String locator = getLocator();
//Report status.
System.out.println("Loading: " + locator);
//System.out.println("Size: " + _contentConnection.getLength());
System.out.println("Size: " + _fileConnection.totalSize());
//The name of the remote file begins after the last forward slash.
int filenameStart = locator.lastIndexOf('/');
//The file name ends at the first instance of a semicolon.
int paramStart = locator.indexOf(';');
//If there is no semicolon, the file name ends at the end of the line.
if (paramStart < 0)
{
paramStart = locator.length();
}
//Extract the file name.
String filename = locator.substring(filenameStart, paramStart);
System.out.println("Filename: " + filename);
//Open a local save file with the same name as the remote file.
_saveFile = (FileConnection) Connector.open("file:///SDCard"+
"/blackberry/music" + filename, Connector.READ_WRITE);
//If the file doesn't already exist, create it.
if (!_saveFile.exists())
{
_saveFile.create();
}
System.out.println("---------- 1");
//Open the file for writing.
_saveFile.setReadable(true);
//Open a shared input stream to the local save file to
//allow many simultaneous readers.
SharedInputStream fileStream = SharedInputStream.getSharedInputStream(
_saveFile.openInputStream());
//Begin reading at the beginning of the file.
fileStream.setCurrentPosition(0);
System.out.println("---------- 2");
//If the local file is smaller than the remote file...
if (_saveFile.fileSize() < _fileConnection.totalSize())
{
System.out.println("---------- 3");
//Did not get the entire file, set the system to try again.
_saveFile.setWritable(true);
System.out.println("---------- 4");
//A non-null save stream is used as a flag later to indicate that
//the file download was incomplete.
_saveStream = _saveFile.openOutputStream();
System.out.println("---------- 5");
//Use a new shared input stream for buffered reading.
_readAhead = SharedInputStream.getSharedInputStream(
_fileConnection.openInputStream());
System.out.println("---------- 6");
}
else
{
//The download is complete.
System.out.println("---------- 7");
_downloadComplete = true;
//We can use the initial input stream to read the buffered media.
_readAhead = fileStream;
System.out.println("---------- 8");
//We can close the remote connection.
_fileConnection.close();
System.out.println("---------- 9");
}
if (_forcedContentType != null)
{
//Use the user-defined content type if it is set.
System.out.println("---------- 10");
_feedToPlayer = new LimitedRateSourceStream(_readAhead,
_forcedContentType);
System.out.println("---------- 11");
}
else
{
System.out.println("---------- 12");
//Otherwise, use the MIME types of the remote file.
// _feedToPlayer = new LimitedRateSourceStream(_readAhead,
_fileConnection));
}
System.out.println("---------- 13");
}
/**
* Destroy and close all existing connections.
*/
public void disconnect() {
try
{
if (_saveStream != null)
{
//Destroy the stream to the local save file.
_saveStream.close();
_saveStream = null;
}
//Close the local save file.
_saveFile.close();
if (_readAhead != null)
{
//Close the reader stream.
_readAhead.close();
_readAhead = null;
}
//Close the remote file connection.
_fileConnection.close();
//Close the stream to the player.
_feedToPlayer.close();
}
catch (Exception e)
{
System.err.println(e.getMessage());
}
}
/**
* Returns the content type of the remote file.
* #return The content type of the remote file.
*/
public String getContentType()
{
return _feedToPlayer.getContentDescriptor().getContentType();
}
/**
* Returns a stream to the buffered resource.
* #return A stream to the buffered resource.
*/
public SourceStream[] getStreams()
{
return new SourceStream[] { _feedToPlayer };
}
/**
* Starts the connection thread used to download the remote file.
*/
public void start() throws IOException
{
//If the save stream is null, we have already completely downloaded
//the file.
if (_saveStream != null)
{
//Open the connection thread to finish downloading the file.
_loaderThread = new ConnectionThread();
_loaderThread.start();
}
}
/**
* Stop the connection thread.
*/
public void stop() throws IOException
{
//Set the boolean flag to stop the thread.
_stop = true;
}
/**
* #see javax.microedition.media.Controllable#getControl(String)
*/
public Control getControl(String controlType)
{
// No implemented Controls.
return null;
}
/**
* #see javax.microedition.media.Controllable#getControls()
*/
public Control[] getControls()
{
// No implemented Controls.
return null;
}
/**
* Force the lower level stream to a given content type. Must be called
* before the connect function in order to work.
* #param contentType The content type to use.
*/
public void setContentType(String contentType)
{
_forcedContentType = contentType;
}
/**
* A stream to the buffered media resource.
*/
private final class LimitedRateSourceStream implements SourceStream
{
/** A stream to the local copy of the remote resource. */
private SharedInputStream _baseSharedStream;
/** Describes the content type of the media file. */
private ContentDescriptor _contentDescriptor;
/**
* Constructor. Creates a LimitedRateSourceStream from
* the given InputStream.
* #param inputStream The input stream used to create a new reader.
* #param contentType The content type of the remote file.
*/
LimitedRateSourceStream(InputStream inputStream, String contentType)
{
System.out.println("[LimitedRateSoruceStream]---------- 1");
_baseSharedStream = SharedInputStream.getSharedInputStream(
inputStream);
System.out.println("[LimitedRateSoruceStream]---------- 2");
_contentDescriptor = new ContentDescriptor(contentType);
System.out.println("[LimitedRateSoruceStream]---------- 3");
}
/**
* Returns the content descriptor for this stream.
* #return The content descriptor for this stream.
*/
public ContentDescriptor getContentDescriptor()
{
return _contentDescriptor;
}
/**
* Returns the length provided by the connection.
* #return long The length provided by the connection.
*/
public long getContentLength()
{
return _fileConnection.totalSize();
}
/**
* Returns the seek type of the stream.
*/
public int getSeekType()
{
return RANDOM_ACCESSIBLE;
//return SEEKABLE_TO_START;
}
/**
* Returns the maximum size (in bytes) of a single read.
*/
public int getTransferSize()
{
return _readLimit;
}
/**
* Writes bytes from the buffer into a byte array for playback.
* #param bytes The buffer into which the data is read.
* #param off The start offset in array b at which the data is written.
* #param len The maximum number of bytes to read.
* #return the total number of bytes read into the buffer, or -1 if
* there is no more data because the end of the stream has been reached.
* #throws IOException
*/
public int read(byte[] bytes, int off, int len) throws IOException
{
System.out.println("[LimitedRateSoruceStream]---------- 5");
System.out.println("Read Request for: " + len + " bytes");
//Limit bytes read to our readLimit.
int readLength = len;
System.out.println("[LimitedRateSoruceStream]---------- 6");
if (readLength > getReadLimit())
{
readLength = getReadLimit();
}
//The number of available byes in the buffer.
int available;
//A boolean flag indicating that the thread should pause
//until the buffer has increased sufficiently.
boolean paused = false;
System.out.println("[LimitedRateSoruceStream]---------- 7");
for (;;)
{
available = _baseSharedStream.available();
System.out.println("[LimitedRateSoruceStream]---------- 8");
if (_downloadComplete)
{
//Ignore all restrictions if downloading is complete.
System.out.println("Complete, Reading: " + len +
" - Available: " + available);
return _baseSharedStream.read(bytes, off, len);
}
else if(_bufferingComplete)
{
if (paused && available > getResumeBytes())
{
//If the video is paused due to buffering, but the
//number of available byes is sufficiently high,
//resume playback of the media.
System.out.println("Resuming - Available: " +
available);
paused = false;
return _baseSharedStream.read(bytes, off, readLength);
}
else if(!paused && (available > getPauseBytes() ||
available > readLength))
{
//We have enough information for this media playback.
if (available < getPauseBytes())
{
//If the buffer is now insufficient, set the
//pause flag.
paused = true;
}
System.out.println("Reading: " + readLength +
" - Available: " + available);
return _baseSharedStream.read(bytes, off, readLength);
}
else if(!paused)
{
//Set pause until loaded enough to resume.
paused = true;
}
}
else
{
//We are not ready to start yet, try sleeping to allow the
//buffer to increase.
try
{
Thread.sleep(500);
}
catch (Exception e)
{
System.err.println(e.getMessage());
}
}
}
}
/**
* #see javax.microedition.media.protocol.SourceStream#seek(long)
*/
public long seek(long where) throws IOException
{
_baseSharedStream.setCurrentPosition((int) where);
return _baseSharedStream.getCurrentPosition();
}
/**
* #see javax.microedition.media.protocol.SourceStream#tell()
*/
public long tell()
{
return _baseSharedStream.getCurrentPosition();
}
/**
* Close the stream.
* #throws IOException
*/
void close() throws IOException
{
_baseSharedStream.close();
}
/**
* #see javax.microedition.media.Controllable#getControl(String)
*/
public Control getControl(String controlType)
{
// No implemented controls.
return null;
}
/**
* #see javax.microedition.media.Controllable#getControls()
*/
public Control[] getControls()
{
// No implemented controls.
return null;
}
}
/**
* A thread which downloads the remote file and writes it to the local file.
*/
private final class ConnectionThread extends Thread
{
/**
* Download the remote media file, then write it to the local
* file.
* #see java.lang.Thread#run()
*/
public void run()
{
try
{
byte[] data = new byte[READ_CHUNK];
int len = 0;
//Until we reach the end of the file.
while (-1 != (len = _readAhead.read(data)))
{
_totalRead += len;
if (!_bufferingComplete && _totalRead > getStartBuffer())
{
//We have enough of a buffer to begin playback.
_bufferingComplete = true;
System.out.println("Initial Buffering Complete");
}
if (_stop)
{
//Stop reading.
return;
}
}
System.out.println("Downloading Complete");
System.out.println("Total Read: " + _totalRead);
//If the downloaded data is not the same size
//as the remote file, something is wrong.
if (_totalRead != _fileConnection.totalSize())
{
System.err.println("* Unable to Download entire file *");
}
_downloadComplete = true;
_readAhead.setCurrentPosition(0);
//Write downloaded data to the local file.
while (-1 != (len = _readAhead.read(data)))
{
_saveStream.write(data);
}
}
catch (Exception e)
{
System.err.println(e.toString());
}
}
}
/**
* Gets the minimum forward byte buffer which must be maintained in
* order for the video to keep playing.
* #return The pause byte buffer.
*/
int getPauseBytes()
{
return _pauseBytes;
}
/**
* Sets the minimum forward buffer which must be maintained in order
* for the video to keep playing.
* #param pauseBytes The new pause byte buffer.
*/
void setPauseBytes(int pauseBytes)
{
_pauseBytes = pauseBytes;
}
/**
* Gets the maximum size (in bytes) of a single read.
* #return The maximum size (in bytes) of a single read.
*/
int getReadLimit()
{
return _readLimit;
}
/**
* Sets the maximum size (in bytes) of a single read.
* #param readLimit The new maximum size (in bytes) of a single read.
*/
void setReadLimit(int readLimit)
{
_readLimit = readLimit;
}
/**
* Gets the minimum forward byte buffer required to resume
* playback after a pause.
* #return The resume byte buffer.
*/
int getResumeBytes()
{
return _resumeBytes;
}
/**
* Sets the minimum forward byte buffer required to resume
* playback after a pause.
* #param resumeBytes The new resume byte buffer.
*/
void setResumeBytes(int resumeBytes)
{
_resumeBytes = resumeBytes;
}
/**
* Gets the minimum number of bytes that must be buffered before the
* media file will begin playing.
* #return The start byte buffer.
*/
int getStartBuffer()
{
return _startBuffer;
}
/**
* Sets the minimum number of bytes that must be buffered before the
* media file will begin playing.
* #param startBuffer The new start byte buffer.
*/
void setStartBuffer(int startBuffer)
{
_startBuffer = startBuffer;
}
}
And in this way i use it:
LimitedRateStreamingSource source = new
LimitedRateStreamingSource("file:///SDCard/music3.mp3");
source.setContentType("audio/mpeg");
mediaPlayer = javax.microedition.media.Manager.createPlayer(source);
mediaPlayer.addPlayerListener(this);
mediaPlayer.realize();
mediaPlayer.prefetch();
After start I to use mediaPlayer.getDuration it returns lets say around 24:22 (the inbuild media player in the blackberry say the file length is 4:05)
I tried to get the duration in the listener and there it unfortunately returned around 64 minutes, so I'm sure something is not good inside the datasoruce....
code that converts
String getElapsedTimeMinutesSeconds(long elapsedTime) {
long Seconds=(elapsedTime/1000)%60;
long Minutes=(elapsedTime/(1000*60))%60;
long Hours=(elapsedTime/(1000*60*60))%24;
return ""+Minutes + ":"+Seconds;
}
Player.setMediaTime() and Player.getMediaTime() both refer to time in microseconds, not milliseconds. So to get the number of elapsed seconds, you need to divide by 1000000 instead of just 1000.
Sometime earlier I faced with the problem to play unlimited size audio. I fix it.
Here's the link:
http://supportforums.blackberry.com/t5/Java-Development/Use-custom-DataSource-to-play-audio/m-p/1373247#M178928