How to filter on Apache Edgent and also show the values which were filtered? - apache-edgent

I am using Apache Edgent (Java framework) to poll values from a HCSR04 ultrasonic sensor on a Raspberry Pi every 3 seconds. I use a filter to not get values from 50cm to 80cm.
UltrasonicStream sensor = new UltrasonicStream();
DirectProvider dp = new DirectProvider();
Topology topology = dp.newTopology();
TStream<Double> tempReadings = topology.poll(sensor, 3, TimeUnit.SECONDS);
TStream<Double> filteredReadings = tempReadings.filter(reading -> reading < 50 || reading > 80);
System.out.println("filter added: tempReadings.filter(reading -> reading < 50 || reading > 80);");
filteredReadings.print();
dp.submit(topology);
I want to show some message when the values are filtered. When the values do not match with my filter I can poll them, but when they match I am not returning, that is ok. However, I want just to show that a value was filtered using Apache Edgent libraries. I know that I can do something on the public double get() method, but I wonder if I could do this trick with some method of the Apache Edgent.
public class UltrasonicStream implements Supplier {
private static final long serialVersionUID = -6511218542753341056L;
private static GpioPinDigitalOutput sensorTriggerPin;
private static GpioPinDigitalInput sensorEchoPin;
private static final GpioController gpio = GpioFactory.getInstance();
private double currentDistance = -1.0;
/**
* The HCSR04 Ultrasonic sensor is connected on the physical pin 16 and 18 which
* correspond to the GPIO 04 and 05 of the WiringPi library.
*/
public UltrasonicStream() {
// Trigger pin as OUTPUT
sensorTriggerPin = gpio.provisionDigitalOutputPin(RaspiPin.GPIO_04);
// Echo pin as INPUT
sensorEchoPin = gpio.provisionDigitalInputPin(RaspiPin.GPIO_05, PinPullResistance.PULL_DOWN);
}
/**
* This is the override method of the Supplier interface from Apache Edgent
*/
#Override
public Double get() {
try {
System.out.print("Distance in centimeters: ");
currentDistance = getDistance();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return currentDistance;
}
/**
* Retrieve the distance measured by the HCSR04 Ultrasonic sensor connected on a
* Raspberry Pi 3+B
*
* #return the distance in centimeters
* #throws InterruptedException
*/
public double getDistance() throws InterruptedException {
double distanceCM = -1;
try {
// Thread.sleep(2000);
sensorTriggerPin.high(); // Make trigger pin HIGH
Thread.sleep((long) 0.01);// Delay for 10 microseconds
sensorTriggerPin.low(); // Make trigger pin LOW
// Wait until the ECHO pin gets HIGH
while (sensorEchoPin.isLow()) {
}
// Store the current time to calculate ECHO pin HIGH time.
long startTime = System.nanoTime();
// Wait until the ECHO pin gets LOW
while (sensorEchoPin.isHigh()) {
}
// Store the echo pin HIGH end time to calculate ECHO pin HIGH time.
long endTime = System.nanoTime();
distanceCM = ((((endTime - startTime) / 1e3) / 2) / 29.1);
// Printing out the distance in centimeters
// System.out.println("Distance: " + distanceCM + " centimeters");
return distanceCM;
} catch (InterruptedException e) {
e.printStackTrace();
}
return distanceCM;
}
}

You can use TStream.split() to create two streams: one whose tuples match your filter predicate and one for those that don't. You can then do whatever you want with either stream. e.g. TStream.peek(t -> System.out.println("excluded: " + t)), or TStream.print(...)

I implemented like this:
UltrasonicStream sensor = new UltrasonicStream();
DirectProvider dp = new DirectProvider();
Topology topology = dp.newTopology();
TStream<Double> tempReadings = topology.poll(sensor, 3, TimeUnit.SECONDS);
TStream<Double> filteredReadings = tempReadings.filter(reading -> {
boolean threshold = reading < 20 || reading > 80;
if (!threshold) {
System.out.println(String.format("Threshold reached: %s cm", reading));
}
return threshold;
});
filteredReadings.print();
dp.submit(topology);

Related

Using UWP monitor live audio and detect gun-fire/clap sound

I am developing a new UWP app which should monitor sound and fire a event for each sudden sound blow (something like gun fire or clap).
It needs to enable default Audio Input and monitor live audio.
Set audio sensitivity for identifying environment noise and recognizing clap/gun-fire
When there is a high frequency sound like a clap/gun-fire sound (Ideally it should be like configured frequency like +/-40 then it is a gun-fire/clap) then it should call a event.
No need to save Audio
I tried to implement this
SoundMonitoringPage:
public sealed partial class MyPage : Page
{
private async void Page_Loaded(object sender, RoutedEventArgs e)
{
string deviceId = Windows.Media.Devices.MediaDevice.GetDefaultAudioCaptureId(Windows.Media.Devices.AudioDeviceRole.Communications);
gameChatAudioStateMonitor = AudioStateMonitor.CreateForCaptureMonitoringWithCategoryAndDeviceId(MediaCategory.GameChat, deviceId);
gameChatAudioStateMonitor.SoundLevelChanged += GameChatSoundLevelChanged;
//other logic
}
}
Sound Level Change:
private void GameChatSoundLevelChanged(AudioStateMonitor sender, object args)
{
switch (sender.SoundLevel)
{
case SoundLevel.Full:
LevelChangeEvent();
break;
case SoundLevel.Muted:
LevelChangeEvent();
break;
case SoundLevel.Low:
// Audio capture should never be "ducked", only muted or full volume.
Debug.WriteLine("Unexpected audio state change.");
break;
}
}
ENV: windows 10 (v1809) IDE: VS 2017
Not sure if this is the right approach. This is not enabling audio and not hitting the level change event.
I see other options in WinForms & NAudio tutorial here. Probably with Sampling frequency I can check events... Doesn't have must tutorial on using NAudio with UWP to plot the graph and identify the frequency.
Update:
Followed suggestion from #Rob Caplan - MSFT, here is what I ended up with
IMemoryBufferByteAccess.cs
// We are initializing a COM interface for use within the namespace
// This interface allows access to memory at the byte level which we need to populate audio data that is generated
[ComImport]
[Guid("5B0D3235-4DBA-4D44-865E-8F1D0E4FD04D")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe interface IMemoryBufferByteAccess
{
void GetBuffer(out byte* buffer, out uint capacity);
}
GunFireMonitorPage.xaml.cs
public sealed partial class GunFireMonitorPage : Page
{
private MainPage _rootPage;
public static GunFireMonitorPage Current;
private AudioGraph _graph;
private AudioDeviceOutputNode _deviceOutputNode;
private AudioFrameInputNode _frameInputNode;
public double Theta;
public DrivePage()
{
InitializeComponent();
Current = this;
}
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
_rootPage = MainPage.Current;
await CreateAudioGraph();
}
protected override void OnNavigatedFrom(NavigationEventArgs e)
{
_graph?.Dispose();
}
private void Page_Loaded(object sender, RoutedEventArgs e)
{
}
private unsafe AudioFrame GenerateAudioData(uint samples)
{
// Buffer size is (number of samples) * (size of each sample)
// We choose to generate single channel (mono) audio. For multi-channel, multiply by number of channels
uint bufferSize = samples * sizeof(float);
AudioFrame audioFrame = new AudioFrame(bufferSize);
using (AudioBuffer buffer = audioFrame.LockBuffer(AudioBufferAccessMode.Write))
using (IMemoryBufferReference reference = buffer.CreateReference())
{
// Get the buffer from the AudioFrame
// ReSharper disable once SuspiciousTypeConversion.Global
// ReSharper disable once UnusedVariable
((IMemoryBufferByteAccess) reference).GetBuffer(out var dataInBytes, out var capacityInBytes);
// Cast to float since the data we are generating is float
var dataInFloat = (float*)dataInBytes;
float freq = 1000; // choosing to generate frequency of 1kHz
float amplitude = 0.3f;
int sampleRate = (int)_graph.EncodingProperties.SampleRate;
double sampleIncrement = (freq * (Math.PI * 2)) / sampleRate;
// Generate a 1kHz sine wave and populate the values in the memory buffer
for (int i = 0; i < samples; i++)
{
double sinValue = amplitude * Math.Sin(Theta);
dataInFloat[i] = (float)sinValue;
Theta += sampleIncrement;
}
}
return audioFrame;
}
private void node_QuantumStarted(AudioFrameInputNode sender, FrameInputNodeQuantumStartedEventArgs args)
{
// GenerateAudioData can provide PCM audio data by directly synthesizing it or reading from a file.
// Need to know how many samples are required. In this case, the node is running at the same rate as the rest of the graph
// For minimum latency, only provide the required amount of samples. Extra samples will introduce additional latency.
uint numSamplesNeeded = (uint)args.RequiredSamples;
if (numSamplesNeeded != 0)
{
AudioFrame audioData = GenerateAudioData(numSamplesNeeded);
_frameInputNode.AddFrame(audioData);
}
}
private void Button_Click(object sender, RoutedEventArgs e)
{
if (generateButton.Content != null && generateButton.Content.Equals("Generate Audio"))
{
_frameInputNode.Start();
generateButton.Content = "Stop";
audioPipe.Fill = new SolidColorBrush(Colors.Blue);
}
else if (generateButton.Content != null && generateButton.Content.Equals("Stop"))
{
_frameInputNode.Stop();
generateButton.Content = "Generate Audio";
audioPipe.Fill = new SolidColorBrush(Color.FromArgb(255, 49, 49, 49));
}
}
private async Task CreateAudioGraph()
{
// Create an AudioGraph with default settings
AudioGraphSettings settings = new AudioGraphSettings(AudioRenderCategory.Media);
CreateAudioGraphResult result = await AudioGraph.CreateAsync(settings);
if (result.Status != AudioGraphCreationStatus.Success)
{
// Cannot create graph
_rootPage.NotifyUser($"AudioGraph Creation Error because {result.Status.ToString()}", NotifyType.ErrorMessage);
return;
}
_graph = result.Graph;
// Create a device output node
CreateAudioDeviceOutputNodeResult deviceOutputNodeResult = await _graph.CreateDeviceOutputNodeAsync();
if (deviceOutputNodeResult.Status != AudioDeviceNodeCreationStatus.Success)
{
// Cannot create device output node
_rootPage.NotifyUser(
$"Audio Device Output unavailable because {deviceOutputNodeResult.Status.ToString()}", NotifyType.ErrorMessage);
speakerContainer.Background = new SolidColorBrush(Colors.Red);
}
_deviceOutputNode = deviceOutputNodeResult.DeviceOutputNode;
_rootPage.NotifyUser("Device Output Node successfully created", NotifyType.StatusMessage);
speakerContainer.Background = new SolidColorBrush(Colors.Green);
// Create the FrameInputNode at the same format as the graph, except explicitly set mono.
AudioEncodingProperties nodeEncodingProperties = _graph.EncodingProperties;
nodeEncodingProperties.ChannelCount = 1;
_frameInputNode = _graph.CreateFrameInputNode(nodeEncodingProperties);
_frameInputNode.AddOutgoingConnection(_deviceOutputNode);
frameContainer.Background = new SolidColorBrush(Colors.Green);
// Initialize the Frame Input Node in the stopped state
_frameInputNode.Stop();
// Hook up an event handler so we can start generating samples when needed
// This event is triggered when the node is required to provide data
_frameInputNode.QuantumStarted += node_QuantumStarted;
// Start the graph since we will only start/stop the frame input node
_graph.Start();
}
}
GunFireMonitorPage.xaml
<Page
x:Class="SmartPileInspector.xLite.GunFireMonitorPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d" Loaded="Page_Loaded"
HorizontalAlignment="Center"
Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
<ScrollViewer HorizontalAlignment="Center">
<StackPanel HorizontalAlignment="Center">
<!-- more page content -->
<Grid HorizontalAlignment="Center">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"/>
<ColumnDefinition Width="*"/>
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="55"></RowDefinition>
</Grid.RowDefinitions>
</Grid>
<AppBarButton x:Name="generateButton" Content="Generate Audio" Click="Button_Click" MinWidth="120" MinHeight="45" Margin="0,50,0,0"/>
<Border x:Name="frameContainer" BorderThickness="0" Background="#4A4A4A" MinWidth="120" MinHeight="45" Margin="0,20,0,0">
<TextBlock x:Name="frame" Text="Frame Input" VerticalAlignment="Center" HorizontalAlignment="Center" />
</Border>
<StackPanel>
<Rectangle x:Name="audioPipe" Margin="0,20,0,0" Height="10" MinWidth="160" Fill="#313131" HorizontalAlignment="Stretch"/>
</StackPanel>
<Border x:Name="speakerContainer" BorderThickness="0" Background="#4A4A4A" MinWidth="120" MinHeight="45" Margin="0,20,0,0">
<TextBlock x:Name="speaker" Text="Output Device" VerticalAlignment="Center" HorizontalAlignment="Center" />
</Border>
<!--</AppBar>-->
</StackPanel>
</ScrollViewer>
</Page>
There is no graph generated. And there is continuous beep sound with blue line.
Any help is greatly appreciated
Update: Implemented AudioVisualizer
With the help of AudioVisualizer, I was able to plot the lice audio graph.
AudioGraph _graph;
AudioDeviceInputNode _inputNode;
PlaybackSource _source;
SourceConverter _converter;
protected override void OnNavigatedTo(NavigationEventArgs e)
{
_rootPage = MainPage.Current;
_rootPage.SetDimensions(700, 600);
base.OnNavigatedTo(e);
CreateAudioGraphAsync();
}
protected override void OnNavigatedFrom(NavigationEventArgs e)
{
base.OnNavigatedFrom(e);
_graph?.Stop();
_graph?.Dispose();
_graph = null;
}
async void CreateAudioGraphAsync()
{
var graphResult = await AudioGraph.CreateAsync(new AudioGraphSettings(Windows.Media.Render.AudioRenderCategory.Media));
if (graphResult.Status != AudioGraphCreationStatus.Success)
throw new InvalidOperationException($"Graph creation failed {graphResult.Status}");
_graph = graphResult.Graph;
var inputNodeResult = await _graph.CreateDeviceInputNodeAsync(MediaCategory.Media);
if (inputNodeResult.Status == AudioDeviceNodeCreationStatus.Success)
{
_inputNode = inputNodeResult.DeviceInputNode;
_source = PlaybackSource.CreateFromAudioNode(_inputNode);
_converter = new SourceConverter
{
Source = _source.Source,
MinFrequency = 110.0f,
MaxFrequency = 3520.0f,
FrequencyCount = 12 * 5 * 5,
FrequencyScale = ScaleType.Linear,
SpectrumRiseTime = TimeSpan.FromMilliseconds(20),
SpectrumFallTime = TimeSpan.FromMilliseconds(200),
RmsRiseTime = TimeSpan.FromMilliseconds(20),
RmsFallTime = TimeSpan.FromMilliseconds(500),
ChannelCount = 1
};
// Note A2
// Note A7
// 5 octaves, 5 bars per note
// Use RMS to gate noise, fast rise slow fall
NotesSpectrum.Source = _converter;
_graph.Start();
}
else
{
_rootPage.NotifyUser("Cannot access microphone", NotifyType.ErrorMessage);
}
}
Now the challenge is how do I wire an event when wave frequency is above a threshold? In that event I would like to count number of shots, timestamp and it's intensity.
Example Sound
Here is my Recording of live sound, as you can here, when there is that big hammer strike (every second or less), I would like to call a event.
You can find the decibels of a frame by finding the average amplitude of all the pcm data from that frame.I believe you want create a graph that handles the input so that looks like this
private static event LoudNoise<double>;
private static int quantum = 0;
static AudioGraph ingraph;
private static AudioDeviceInputNode deviceInputNode;
private static AudioFrameOutputNode frameOutputNode;
public static async Task<bool> CreateInputDeviceNode(string deviceId)
{
Console.WriteLine("Creating AudioGraphs");
// Create an AudioGraph with default settings
AudioGraphSettings graphsettings = new AudioGraphSettings(AudioRenderCategory.Media);
graphsettings.EncodingProperties = new AudioEncodingProperties();
graphsettings.EncodingProperties.Subtype = "Float";
graphsettings.EncodingProperties.SampleRate = 48000;
graphsettings.EncodingProperties.ChannelCount = 2;
graphsettings.EncodingProperties.BitsPerSample = 32;
graphsettings.EncodingProperties.Bitrate = 3072000;
//settings.DesiredSamplesPerQuantum = 960;
//settings.QuantumSizeSelectionMode = QuantumSizeSelectionMode.ClosestToDesired;
CreateAudioGraphResult graphresult = await AudioGraph.CreateAsync(graphsettings);
if (graphresult.Status != AudioGraphCreationStatus.Success)
{
// Cannot create graph
return false;
}
ingraph = graphresult.Graph;AudioGraphSettings nodesettings = new AudioGraphSettings(AudioRenderCategory.GameChat);
nodesettings.EncodingProperties = AudioEncodingProperties.CreatePcm(48000, 2, 32);
nodesettings.DesiredSamplesPerQuantum = 960;
nodesettings.QuantumSizeSelectionMode = QuantumSizeSelectionMode.ClosestToDesired;
frameOutputNode = ingraph.CreateFrameOutputNode(ingraph.EncodingProperties);
quantum = 0;
ingraph.QuantumStarted += Graph_QuantumStarted;
DeviceInformation selectedDevice;
string device = Windows.Media.Devices.MediaDevice.GetDefaultAudioCaptureId(Windows.Media.Devices.AudioDeviceRole.Default);
if (!string.IsNullOrEmpty(device))
{
selectedDevice = await DeviceInformation.CreateFromIdAsync(device);
} else
{
return false;
}
CreateAudioDeviceInputNodeResult result =
await ingraph.CreateDeviceInputNodeAsync(MediaCategory.Media, nodesettings.EncodingProperties, selectedDevice);
if (result.Status != AudioDeviceNodeCreationStatus.Success)
{
// Cannot create device output node
return false;
}
deviceInputNode = result.DeviceInputNode;
deviceInputNode.AddOutgoingConnection(frameOutputNode);
frameOutputNode.Start();
ingraph.Start();
return true;
}
private static void Graph_QuantumStarted(AudioGraph sender, object args)
{
if (++quantum % 2 == 0)
{
AudioFrame frame = frameOutputNode.GetFrame();
float[] dataInFloats;
using (AudioBuffer buffer = frame.LockBuffer(AudioBufferAccessMode.Write))
using (IMemoryBufferReference reference = buffer.CreateReference())
{
// Get the buffer from the AudioFrame
((IMemoryBufferByteAccess)reference).GetBuffer(out byte* dataInBytes, out uint capacityInBytes);
float* dataInFloat = (float*)dataInBytes;
dataInFloats = new float[capacityInBytes / sizeof(float)];
for (int i = 0; i < capacityInBytes / sizeof(float); i++)
{
dataInFloats[i] = dataInFloat[i];
}
}
double decibels = 0f;
foreach (var sample in dataInFloats)
{
decibels += Math.Abs(sample);
}
decibels = 20 * Math.Log10(decibels / dataInFloats.Length);
// You can pass the decibel value where ever you'd like from here
if (decibels > 10)
{
LoudNoise?.Invoke(this, decibels);
}
}
}
P.S. I did all of this static but naturally it'll work if it's all in the same instance
I also copied this partially from my own project so it may have some parts I forgot to trim. Hope it helps
Answering the "is this the right approach" question: no, the AudioStateMonitor will not help with the problem.
AudioStateMonitor.SoundLevelChanged tells you if the system is ducking your sound so it doesn't interfere with something else. For example, it may mute music in favour of the telephone ringer. SoundLevelChanged doesn't tell you anything about the volume or frequency of recorded sound, which is what you'll need to detect your handclap.
The right approach will be along the lines of using an AudioGraph (or WASAPI, but not from C#) to capture the raw audio into an AudioFrameOutputNode to process the signal and then run that through an FFT to detect sounds in your target frequencies and volumes. The AudioCreation sample demonstrates using an AudioGraph, but not specifically AudioFrameOutputNode.
Per https://home.howstuffworks.com/clapper1.htm clapping will be in a frequency range of 2200Hz to 2800Hz.
Recognizing gunshots looks like it's significantly more complicated, with different guns having very different signatures. A quick search found several research papers on this rather than trivial algorithms. I suspect you'll want some sort of Machine Learning to classify these. Here's a previous thread discussing using ML to differ between gunshots and non-gunshots: SVM for one Vs all acoustic signal classification

Cisco JTAPI phone register/unregister status

I am using the below code to check the phone status(if phone is up or down). When phone is down sends an alarm. However this doesn't show when 8800 series phones are down. Is there any other method to check the Phone register/unregister status?
#Override public void terminalChangedEvent(TermEv[] eventList) {
if ( eventList != null ) {
for (TermEv eventList1 : eventList) {
if (eventList1 instanceof CiscoTermInServiceEv){
if(terminalInService.test()==true){
LogSQL.removeLog(terminal.getName());
}
System.out.println(terminal.getName());
terminalInService.set();
return;
} else if (eventList1 instanceof CiscoTermOutOfServiceEv &&
terminalInService.test()==true) {
offline();
}
}
}
}
Second Question, I was not able to find the methods or documentation about "com.cisco.cti.util.Condition" class. What does Condition.set() and Condition.test() methods do?
Looks like you have the right general idea - JTAPI should work fine for 88xx models, assuming you have the correct device->user association, and user permissions (Standard CTI Enabled, and Standard CTI Allow Control of Phones supporting Connected Xfer and conf needed for 88xx).
Here is my version working on CUCM 11.5:
package com.mycompany.app;
import com.cisco.jtapi.extensions.*;
import java.util.*;
import javax.telephony.*;
import javax.telephony.events.*;
import javax.telephony.callcontrol.*;
import javax.telephony.callcontrol.events.*;
import com.cisco.cti.util.Condition;
public class DataTerm implements ProviderObserver, TerminalObserver {
public static final int OUT_OF_SERVICE = 0;
public static final int IN_SERVICE = 1;
private Address destAddress;
private CiscoTerminal observedTerminal;
private boolean addressInService;
private boolean terminalInService;
protected int state = OUT_OF_SERVICE;
Condition conditionInService = new Condition();
Provider provider;
public DataTerm(String[] args) {
try {
System.out.println("Initializing Jtapi");
String providerName = "ds-ucm115-1.cisco.com";
String login = "dstaudt";
String passwd = "password";
String dest = "2999";
JtapiPeer peer = JtapiPeerFactory.getJtapiPeer(null);
String providerString = providerName + ";login=" + login + ";passwd=" + passwd;
System.out.println("Opening " + providerString + "...\n");
provider = peer.getProvider(providerString);
provider.addObserver(this);
conditionInService.waitTrue();
this.destAddress = provider.getAddress(dest);
this.observedTerminal = (CiscoTerminal) destAddress.getTerminals()[0];
try {
if (destAddress != null) {
System.out.println("Adding Terminal Observer to Terminal" + observedTerminal.getName());
observedTerminal.addObserver(this);
}
} catch (Exception e) {
}
} catch (Exception e) {
System.out.println("Caught exception " + e);
}
}
public void terminalChangedEvent(TermEv[] events) {
for (int i = 0; i < events.length; i++) {
Terminal terminal = events[i].getTerminal();
switch (events[i].getID()) {
case CiscoTermInServiceEv.ID:
System.out.println("Received " + events[i] + "for " + terminal.getName());
terminalInService = true;
break;
case CiscoTermOutOfServiceEv.ID:
System.out.println("Received " + events[i] + "for " + terminal.getName());
terminalInService = false;
if (state != OUT_OF_SERVICE) { // you only want to notify when you had notified earlier that you are IN_SERVICE
state = OUT_OF_SERVICE;
}
break;
}
}
}
public void providerChangedEvent(ProvEv[] eventList) {
if (eventList != null) {
for (int i = 0; i < eventList.length; i++) {
if (eventList[i] instanceof ProvInServiceEv) {
conditionInService.set();
}
}
}
}
}
The "com.cisco.cti.util.Condition" seems to be based on this pattern:
public interface Condition
Condition factors out the Object monitor methods (wait, notify and notifyAll) into distinct objects to give the effect of having multiple wait-sets per object, by combining them with the use of arbitrary Lock implementations. Where a Lock replaces the use of synchronized methods and statements, a Condition replaces the use of the Object monitor methods.
Conditions (also known as condition queues or condition variables) provide a means for one thread to suspend execution (to "wait") until notified by another thread that some state condition may now be true. Because access to this shared state information occurs in different threads, it must be protected, so a lock of some form is associated with the condition. The key property that waiting for a condition provides is that it atomically releases the associated lock and suspends the current thread, just like Object.wait.
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Condition.html

How to retrieve more than 100 messages from the history of a PubNub channel?

The page about the PubNub History API states that
The history() function returns a list of up to 100 messages, the start
time token and the ending time token.
Is there a way to retrieve more than the 100 messages?
I'm currently not a paying customer of PubNub.
PubNub Load History More than 100 Messages
Sometimes you want to slice back in time over a linear stream of data. And often you'll want to do this at different levels of granularity. That is why PubNub Storage and Playback APIs provide maximum level of flexibility. However sometimes it ends up being a bit tricky to load data with the preferred result set.
PubNub Real-Time Network Storage and Playback
There are several considerations you may be seeking when loading transaction history over timelines that can potentially span millions of message in the transaction set. There are some great options available to you and we will cover two of them right now. The examples will be coded in JavaScript. The first example loads a summary of the data by grabbing the snapshots for the beginning of each hour for the past 24 hours. The second example shows you how to load all transactions in full detail and maximum granularity.
All Reference Files can be found on this GIST: Loading History from PubNub Mt.Gox Trades
Example PubNub Mt.Gox History JavaScript Usage
<script src="https://cdn.pubnub.com/pubnub.min.js"></script>
<script src="mtgox-history.js"></script>
<script>(function(){
// LOAD HOURLY SUMMARY
MTGOX.history.hourly({
channel : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
data : function(response) { console.log(JSON.stringify(response)) },
error : function() { console.log("NETWORK ERROR") }
});
// LOAD ALL WITH LIMITER OPTION
MTGOX.history.full({
limit : 500, // SET LIMIT AS HIGH AS NEEDED TO LOAD MORE!
channel : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
data : function(messages) { console.log(messages) },
error : function(e) { console.log("NETWORK ERROR") }
});
})();</script>
NOTE: Running MTGOX.history.hourly() method will generate a list of snapshots per hour over the last 24 hours.
NOTE: Running MTGOX.history.full() method will generate maximum resolution detail with a lot of data. You can get a full dump or partial dump as needed; and you should increase the limit parameter in order to grab more data points.
This following JavaScript file will provide you the MTGOX interface.
PubNub Mt.Gox History JavaScript Loader
//
// mtgox-history.js
//
(function(){
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
// INITIALIZE PUBNUB
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
var pubnub = PUBNUB.init({
subscribe_key : 'sub-c-50d56e1e-2fd9-11e3-a041-02ee2ddab7fe'
});
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
// MTGOX HISTORY INTERFACE
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
window.MTGOX = {
history : {
hourly : hourly,
full : full
}
};
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
// GET ALL DATA FOREVER (WITH LIMIT OF COURSE)
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
/*
MTGOX.history.full({
limit : 1000,
channel : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
data : function(messages) { console.log(messages) },
error : function(e) { console.log("NETWORK ERROR") }
});
*/
function full(args) {
var chan = args['channel'] ||'d5f06780-30a8-4a48-a2f8-7ed181b4a13f'
, callback = args['data'] || function(){}
, error = args['error'] || function(){}
, limit = +args['limit'] || 5000
, start = 0
, count = 100
, history = []
, params = {
channel : chan,
count : count,
callback : function(messages) {
var msgs = messages[0];
start = messages[1];
params.start = start;
PUBNUB.each( msgs.reverse(), function(m) {history.push(m)} );
if (history.length >= limit) return callback(history);
if (msgs.length < count) return callback(history);
count = 100;
add_messages();
},
error : function(e) {
callback(history);
error(history);
}
};
add_messages();
function add_messages() { pubnub.history(params) }
}
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
// GET 24 HOURS IN HOURLY INCREMENTS
// -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
/*
MTGOX.history.hourly({
channel : 'd5f06780-30a8-4a48-a2f8-7ed181b4a13f',
data : function(response) { console.log(response) },
error : function() { console.log('ERROR') }
});
*/
function hourly(setup) {
var limit = 24;
var count = 0;
var chan = setup['channel'] ||'d5f06780-30a8-4a48-a2f8-7ed181b4a13f';
var cb = setup['data'] || function(){};
var eb = setup['error'] || function(){};
var now = new Date();
now.setUTCHours(0);
now.setUTCMinutes(0);
now.setUTCSeconds(0);
now.setUTCMilliseconds(0);
var utc_now = now.getTime();
var vectors = [];
PUBNUB.each( (new Array(limit)).join(',').split(','), function( _, d ) {
var day = utc_now - 3600000 * d;
pubnub.history({
limit : 1,
channel : chan,
start : day * 10000,
error : function() { count++; eb(); },
callback : function(messages) {
// DONE?
if (++count == limit) return cb(vectors);
// ADD TIME SLICES
var res = +(((messages[0][0]||{}).ticker||{}).avg||{}).value;
res && vectors.push([ new Date(day).getUTCHours(), res ]);
// KEEP IT SORTED
vectors.sort(function(a,b){ return a[0] > b[0] && -1 || 1 });
}
})
} );
}
})();
Mt.Gox PubNub Channel Listing for Tickers, Depth and Trades
The following is a list of channels provided by Mt.Gox data feed options you can use in the history channel parameter field.
{
"TICKER.ltcgbp": "0102A446-E4D4-4082-8E83-CC02822F9172",
"TICKER.ltccny": "0290378C-E3D7-4836-8CB1-2BFAE20CC492",
"DEPTH.btchkd": "049F65DC-3AF3-4FFD-85A5-AAC102B2A579",
"DEPTH.btceur": "057BDC6B-9F9C-44E4-BC1A-363E4443CE87",
"TICKER.nmcaud": "08C65460-CBD9-492E-8473-8507DFA66AE6",
"TICKER.btceur": "0BB6DA8B-F6C6-4ECF-8F0D-A544AD948C15",
"DEPTH.btckrw": "0C84BDA7-E613-4B19-AE2A-6D26412C9F70",
"DEPTH.btccny": "0D1ECAD8-E20F-459E-8BED-0BDCF927820F",
"TICKER.btccad": "10720792-084D-45BA-92E3-CF44D9477775",
"DEPTH.btcchf": "113FEC5F-294D-4929-86EB-8CA4C3FD1BED",
"TICKER.ltcnok": "13616AE8-9268-4A43-BDF7-6B8D1AC814A2",
"TICKER.ltcusd": "1366A9F3-92EB-4C6C-9CCC-492A959ECA94",
"TICKER.btcbtc": "13EDFF67-CFA0-4D99-AA76-52BD15D6A058",
"TICKER.ltccad": "18B55737-3F5C-4583-AF63-6EB3951EAD72",
"TICKER.nmccny": "249FDEFD-C6EB-4802-9F54-064BC83908AA",
"DEPTH.btcusd": "24E67E0D-1CAD-4CC0-9E7A-F8523EF460FE",
"TICKER.btcchf": "2644C164-3DB7-4475-8B45-C7042EFE3413",
"DEPTH.btcaud": "296EE352-DD5D-46F3-9BEA-5E39DEDE2005",
"TICKER.btcczk": "2A968B7F-6638-40BA-95E7-7284B3196D52",
"TICKER.btcsgd": "2CB73ED1-07F4-45E0-8918-BCBFDA658912",
"TICKER.nmcjpy": "314E2B7A-A9FA-4249-BC46-B7F662ECBC3A",
"TICKER.btcnmc": "36189B8C-CFFA-40D2-B205-FB71420387AE",
"DEPTH.btcinr": "414FDB18-8F70-471C-A9DF-B3C2740727EA",
"DEPTH.btcsgd": "41E5C243-3D44-4FAD-B690-F39E1DBB86A8",
"TICKER.btcltc": "48B6886F-49C0-4614-B647-BA5369B449A9",
"TICKER.ltceur": "491BC9BB-7CD8-4719-A9E8-16DAD802FFAC",
"TICKER.btcinr": "55E5FEB8-FEA5-416B-88FA-40211541DECA",
"TICKER.ltcjpy": "5AD8E40F-6DF3-489F-9CF1-AF28426A50CF",
"DEPTH.btccad": "5B234CC3-A7C1-47CE-854F-27AEE4CDBDA5",
"TICKER.btcnzd": "5DDD27CA-2466-4D1A-8961-615DEDB68BF1",
"DEPTH.btcgbp": "60C3AF1B-5D40-4D0E-B9FC-CCAB433D2E9C",
"DEPTH.btcnok": "66DA7FB4-6B0C-4A10-9CB7-E2944E046EB5",
"DEPTH.btcthb": "67879668-532F-41F9-8EB0-55E7593A5AB8",
"TICKER.btcsek": "6CAF1244-655B-460F-BEAF-5C56D1F4BEA7",
"TICKER.btcnok": "7532E866-3A03-4514-A4B1-6F86E3A8DC11",
"TICKER.btcgbp": "7B842B7D-D1F9-46FA-A49C-C12F1AD5A533",
"TRADE.LAG": "85174711-BE64-4DE1-B783-0628995D7914",
"DEPTH.btcsek": "8F1FEFAA-7C55-4420-ADA0-4DE15C1C38F3",
"DEPTH.btcdkk": "9219ABB0-B50C-4007-B4D2-51D1711AB19C",
"DEPTH.btcjpy": "94483E07-D797-4DD4-BC72-DC98F1FD39E3",
"TICKER.nmcusd": "9AAEFD15-D101-49F3-A2FD-6B63B85B6BED",
"TICKER.ltcaud": "A046600A-A06C-4EBF-9FFB-BDC8157227E8",
"TICKER.btcjpy": "A39AE532-6A3C-4835-AF8C-DDA54CB4874E",
"DEPTH.btcczk": "A7A970CF-4F6C-4D85-A74E-AC0979049B87",
"TICKER.ltcdkk": "B10A706E-E8C7-4EA8-9148-669F86930B36",
"TICKER.btcpln": "B4A02CB3-2E2D-4A88-AEEA-3C66CB604D01",
"TEST": "BAD99F24-FA8B-4938-BFDF-0C1831FC6665",
"TICKER.btcrub": "BD04F720-3C70-4DCE-AE71-2422AB862C65",
"TICKER.nmcgbp": "BF5126BA-5187-456F-8AE6-963678D0607F",
"TICKER.btckrw": "BF85048D-4DB9-4DBE-9CA3-5B83A1A4186E",
"TICKER.btccny": "C251EC35-56F9-40AB-A4F6-13325C349DE4",
"DEPTH.btcnzd": "CEDF8730-BCE6-4278-B6FE-9BEE42930E95",
"TICKER.btchkd": "D3AE78DD-01DD-4074-88A7-B8AA03CD28DD",
"TICKER.btcthb": "D58E3B69-9560-4B9E-8C58-B5C0F3FDA5E1",
"TICKER.btcusd": "D5F06780-30A8-4A48-A2F8-7ED181B4A13F",
"DEPTH.btcrub": "D6412CA0-B686-464C-891A-D1BA3943F3C6",
"TICKER.nmceur": "D8512D04-F262-4A14-82F2-8E5C96C15E68",
"TRADE.btc": "DBF1DEE9-4F2E-4A08-8CB7-748919A71B21",
"TICKER.nmccad": "DC28033E-7506-484C-905D-1C811A613323",
"DEPTH.btcpln": "E4FF055A-F8BF-407E-AF76-676CAD319A21",
"TICKER.btcdkk": "E5CE0604-574A-4059-9493-80AF46C776B3",
"TICKER.btcaud": "EB6AAA11-99D0-4F64-9E8C-1140872A423D"
}
See https://help.pubnub.com/entries/24113341-How-do-I-Page-Through-Stored-Messages-
Contact PubNub support (help#...) if further assistance is needed
The below Java class can be used to easily retrieve and process long time ranges of history messages in an advanced for loop of this form:
PubnubHistoryExcerpt history = new PubnubHistoryExcerpt(pubnub, channel, start, end);
for (Object message : history) {
// do something with the message object
}
The messages are retrieved on the fly, so no memory problem occurs.
Below is the complete code. You can find a fully runnable usage example in the main() method inside of the class.
I haven't yet tested this class extensively. Enhancements are welcome.
/*
* PubnubHistoryExcerpt.java
*
* This file is distributed under the FreeBSD License:
*
* Copyright (c) 2014, Daniel S. (http://stackoverflow.com/users/1838726/daniel-s)
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
* ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* The views and conclusions contained in the software and documentation are those
* of the authors and should not be interpreted as representing official policies,
* either expressed or implied, of the FreeBSD Project.
*/
import java.math.BigDecimal;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.NoSuchElementException;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import com.pubnub.api.Callback;
import com.pubnub.api.Pubnub;
import com.pubnub.api.PubnubError;
/**
* You can use this class to iterate over historical PubNub messages. The messages are retrieved transparently while you
* iterate over the history excerpt. This class and the returned iterators are thread-safe.
*
* See {#link #main(String[])} for a usage example.
*/
public class PubnubHistoryExcerpt implements Iterable<Object> {
/**
* This main method contains a usage example for this class. It downloads last 3 hour's messages of the MtGox BTC/USD ticker
* channel from PubNub and outputs the timestamp and USD value which are found in the messages.
*/
public static void main(String[] args) throws JSONException {
String PUBNUB_SUBSCRIBE_KEY_MTGOX = "sub-c-50d56e1e-2fd9-11e3-a041-02ee2ddab7fe";
String PUBNUB_CHANNEL_MTGOX_TICKER_BTCUSD = "d5f06780-30a8-4a48-a2f8-7ed181b4a13f";
Pubnub pubnub = new Pubnub(null, PUBNUB_SUBSCRIBE_KEY_MTGOX);
long ONE_HOUR_IN_MILLIS = 60 * 60 * 1000;
long end = System.currentTimeMillis();
long start = end - 3 * ONE_HOUR_IN_MILLIS;
// convert from milliseconds as time unit (10^-3 seconds) to
// pubnub's better-than-microsecond precision time units (10^-7 seconds)
start *= 10000;
end *= 10000;
PubnubHistoryExcerpt history = new PubnubHistoryExcerpt(pubnub, PUBNUB_CHANNEL_MTGOX_TICKER_BTCUSD, start, end);
DefaultDateFormat dateFormat = DefaultDateFormat.create();
for (Object message : history) {
JSONObject messageJson = (JSONObject) message;
JSONObject ticker = messageJson.getJSONObject("ticker");
long instant = ticker.getLong("now");
BigDecimal value = new BigDecimal(ticker.getJSONObject("last_local").getString("value"));
instant /= 1000; // convert from microseconds to milliseconds
System.out.println(dateFormat.format(instant) + ": " + value);
}
System.exit(0);
}
/**
* This is the maximum number of messages to fetch in one batch. If you fetch many messages, higher numbers improve
* performance. Setting this to a value higher than 100 doesn't have an effect, because Pubnub currently doesn't
* support fetching more than this many messages at once.
*/
private static final int BATCH_SIZE = 100;
private final Pubnub pubnub;
private final String channel;
private final long start;
private final long end;
/**
* Constructs a new excerpt over which you can iterate. Insances represent an excerpt. No retrieval operations are
* started unless you call iterator().next() for the first time.
*
* #param pubnub
* The Pubnub connection to use for retrieving messages.
* #param channel
* The channel for which to retrieve historical messages.
* #param start
* The beginning of the time interval for which to retrieve messages, in pubnub's time units (10^-7
* seconds, so milliseconds * 10000) since 1970-01-01 00:00:00).
* #param end
* The end of the time interval for which to retrieve messages, in pubnub's time units (10^-7 seconds, so
* milliseconds * 10000) since 1970-01-01 00:00:00).
*/
private PubnubHistoryExcerpt(Pubnub pubnub, String channel, long start, long end) {
this.pubnub = pubnub;
this.channel = channel;
this.start = start;
this.end = end;
}
public Iterator<Object> iterator() {
return new Iter();
}
private class Iter implements Iterator<Object> {
/**
* This list is used as a fifo buffer for messages retrieves through this iterator. It also acts as the main
* synchronization lock for synchronizing access between threads accessing this class as an iterator and threads
* calling back from the Pubnub API.
*/
private LinkedList<Object> buffer = new LinkedList<Object>();
/**
* This field stores the end of the time range of the previous batch retrieval, in Pubnub time units (10th of a
* microsecond, so milliseconds*10000). For the following batch retrieval, this is used as the start time for
* retrieving the following messages.
*/
private long prevBatchTimeRangeEnd = PubnubHistoryExcerpt.this.start;
/**
* Retrieval of messages is handled asynchronously. That means that exceptions which are thrown during retrieval
* can't automatically be propagated through to the code which invokes <code>next()</code> or
* <code>hasNext()</code> . Therefor, such an exception is stored temporarily in this field and then re-thrown
* from within <code>next()</code> or <code>hasNext()</code>.
*/
private Exception caughtDuringRetrieval = null;
/**
* This object is used to wait on and to notify about updates of the buffer.
*/
private Object notifier = new Object();
/**
* Because of spurious wakeups that can happen during wait(), this field is necessary to tell the waiting thread
* if retrieval is still running.
*/
private boolean retrieving = false;
/**
* The callback object to use for retrieving messages. This is stored in a field here for re-use. This is a
* compromise between performance and low memory footprint, slightly in favor of performance.
*/
private InternalCallback internalCallback = new InternalCallback();
private void retrieveNextBatch() {
synchronized (notifier) {
this.retrieving = true;
// String startStr = DefaultDateFormat.create().format(prevBatchTimeRangeEnd / 10000);
// String endStr = DefaultDateFormat.create().format(end / 10000);
// System.out.println("fetching from " + startStr + " till " + endStr);
if (Iter.this.prevBatchTimeRangeEnd < PubnubHistoryExcerpt.this.end) {
PubnubHistoryExcerpt.this.pubnub.history( //
PubnubHistoryExcerpt.this.channel, //
Iter.this.prevBatchTimeRangeEnd, //
PubnubHistoryExcerpt.this.end, //
BATCH_SIZE, //
false, //
Iter.this.internalCallback //
);
waitUntilNextBatchRetrievalFinished();
}
}
}
private void waitUntilNextBatchRetrievalFinished() {
while (this.retrieving) {
try {
this.notifier.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
private class InternalCallback extends Callback {
#Override
public void successCallback(String channel, Object message) {
synchronized (Iter.this.notifier) {
try {
processSuccessCallback(channel, message);
} catch (Exception e) {
Iter.this.caughtDuringRetrieval = e;
} finally {
Iter.this.retrieving = false;
Iter.this.notifier.notifyAll();
}
}
}
#Override
public void errorCallback(String channel, PubnubError error) {
Iter.this.caughtDuringRetrieval = new Exception("" + //
error.getClass().getName() + ": " + //
error.getErrorString() + //
" (code=" + error.errorCode + "; extendedCode=" + error.errorCodeExtended + ")");
Iter.this.caughtDuringRetrieval.fillInStackTrace();
}
}
private void processSuccessCallback(String channel, Object message) throws JSONException {
if (message == null)
throw new NullPointerException("retrieved message is null");
if (!(message instanceof JSONArray))
throw new RuntimeException("retrieved message is not a " + JSONArray.class.getName());
JSONArray historyMessage = (JSONArray) message;
// System.out.println(historyMessage.toString(2));
JSONArray messageList = extractMessageList(historyMessage);
long batchTimeRangeEnd = extractBatchTimeRangeEnd(historyMessage);
if (batchTimeRangeEnd > 0)
Iter.this.prevBatchTimeRangeEnd = batchTimeRangeEnd;
else
Iter.this.prevBatchTimeRangeEnd = end;
processMessageList(messageList);
}
private void processMessageList(JSONArray messageList) {
int i = 0;
for (; i < messageList.length(); i++) {
JSONObject message;
try {
message = messageList.getJSONObject(i);
} catch (JSONException e) {
String str;
try {
str = messageList.toString(2);
} catch (JSONException secondaryE) {
str = "(couldn't convert messageList to String because of " + secondaryE.toString() + ")";
}
throw new RuntimeException("couldn't extract message at index " + i + " from messageList (messageList:\n" + str
+ "\n(end of messageList)\n)", e);
}
Iter.this.buffer.add(message);
}
}
private long extractBatchTimeRangeEnd(JSONArray historyMessage) {
long batchTimeRangeEnd;
try {
batchTimeRangeEnd = historyMessage.getLong(2);
} catch (JSONException e) {
String str = safeConvertHistoryMessageToString(historyMessage);
throw new RuntimeException("could not extract element 2 (batchTimeRangeEnd) of retrieved historyMessage (historyMessage:\n" + str
+ "\n(end of historyMessage)\n)", e);
}
return batchTimeRangeEnd;
}
private String safeConvertHistoryMessageToString(JSONArray historyMessage) {
String str;
try {
str = historyMessage.toString(2);
} catch (JSONException secondaryE) {
str = "(couldn't convert historyMessage to String because of " + secondaryE.toString() + ")";
}
return str;
}
private JSONArray extractMessageList(JSONArray historyMessage) {
JSONArray messageArJson;
try {
messageArJson = historyMessage.getJSONArray(0);
} catch (JSONException e) {
String str = safeConvertHistoryMessageToString(historyMessage);
throw new RuntimeException("could not extract element 0 (messageList) of retrieved historyMessage (historyMessage:\n" + str
+ "\n(end of historyMessage)\n)", e);
}
return messageArJson;
}
public boolean hasNext() {
synchronized (Iter.this.buffer) {
ensureNotInExceptionState();
if (Iter.this.buffer.isEmpty())
retrieveNextBatch();
return !Iter.this.buffer.isEmpty();
}
}
public Object next() {
synchronized (Iter.this.buffer) {
if (!hasNext()) {
throw new NoSuchElementException("there are no more elements in this iterator");
}
Object result = Iter.this.buffer.removeFirst();
return result;
}
}
private void ensureNotInExceptionState() {
if (caughtDuringRetrieval != null) {
throw new RuntimeException("an exception was caught already by a previous attempt to access this iterator", caughtDuringRetrieval);
}
}
public void remove() {
throw new UnsupportedOperationException(getClass().getName() + " doesn't support remove()");
}
}
}

JList updating with live captured packets info (using jNetPcap) is causing list blanking

I am coding app which is capturing packet from 2 NI Cards at the same time in specific thread for them.
I am using jnetPcap and I am getting captured packet in jpackethanler's method nextPacket i need to show info from the current packet in JList but when I use simply defaultListModel and i write model1.addElement(packetinfo) then JList randomly goes in blank.
My code :
new Thread(){
#Override
public void run(){
StringBuilder errbuf = new StringBuilder(); // For any error msgs
int snaplen = 64 * 1024; // Capture all packets, no trucation
int flags = Pcap.MODE_PROMISCUOUS; // capture all packets
int timeout = 10 * 1000; // 10 seconds in millis
Pcap pcap1 =
Pcap.openLive(Variables.getDevice1().getName(), snaplen, flags, timeout, errbuf);
if (pcap1 == null) {
System.err.printf("Error while opening device for capture: "
+ errbuf.toString());
return;
}
PcapPacketHandler<String> jpacketHandler1 = new PcapPacketHandler<String>() {
int count = 1;
#Override
public void nextPacket(PcapPacket packet, String user) {
// ALL PACKETS FROM DEVICE 1 HERE
int packetSize = packet.size();
int packetCount = count++;
String desc = String.format("No.: %15d | HDRSize : %-4d", packetCount,packetSize);
device1Model.addElement(desc); // this adds desc to JLIST
}
};
pcap1.loop(Pcap.LOOP_INFINITE, jpacketHandler1, "");
pcap1.close();
}
}.start();
What do change to be more smooth and at the same time there will be no packet looses. Because i need to catch every packet for right function of app.
Thank You.

Player with custom data source on Blackberry

I must create a custom media player within the application with support for mp3 and wav files. I read in the documentation I can't seek or get the media file duration without a custom datasource.
I checked the demo in the JDE 4.6 but I have still problems... I can't get the duration, it returns much more then expected so I'm sure I screwed up something while I modified the code to read the mp3 file locally from the filesystem.
Can somebody tell me what I did wrong? (I can hear the mp3, so the player plays it correctly from start to end)
I must support OSs >= 4.6.
Here is my modified datasource:
/* LimitedRateStreaminSource.java
*
* Copyright © 1998-2009 Research In Motion Ltd.
*
* Note: For the sake of simplicity, this sample application may not leverage
* resource bundles and resource strings. However, it is STRONGLY recommended
* that application developers make use of the localization features available
* within the BlackBerry development platform to ensure a seamless application
* experience across a variety of languages and geographies.
* For more information on localizing your application, please refer to the
* BlackBerry Java Development Environment Development Guide associated with
* this release.
*/
package com.halcyon.tawkwidget.model;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.microedition.io.Connector;
import javax.microedition.io.file.FileConnection;
import javax.microedition.media.Control;
import javax.microedition.media.protocol.ContentDescriptor;
import javax.microedition.media.protocol.DataSource;
import javax.microedition.media.protocol.SourceStream;
import net.rim.device.api.io.SharedInputStream;
/**
* The data source used by the BufferedPlayback's media player.
*/
public final class LimitedRateStreamingSource extends DataSource
{
/** The max size to be read from the stream at one time. */
private static final int READ_CHUNK = 512; // bytes
/** A reference to the field which displays the load status. */
//private TextField _loadStatusField;
/** A reference to the field which displays the player status. */
//private TextField _playStatusField;
/**
* The minimum number of bytes that must be buffered before the media file
* will begin playing.
*/
private int _startBuffer = 200000;
/** The maximum size (in bytes) of a single read. */
private int _readLimit = 32000;
/**
* The minimum forward byte buffer which must be maintained in order for
* the video to keep playing. If the forward buffer falls below this
* number, the playback will pause until the buffer increases.
*/
private int _pauseBytes = 64000;
/**
* The minimum forward byte buffer required to resume
* playback after a pause.
*/
private int _resumeBytes = 128000;
/** The stream connection over which media content is passed. */
//private ContentConnection _contentConnection;
private FileConnection _fileConnection;
/** An input stream shared between several readers. */
private SharedInputStream _readAhead;
/** A stream to the buffered resource. */
private LimitedRateSourceStream _feedToPlayer;
/** The MIME type of the remote media file. */
private String _forcedContentType;
/** A counter for the total number of buffered bytes */
private volatile int _totalRead;
/** A flag used to tell the connection thread to stop */
private volatile boolean _stop;
/**
* A flag used to indicate that the initial buffering is complete. In
* other words, that the current buffer is larger than the defined start
* buffer size.
*/
private volatile boolean _bufferingComplete;
/** A flag used to indicate that the remote file download is complete. */
private volatile boolean _downloadComplete;
/** The thread which retrieves the remote media file. */
private ConnectionThread _loaderThread;
/** The local save file into which the remote file is written. */
private FileConnection _saveFile;
/** A stream for the local save file. */
private OutputStream _saveStream;
/**
* Constructor.
* #param locator The locator that describes the DataSource.
*/
public LimitedRateStreamingSource(String locator)
{
super(locator);
}
/**
* Open a connection to the locator.
* #throws IOException
*/
public void connect() throws IOException
{
//Open the connection to the remote file.
_fileConnection = (FileConnection)Connector.open(getLocator(),
Connector.READ);
//Cache a reference to the locator.
String locator = getLocator();
//Report status.
System.out.println("Loading: " + locator);
//System.out.println("Size: " + _contentConnection.getLength());
System.out.println("Size: " + _fileConnection.totalSize());
//The name of the remote file begins after the last forward slash.
int filenameStart = locator.lastIndexOf('/');
//The file name ends at the first instance of a semicolon.
int paramStart = locator.indexOf(';');
//If there is no semicolon, the file name ends at the end of the line.
if (paramStart < 0)
{
paramStart = locator.length();
}
//Extract the file name.
String filename = locator.substring(filenameStart, paramStart);
System.out.println("Filename: " + filename);
//Open a local save file with the same name as the remote file.
_saveFile = (FileConnection) Connector.open("file:///SDCard"+
"/blackberry/music" + filename, Connector.READ_WRITE);
//If the file doesn't already exist, create it.
if (!_saveFile.exists())
{
_saveFile.create();
}
System.out.println("---------- 1");
//Open the file for writing.
_saveFile.setReadable(true);
//Open a shared input stream to the local save file to
//allow many simultaneous readers.
SharedInputStream fileStream = SharedInputStream.getSharedInputStream(
_saveFile.openInputStream());
//Begin reading at the beginning of the file.
fileStream.setCurrentPosition(0);
System.out.println("---------- 2");
//If the local file is smaller than the remote file...
if (_saveFile.fileSize() < _fileConnection.totalSize())
{
System.out.println("---------- 3");
//Did not get the entire file, set the system to try again.
_saveFile.setWritable(true);
System.out.println("---------- 4");
//A non-null save stream is used as a flag later to indicate that
//the file download was incomplete.
_saveStream = _saveFile.openOutputStream();
System.out.println("---------- 5");
//Use a new shared input stream for buffered reading.
_readAhead = SharedInputStream.getSharedInputStream(
_fileConnection.openInputStream());
System.out.println("---------- 6");
}
else
{
//The download is complete.
System.out.println("---------- 7");
_downloadComplete = true;
//We can use the initial input stream to read the buffered media.
_readAhead = fileStream;
System.out.println("---------- 8");
//We can close the remote connection.
_fileConnection.close();
System.out.println("---------- 9");
}
if (_forcedContentType != null)
{
//Use the user-defined content type if it is set.
System.out.println("---------- 10");
_feedToPlayer = new LimitedRateSourceStream(_readAhead,
_forcedContentType);
System.out.println("---------- 11");
}
else
{
System.out.println("---------- 12");
//Otherwise, use the MIME types of the remote file.
// _feedToPlayer = new LimitedRateSourceStream(_readAhead,
_fileConnection));
}
System.out.println("---------- 13");
}
/**
* Destroy and close all existing connections.
*/
public void disconnect() {
try
{
if (_saveStream != null)
{
//Destroy the stream to the local save file.
_saveStream.close();
_saveStream = null;
}
//Close the local save file.
_saveFile.close();
if (_readAhead != null)
{
//Close the reader stream.
_readAhead.close();
_readAhead = null;
}
//Close the remote file connection.
_fileConnection.close();
//Close the stream to the player.
_feedToPlayer.close();
}
catch (Exception e)
{
System.err.println(e.getMessage());
}
}
/**
* Returns the content type of the remote file.
* #return The content type of the remote file.
*/
public String getContentType()
{
return _feedToPlayer.getContentDescriptor().getContentType();
}
/**
* Returns a stream to the buffered resource.
* #return A stream to the buffered resource.
*/
public SourceStream[] getStreams()
{
return new SourceStream[] { _feedToPlayer };
}
/**
* Starts the connection thread used to download the remote file.
*/
public void start() throws IOException
{
//If the save stream is null, we have already completely downloaded
//the file.
if (_saveStream != null)
{
//Open the connection thread to finish downloading the file.
_loaderThread = new ConnectionThread();
_loaderThread.start();
}
}
/**
* Stop the connection thread.
*/
public void stop() throws IOException
{
//Set the boolean flag to stop the thread.
_stop = true;
}
/**
* #see javax.microedition.media.Controllable#getControl(String)
*/
public Control getControl(String controlType)
{
// No implemented Controls.
return null;
}
/**
* #see javax.microedition.media.Controllable#getControls()
*/
public Control[] getControls()
{
// No implemented Controls.
return null;
}
/**
* Force the lower level stream to a given content type. Must be called
* before the connect function in order to work.
* #param contentType The content type to use.
*/
public void setContentType(String contentType)
{
_forcedContentType = contentType;
}
/**
* A stream to the buffered media resource.
*/
private final class LimitedRateSourceStream implements SourceStream
{
/** A stream to the local copy of the remote resource. */
private SharedInputStream _baseSharedStream;
/** Describes the content type of the media file. */
private ContentDescriptor _contentDescriptor;
/**
* Constructor. Creates a LimitedRateSourceStream from
* the given InputStream.
* #param inputStream The input stream used to create a new reader.
* #param contentType The content type of the remote file.
*/
LimitedRateSourceStream(InputStream inputStream, String contentType)
{
System.out.println("[LimitedRateSoruceStream]---------- 1");
_baseSharedStream = SharedInputStream.getSharedInputStream(
inputStream);
System.out.println("[LimitedRateSoruceStream]---------- 2");
_contentDescriptor = new ContentDescriptor(contentType);
System.out.println("[LimitedRateSoruceStream]---------- 3");
}
/**
* Returns the content descriptor for this stream.
* #return The content descriptor for this stream.
*/
public ContentDescriptor getContentDescriptor()
{
return _contentDescriptor;
}
/**
* Returns the length provided by the connection.
* #return long The length provided by the connection.
*/
public long getContentLength()
{
return _fileConnection.totalSize();
}
/**
* Returns the seek type of the stream.
*/
public int getSeekType()
{
return RANDOM_ACCESSIBLE;
//return SEEKABLE_TO_START;
}
/**
* Returns the maximum size (in bytes) of a single read.
*/
public int getTransferSize()
{
return _readLimit;
}
/**
* Writes bytes from the buffer into a byte array for playback.
* #param bytes The buffer into which the data is read.
* #param off The start offset in array b at which the data is written.
* #param len The maximum number of bytes to read.
* #return the total number of bytes read into the buffer, or -1 if
* there is no more data because the end of the stream has been reached.
* #throws IOException
*/
public int read(byte[] bytes, int off, int len) throws IOException
{
System.out.println("[LimitedRateSoruceStream]---------- 5");
System.out.println("Read Request for: " + len + " bytes");
//Limit bytes read to our readLimit.
int readLength = len;
System.out.println("[LimitedRateSoruceStream]---------- 6");
if (readLength > getReadLimit())
{
readLength = getReadLimit();
}
//The number of available byes in the buffer.
int available;
//A boolean flag indicating that the thread should pause
//until the buffer has increased sufficiently.
boolean paused = false;
System.out.println("[LimitedRateSoruceStream]---------- 7");
for (;;)
{
available = _baseSharedStream.available();
System.out.println("[LimitedRateSoruceStream]---------- 8");
if (_downloadComplete)
{
//Ignore all restrictions if downloading is complete.
System.out.println("Complete, Reading: " + len +
" - Available: " + available);
return _baseSharedStream.read(bytes, off, len);
}
else if(_bufferingComplete)
{
if (paused && available > getResumeBytes())
{
//If the video is paused due to buffering, but the
//number of available byes is sufficiently high,
//resume playback of the media.
System.out.println("Resuming - Available: " +
available);
paused = false;
return _baseSharedStream.read(bytes, off, readLength);
}
else if(!paused && (available > getPauseBytes() ||
available > readLength))
{
//We have enough information for this media playback.
if (available < getPauseBytes())
{
//If the buffer is now insufficient, set the
//pause flag.
paused = true;
}
System.out.println("Reading: " + readLength +
" - Available: " + available);
return _baseSharedStream.read(bytes, off, readLength);
}
else if(!paused)
{
//Set pause until loaded enough to resume.
paused = true;
}
}
else
{
//We are not ready to start yet, try sleeping to allow the
//buffer to increase.
try
{
Thread.sleep(500);
}
catch (Exception e)
{
System.err.println(e.getMessage());
}
}
}
}
/**
* #see javax.microedition.media.protocol.SourceStream#seek(long)
*/
public long seek(long where) throws IOException
{
_baseSharedStream.setCurrentPosition((int) where);
return _baseSharedStream.getCurrentPosition();
}
/**
* #see javax.microedition.media.protocol.SourceStream#tell()
*/
public long tell()
{
return _baseSharedStream.getCurrentPosition();
}
/**
* Close the stream.
* #throws IOException
*/
void close() throws IOException
{
_baseSharedStream.close();
}
/**
* #see javax.microedition.media.Controllable#getControl(String)
*/
public Control getControl(String controlType)
{
// No implemented controls.
return null;
}
/**
* #see javax.microedition.media.Controllable#getControls()
*/
public Control[] getControls()
{
// No implemented controls.
return null;
}
}
/**
* A thread which downloads the remote file and writes it to the local file.
*/
private final class ConnectionThread extends Thread
{
/**
* Download the remote media file, then write it to the local
* file.
* #see java.lang.Thread#run()
*/
public void run()
{
try
{
byte[] data = new byte[READ_CHUNK];
int len = 0;
//Until we reach the end of the file.
while (-1 != (len = _readAhead.read(data)))
{
_totalRead += len;
if (!_bufferingComplete && _totalRead > getStartBuffer())
{
//We have enough of a buffer to begin playback.
_bufferingComplete = true;
System.out.println("Initial Buffering Complete");
}
if (_stop)
{
//Stop reading.
return;
}
}
System.out.println("Downloading Complete");
System.out.println("Total Read: " + _totalRead);
//If the downloaded data is not the same size
//as the remote file, something is wrong.
if (_totalRead != _fileConnection.totalSize())
{
System.err.println("* Unable to Download entire file *");
}
_downloadComplete = true;
_readAhead.setCurrentPosition(0);
//Write downloaded data to the local file.
while (-1 != (len = _readAhead.read(data)))
{
_saveStream.write(data);
}
}
catch (Exception e)
{
System.err.println(e.toString());
}
}
}
/**
* Gets the minimum forward byte buffer which must be maintained in
* order for the video to keep playing.
* #return The pause byte buffer.
*/
int getPauseBytes()
{
return _pauseBytes;
}
/**
* Sets the minimum forward buffer which must be maintained in order
* for the video to keep playing.
* #param pauseBytes The new pause byte buffer.
*/
void setPauseBytes(int pauseBytes)
{
_pauseBytes = pauseBytes;
}
/**
* Gets the maximum size (in bytes) of a single read.
* #return The maximum size (in bytes) of a single read.
*/
int getReadLimit()
{
return _readLimit;
}
/**
* Sets the maximum size (in bytes) of a single read.
* #param readLimit The new maximum size (in bytes) of a single read.
*/
void setReadLimit(int readLimit)
{
_readLimit = readLimit;
}
/**
* Gets the minimum forward byte buffer required to resume
* playback after a pause.
* #return The resume byte buffer.
*/
int getResumeBytes()
{
return _resumeBytes;
}
/**
* Sets the minimum forward byte buffer required to resume
* playback after a pause.
* #param resumeBytes The new resume byte buffer.
*/
void setResumeBytes(int resumeBytes)
{
_resumeBytes = resumeBytes;
}
/**
* Gets the minimum number of bytes that must be buffered before the
* media file will begin playing.
* #return The start byte buffer.
*/
int getStartBuffer()
{
return _startBuffer;
}
/**
* Sets the minimum number of bytes that must be buffered before the
* media file will begin playing.
* #param startBuffer The new start byte buffer.
*/
void setStartBuffer(int startBuffer)
{
_startBuffer = startBuffer;
}
}
And in this way i use it:
LimitedRateStreamingSource source = new
LimitedRateStreamingSource("file:///SDCard/music3.mp3");
source.setContentType("audio/mpeg");
mediaPlayer = javax.microedition.media.Manager.createPlayer(source);
mediaPlayer.addPlayerListener(this);
mediaPlayer.realize();
mediaPlayer.prefetch();
After start I to use mediaPlayer.getDuration it returns lets say around 24:22 (the inbuild media player in the blackberry say the file length is 4:05)
I tried to get the duration in the listener and there it unfortunately returned around 64 minutes, so I'm sure something is not good inside the datasoruce....
code that converts
String getElapsedTimeMinutesSeconds(long elapsedTime) {
long Seconds=(elapsedTime/1000)%60;
long Minutes=(elapsedTime/(1000*60))%60;
long Hours=(elapsedTime/(1000*60*60))%24;
return ""+Minutes + ":"+Seconds;
}
Player.setMediaTime() and Player.getMediaTime() both refer to time in microseconds, not milliseconds. So to get the number of elapsed seconds, you need to divide by 1000000 instead of just 1000.
Sometime earlier I faced with the problem to play unlimited size audio. I fix it.
Here's the link:
http://supportforums.blackberry.com/t5/Java-Development/Use-custom-DataSource-to-play-audio/m-p/1373247#M178928

Resources