HoloLens 2 laggy TCP client [duplicate] - hololens

I'm trying to create a TCP client/server program between a python server (first PC) and an UWP client app (second PC).
I want to simulate a fast python server that sends a message every 10ms and a slower UWP app with an Update function that has to display the last message received every 40ms.
To do this, on the UWP side, I create a Task that reads the message sent by python, saves it in a variable (this.message), and prints it. The Update function reads this variable and prints it every 40ms (it waits 40ms after each print).
The client/server exchange works but I obtain a strange behavior. The Update function sometimes "blocks" on the same value, see the output:
...
[Task] Message = 1105, Ts = 1626767660488
[Task] Message = 1106, Ts = 1626767660495
[Task] Message = 1107, Ts = 1626767660502
[Task] Message = 1108, Ts = 1626767660508
[Task] Message = 1109, Ts = 1626767660512
[Task] Message = 1110, Ts = 1626767660516
[Task] Message = 1111, Ts = 1626767660519
[Task] Message = 1112, Ts = 1626767660523
[Task] Message = 1113, Ts = 1626767660527
[Task] Message = 1114, Ts = 1626767660530
[Task] Message = 1115, Ts = 1626767660534
[Task] Message = 1116, Ts = 1626767660537
[Task] Message = 1117, Ts = 1626767660541
[Update] Message = 1107
[Task] Message = 1118, Ts = 1626767660546
[Task] Message = 1119, Ts = 1626767660551
[Task] Message = 1120, Ts = 1626767660554
[Task] Message = 1121, Ts = 1626767660558
[Task] Message = 1122, Ts = 1626767660562
[Update] Message = 1122
[Update] Message = 1122
[Task] Message = 1123, Ts = 1626767660693
[Task] Message = 1124, Ts = 1626767660697
[Task] Message = 1125, Ts = 1626767660701
[Task] Message = 1126, Ts = 1626767660705
[Task] Message = 1127, Ts = 1626767660708
[Task] Message = 1128, Ts = 1626767660712
[Task] Message = 1129, Ts = 1626767660716
[Task] Message = 1130, Ts = 1626767660720
[Task] Message = 1131, Ts = 1626767660724
[Task] Message = 1132, Ts = 1626767660727
[Task] Message = 1133, Ts = 1626767660731
[Task] Message = 1134, Ts = 1626767660735
[Task] Message = 1135, Ts = 1626767660739
[Task] Message = 1136, Ts = 1626767660742
[Task] Message = 1137, Ts = 1626767660746
[Update] Message = 1124
[Task] Message = 1138, Ts = 1626767660757
[Task] Message = 1139, Ts = 1626767660760
[Task] Message = 1140, Ts = 1626767660764
[Task] Message = 1141, Ts = 1626767660768
[Task] Message = 1142, Ts = 1626767660772
[Task] Message = 1143, Ts = 1626767660775
[Update] Message = 1143
[Task] Message = 1144, Ts = 1626767660834
[Task] Message = 1145, Ts = 1626767660841
[Task] Message = 1146, Ts = 1626767660848
[Task] Message = 1147, Ts = 1626767660856
[Task] Message = 1148, Ts = 1626767660862
[Task] Message = 1149, Ts = 1626767660869
[Task] Message = 1150, Ts = 1626767660875
...
Python side:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('', 4444))
print("[SERVER TCP] Waiting connection ...")
sock.listen(1)
client_socket, address = sock.accept()
data = client_socket.recv(1024)
print(data)
print("[SERVER TCP] Connection done")
x = 0
while True:
x = x + 1
msg = f"{x}\n"
client_socket.send(msg.encode())
sys.stdout.write(msg)
time.sleep(0.01)
UWP app side (MainPage.xaml.cs):
using System;
using System.IO;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Navigation;
using System.Threading.Tasks;
using System.Diagnostics;
namespace ClientUWP
{
/// <summary>
/// An empty page that can be used on its own or navigated to within a Frame.
/// </summary>
public sealed partial class MainPage : Page
{
volatile String message;
Task T;
public MainPage()
{
this.InitializeComponent();
}
protected override void OnNavigatedTo(NavigationEventArgs e)
{
this.StartClient();
this.Update();
}
private async void StartClient()
{
try
{
var streamSocket = new Windows.Networking.Sockets.StreamSocket();
var hostName = new Windows.Networking.HostName("192.168.1.50");
string PortNumber = "4444";
string request = "Hello, World!";
await streamSocket.ConnectAsync(hostName, PortNumber);
using (Stream outputStream = streamSocket.OutputStream.AsStreamForWrite())
{
using (var streamWriter = new StreamWriter(outputStream))
{
await streamWriter.WriteLineAsync(request);
await streamWriter.FlushAsync();
}
}
T = Task.Run(() =>
{
using (Stream inputStream = streamSocket.InputStream.AsStreamForRead())
{
using (StreamReader streamReader = new StreamReader(inputStream))
{
while (true)
{
this.message = streamReader.ReadLine();
Debug.WriteLine("[Task] Message = : " + this.message);
}
}
}
});
}
catch (Exception ex)
{
Windows.Networking.Sockets.SocketErrorStatus webErrorStatus = Windows.Networking.Sockets.SocketError.GetStatus(ex.GetBaseException().HResult);
Debug.WriteLine(webErrorStatus.ToString());
}
}
private async void Update()
{
while (true)
{
Debug.WriteLine("[Update] Message = " + this.message);
await Task.Delay(40);
}
T.Wait();
}
}
}
Any idea how to solve this issue? Thanks!
Update Post:
When I print the timestamps in Task, it seems that Task freezes until the end of Update function.

Your data is essentially a random output. There are many reasons could affect the output, such as the network delay, CPU usage. For example, if your network lag is 160ms, during these 160ms, your client can’t receive messages, but your update method is continued, which causes four update messages will appear continuously.
Therefore, I suggest you don’t separate these two tasks, you could do these in a Task. You could calculate the message numbers instead of using Task.Delay(40) to determine when to print the update message, in other words, you could print an update message every 40 messages

As per Arya's answer, it's almost impossible to guarantee an exact interval between operations because of external factors. As I mentioned in the comments, you can however trigger the reading of the message at more or less regular intervals and avoid reading twice as follows (this is a Console NET Framework app; no need to completely change what you did, I just adapted for non UWP app):
public class Program
{
public static void Main(string[] args)
{
StartTimer();
StartClient();
Console.ReadKey();
}
private static string message;
private static readonly Stopwatch sw = new Stopwatch();
private static readonly object lockobject = new object();
private static Timer timer;
private static void StartClient()
{
const string ip = "127.0.0.1";
const int port = 4444;
var tcpClient = new TcpClient();
try
{
var ipep = new IPEndPoint(IPAddress.Parse(ip), port);
tcpClient.Connect(ipep);
Task.Run(() => {
using (var networkStream = tcpClient.GetStream())
using (var writer = new StreamWriter(networkStream))
using (var reader = new StreamReader(networkStream, Encoding.UTF8))
{
writer.WriteLine("Hello, World!");
writer.Flush();
sw.Start();
while (true)
{
try
{
// Add this lock for synchronization on message between the threads.
lock (lockobject)
{
message = reader.ReadLine();
Console.WriteLine($"[Task] Message: {message} at {sw.ElapsedMilliseconds}ms");
}
}
catch (Exception ex)
{
// Break the loop and stop timer in case of exception.
timer.Change(Timeout.Infinite, Timeout.Infinite);
Console.WriteLine(ex);
break;
}
}
}
});
}
catch (Exception ex)
{
Console.WriteLine(ex);
timer.Change(Timeout.Infinite, Timeout.Infinite);
}
}
private static void StartTimer()
{
// Replace the loop by a timer and lock on the read message operation.
timer = new Timer(_ =>
{
lock (lockobject)
{
Console.WriteLine($"[Timer] Message: {message} at { sw.ElapsedMilliseconds}ms");
}
},
null, 0, 40);
}
}
The output with this is as follows:
[Timer] Message: at 0ms
[Task] Message: 1 at 0ms
[Task] Message: 2 at 18ms
[Task] Message: 3 at 33ms
[Task] Message: 4 at 49ms
[Task] Message: 5 at 65ms
[Timer] Message: 5 at 66ms <- timer message
[Task] Message: 6 at 81ms
[Task] Message: 7 at 97ms
[Task] Message: 8 at 112ms
[Timer] Message: 8 at 113ms <- timer message
[Task] Message: 9 at 128ms
[Task] Message: 10 at 144ms
[Task] Message: 11 at 160ms
[Timer] Message: 11 at 161ms <- timer message
[Task] Message: 12 at 176ms
[Task] Message: 13 at 192ms
[Task] Message: 14 at 207ms
[Timer] Message: 14 at 208ms
[Task] Message: 15 at 223ms
[Task] Message: 16 at 239ms
[Task] Message: 17 at 255ms
[Timer] Message: 17 at 256ms
...
You get Timer (former Update) reads at regular intervals and no duplicates. If you need even more accurate intervals, it starts to become much more tricky (see this answer). Not sure this is required for your scenario though.

Related

Sending byte[] array to Host using Chrome Native Messaging

I am trying to send a JSON object that contains a byte[] to a "Console Application" using a Chrome Extention by NativeMessaging.
I have no problems with some of the calls, but I get a problem when I try to send a byte array.
Behavior is following:
ChromeExtention creates a JSON object that contains a list and a byte[].
NativeApp receives it but remains somehow blocked.
reader.Read(buffer, 0, buffer.Length);
I will post here a version of code that I hope it helps.
Native App.
Program.cs
public static void Main(string[] args)
{
m_Logger = LogManager.Setup().GetCurrentClassLogger();
m_Logger.Info("App Started!");
try
{
JObject command = Read();
if (command != null)
{
var ceCommand = command.ToObject<CommandModel>();
if (ceCommand == null)
{
m_Logger.Warn("Could not deserialize command from extention");
return;
}
//do other things
}
}
}
catch (Exception ex)
{
throw;
}
}
public static JObject Read()
{
try
{
m_Logger.Debug("Read started");
var stdin = Console.OpenStandardInput();
m_Logger.Debug("StDin initialized");
var length = 0;
var lengthBytes = new byte[4];
stdin.Read(lengthBytes, 0, 4);
length = BitConverter.ToInt32(lengthBytes, 0);
m_Logger.Debug($"Message length: {length}");
var buffer = new char[length];
m_Logger.Debug($"Buffer lenght: {buffer.Length}");
using (var reader = new StreamReader(stdin))
{
m_Logger.Debug("Reading from stream...");
while (reader.Peek() >= 0)
{
m_Logger.Debug("We are in reading process...please wait");
int result = reader.Read(buffer, 0, buffer.Length);
m_Logger.Debug($"Read {result} numer of chars");
}
m_Logger.Debug("Read finished");
}
var stringMessage = new string(buffer);
m_Logger.Info($"Recieved from CE {stringMessage}");
return JsonConvert.DeserializeObject<JObject>(stringMessage);
}
catch (Exception ex)
{
m_Logger.Error(ex, $"Error at reading input data");
return null;
}
}
public static void Write(JToken data)
{
var json = new JObject();
json["data"] = data;
var bytes = System.Text.Encoding.UTF8.GetBytes(json.ToString(Formatting.None));
var stdout = Console.OpenStandardOutput();
stdout.WriteByte((byte)((bytes.Length >> 0) & 0xFF));
stdout.WriteByte((byte)((bytes.Length >> 8) & 0xFF));
stdout.WriteByte((byte)((bytes.Length >> 16) & 0xFF));
stdout.WriteByte((byte)((bytes.Length >> 24) & 0xFF));
stdout.Write(bytes, 0, bytes.Length);
stdout.Flush();
}
/// <summary>
/// Copies the contents of input to output. Doesn't close either stream.
/// </summary>
public static string ByteToString(byte[] array, int count)
{
char[] chars = Encoding.UTF8.GetChars(array, 0, count);
return new string(chars);
}
I deducted that the problem is at reader.Read(buffer, 0, buffer.Length); based on logs that look like this
2022-11-17 16:34:14.9604 INFO App Started!
2022-11-17 16:34:15.0837 DEBUG Read started
2022-11-17 16:34:15.0837 DEBUG StDin initialized
2022-11-17 16:34:15.0837 DEBUG Message length: 1862
2022-11-17 16:34:15.0837 DEBUG Buffer lenght: 1862
2022-11-17 16:34:15.0837 DEBUG Reading from stream...
2022-11-17 16:34:15.0837 DEBUG We are in reading process...please wait
2022-11-17 16:34:15.0837 DEBUG Read 1862 numer of chars
2022-11-17 16:34:44.2786 DEBUG Read finished
2022-11-17 16:34:44.2787 DEBUG Message length: 0
2022-11-17 16:34:44.2787 DEBUG Buffer lenght: 0
2022-11-17 16:34:44.2786 INFO Recieved from CE {"CommandId":2,"CommandValue":"{\"ImportCertificates\":[],\"byteArray\":\"MIIKWAIBAzCCChQGCSqGSIb3DQEHAaCCCgUEggoBMIIJ/TCCBe4GCSqGSIb3DQEHAaCCBd8EggXbMIIF1zCCBdMGCyqGSIb3DQEMCgECoIIE7jCCBOowHAYKKoZIhvcNAQwBAzAOBAjslY0EWPiyRwICB9AEggTItTgbudKRaEBCdOn1rMgGUoGjVIOxTgT/5Oo5D0gfBe/0yorvQKenVimcfDmWCXZGlsvou5Km7g3yjmK7PTZh3IUuX86bJeyECrJBZa+4ZXyrbkVV7R2GEwy99ACOkevHxBZ7H6deVKdRFDsCSC+cdLpsSGoNNi3Coowqw8i7lzXDPzph64L2Rre7cky/QJdkXKIEUGjxKYUR4cpOOmlLXfbQMR3fOChT5FrxbOnTRTAvVELtwFTh8Gxq55Et8rLgIktQ2eL8FIx43sGspukhZvm5bH05aEwVz5df0T3e3cMsZ13t4oc72phNKx9HiZmPvmi4lUdBwn6qXEJSRg+P5MWGD41StCAoZ/pDtMTPSV5DalLpDpKF6XfxByOSXfn27KYWFZoiEdxCmabS7eomqeD03uWEzWbgPW1hha69Bg1DcmSYZgDfISugtow6p+ozQSjaNnELlbz1SsiBMPbkoNj944IxSKkHgfiqwOyQm3JqHGjDH6Hp9OB3RyuQHoeWLrW9ulmWvnT/Dn+LC0T26YHeVeTO30U86r9ElBiv4iXa5JdflWqFecbjQC1Xs29yBkkFRNrfgJ4Txd/xOz0rhzFxZmcIwiJsQgh+WxSWLG3em02xmtyO7Rcxc6g+onVpEFXKiaYDqTWNDKE2wDSDPZauheyiiluj42gi5A086rSjuqqyidzWQb6FKYs73ZUM8drnYK36RT7zG8TFpV3RdrXbnui8eFRZ4bFHNlptHTwvqpZY/sDDP1R1q1EgrckMbZhOpHhKW/1GRd95eHlSGJPdTROtR3uEARZD0+1zBarw68p1k650DWua88cMrOQPKH1X8Ce++KIdFG2RtfI2P3lhPsD0gdpet0gDEC1peUXfrZVQYD3G8IfEjpYgDpgPa6zjblScidem628MPz4dnTnWrp1S7P2ILlGYinb9uCzSR17IvPoVITQvXk9KpTp1uzuWTnRQHMfUyNvgjHSAOwt3bmx5kEyC/04ujhUvQkiyPMCrDtiO/fLYkrWuqOLNr74l/VAJYU06YR7yNMfG1kEpRYkf96v0s3XK1tZEQBt26lZFPIMl/NOHYpbCfxCGZkpOGIFDojjo72p4VUuyGD/MZ41B99jcNVB2/Sqmc9XwALl/jjkJz6+aCmvLUePeLmhg2Bxrjkmf8iDfDnKByXM5lam3D\"}","SessionKey":"QXCUUKNB0T"}
2022-11-17 16:34:44.2787 DEBUG Reading from stream...
2022-11-17 16:34:44.2787 DEBUG Read finished
2022-11-17 16:34:44.2787 INFO Recieved from CE
However this is not a problem if
byteArray property is smaller in size
2022-11-17 16:32:49.7558 INFO App Started!
2022-11-17 16:32:49.8809 DEBUG Read started
2022-11-17 16:32:49.8809 DEBUG StDin initialized
2022-11-17 16:32:49.8809 DEBUG Message length: 546
2022-11-17 16:32:49.8809 DEBUG Buffer lenght: 546
2022-11-17 16:32:49.8809 DEBUG Reading from stream...
2022-11-17 16:32:49.8809 DEBUG We are in reading process...please wait
2022-11-17 16:32:49.8809 DEBUG Read 546 numer of chars
2022-11-17 16:32:49.8809 DEBUG Read finished
2022-11-17 16:32:49.8809 INFO Recieved from CE {"CommandId":2,"CommandValue":"{\"ImportCertificates\":[],\"byteArray\":\"test/wqewq+./qweqeq//eqwq\"}","SessionKey":"0SZFlRVrQR"}
As you can see here from logs app does the full flow without a problem from Starting and reading and parsing my message.
I suspected that is something regarding the size of the array, but from what I've read you can send up to 1mb of data in one NativeMessage call, data that I want to send is about 4Kb.
Not even close to the max allowed size.
Any ideas on what I am doing wrong?
I will post here also the Javascript code that send this, but I don't believe problem relies in Javascript code.
export const syncCertificates = async () => {
var requestToHost = await api.getInfo();
var responsefromHost = await nativeService.syncCertificates(requestToHost);
return certUploadResponse;
}
export const syncCertificates = async (certs) => {
var command = commands[1];
await sendNativeMessageV2(command.Id, certs);
}
async function sendNativeMessageV2(command, commandValue) {
commandInProgress = command;
var sessionKey = await storage.getSessionKey();
var message =
{
"CommandId": command,
"CommandValue": JSON.stringify(commandValue),
"SessionKey": sessionKey
}
return new Promise((resolve, reject) => {
chrome.runtime.sendNativeMessage('com.allianz.usercertificateautoenrollment', message,
async function (result) {
var response = await onNativeMessage(result);
resolve(response);
}
);
});
}
After more investigations I found this Why is my C# native messaging host hanging when reading a message sent by a Firefox browser extension?
Changing the loop as answered here did the trick.
Change the loop as follows:
using (var reader = new StreamReader(stdin))
{
var offset = 0;
while (offset < length && reader.Peek() >= 0)
{
offset += reader.Read(buffer, offset, length - offset);
}
}

SSE events are received when the connection is closed in JHIpster and Gateway

I have a simple code in Jhipster microservice which sends words periodically:
#GetMapping(path = "/words", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
SseEmitter getWords() {
List<String> dataSets = Arrays.asList("one ", "two", "three", "four", "five");
SseEmitter emitter = new SseEmitter();
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.execute(() -> {
try {
for (String dataSet : dataSets) {
randomDelay();
emitter.send(SseEmitter.event().name("update").id("1").data(dataSet));
}
emitter.complete();
} catch (IOException e) {
emitter.completeWithError(e);
}
});
executor.shutdown();
return emitter;
}
In the frontent I use:
sse = new EventSource(
`${serverURL}/services/news/api/words&access_token=` + (localStorage.getItem('jhi-authenticationToken') || sessionStorage.getItem('jhi-authenticationToken')), {});
sse.onmessage = (event) => {
console.log('event', event)
const data = event.data;
}
Instead of having message by message every 1 second I receive all the messages at once after 5 seconds.
How to make it works periodically and receive message by message?
BTW I use reactive Gateway and non-reactive microservice for the SSE.

MaxAutoLockRenewalDuration is not working for Azure Service Bus

I have created below sample application to test "MaxAutoLockRenewalDuration" functionality.
Here is my test scenario,
Created two console application with the same code.
Add one message in the queue
Run Console App1 which receives the message
Run Console App2 which receives the same message.
With below code I have setup MaxAutoLockRenewalDuration to 10 minutes.
To my understanding of "MaxAutoLockRenewalDuration" it should renew the lock automatically till 10 minutes and second console app should not receive the same message.
public class Program
{
static string connectionString = "***";
static string queueName = "firstqueue";
static async Task Main(string[] args)
{
try
{
//uncomment below if you would like to add message to queue
//await CreateMessage(queueName, "Message 1 to test 'MaxAutoLockRenewalDuration'");
await ReceiveMessagesAsync();
}
catch (Exception ex)
{
throw;
}
Console.ReadKey();
}
private static async Task CreateMessage(string queueName, string textMessage)
{
// create a Service Bus client
await using (ServiceBusClient client = new ServiceBusClient(connectionString))
{
// create a sender for the queue
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send
ServiceBusMessage message = new ServiceBusMessage(textMessage);
// send the message
await sender.SendMessageAsync(message);
Console.WriteLine($"Sent a single message to the queue: {queueName}");
}
}
// handle received messages
static async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine($"Received: {body}");
System.Threading.Thread.Sleep(TimeSpan.FromMinutes(5));
// complete the message. messages is deleted from the queue.
await args.CompleteMessageAsync(args.Message);
}
// handle any errors when receiving messages
static Task ErrorHandler(ProcessErrorEventArgs args)
{
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
static async Task ReceiveMessagesAsync()
{
var processorOptions = new ServiceBusProcessorOptions
{
AutoCompleteMessages = false,
MaxConcurrentCalls = 1,
MaxAutoLockRenewalDuration = TimeSpan.FromMinutes(10),
ReceiveMode = ServiceBusReceiveMode.PeekLock,
PrefetchCount = 1
};
await using (ServiceBusClient client = new ServiceBusClient(connectionString))
{
// create a processor that we can use to process the messages
ServiceBusProcessor processor = client.CreateProcessor(queueName, processorOptions);
// add handler to process messages
processor.ProcessMessageAsync += MessageHandler;
// add handler to process any errors
processor.ProcessErrorAsync += ErrorHandler;
// start processing
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
// stop processing
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
}
static async Task ReceiveMessagesAsync(string queueName)
{
await using (ServiceBusClient client = new ServiceBusClient(connectionString))
{
// create a processor that we can use to process the messages
ServiceBusProcessor processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
// add handler to process messages
processor.ProcessMessageAsync += MessageHandler;
// add handler to process any errors
processor.ProcessErrorAsync += ErrorHandler;
// start processing
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
// stop processing
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
}
}
I'm using your code with Azure.Messaging.ServiceBus Version "7.1.1" and "7.1.2", the "MaxAutoLockRenewalDuration" functionality works well.
Here are my steps:
1.Send a message to the queue.
2.Run Console App1 which receives the message
3.Keep Console App1 running, and then run Console App2, there is no message received there. Here is the screenshot:
But if you first run Console App1 which receives the message, then close the Console App1 -> then run Console App2, you can see the message in Console App2. This is expected.

SparkStreaming either running 2 times the same command or mail sending 2 times same mail

My spark application sends two mails when I send just 1 string to my Kafka Topic. Here is the interested part of code:
JavaDStream<String> lines = kafkaStream.map ( [returns the 2nd value of the tuple];
lines.foreachRDD(new VoidFunction<JavaRDD<String>>() {
[... some stuff ...]
JavaRDD<String[]> flagAddedRDD = associatedToPersonRDD.map(new Function<String[],String[]>(){
#Override
public String[] call(String[] arg0) throws Exception {
String[] s = new String[arg0.length+1];
System.arraycopy(arg0, 0, s, 0, arg0.length);
int a = FilePrinter.getAge(arg0[CSVExampleDevice.LENGTH+People.BIRTH_DATE]);
int p = Integer.parseInt(arg0[CSVExampleDevice.PULSE]);
if(
((p<=45 || p>=185)&&(a<=12 || a>=70))
||
(p>=190 || p<=40)){
s[arg0.length]="1";
Mailer.sendMail(mailTo, arg0);
}
else
s[arg0.length]="0";
return s;
}
});`
I cannot understand if the mail sends two emails, because the transformation after this one is a save on file and this only return 1 line. The mailer.sendMail:
public static void sendMail(String whoTo, String[] whoIsDying){
Properties props = new Properties();
props.put("mail.smtp.host", "mail.***.com"); //edited
props.put("mail.smtp.port", "25");
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Session session = Session.getInstance(props);
try {
Message message = new MimeMessage(session);
message.setFrom(new InternetAddress("***my-email***")); //edited
for (String string : whoTo.split(","))
message.addRecipient(Message.RecipientType.TO,
new InternetAddress(string));
message.setSubject(whoIsDying[PersonClass.TIMESTAMP]);
message.setText("trial");
System.out.println("INFO: sent mail");
Transport.send(message);
} catch (MessagingException e) {
throw new RuntimeException(e);
}
}
The reason this happens is because I called two actions at the end of the transformation:
FilePrinter.saveAssociatedAsCSV(associatedSavePath, unifiedAssociatedStringRDD.collect()); //first action
JavaRDD<String[]> enrichedWithWeatherRDD = flagAddedRDD.map(new Function<String[],String[]>(){ [some more stuff] });
JavaRDD<String> unifiedEnrichedStringRDD = enrichedWithWeatherRDD.map(unifyArrayIntoString);
FilePrinter.saveEnrichedAsCSV(enrichedSavePath, unifiedEnrichedStringRDD.collect()); //second action
and thus the whole transformation is called again and the mailer part is above both of these actions.

How to read InputStream only once using CustomReceiver

I have written custom receiver to receive the stream that is being generated by one of our application. The receiver starts the process gets the stream and then cals store. However, the receive method gets called multiple times, I have written proper loop break condition, but, could not do it. How to ensure it only reads once and does not read the already processed data.?
Here is my custom receiver code:
class MyReceiver() extends Receiver[String](StorageLevel.MEMORY_AND_DISK_2) with Logging {
def onStart() {
new Thread("Splunk Receiver") {
override def run() { receive() }
}.start()
}
def onStop() {
}
private def receive() {
try {
/* My Code to run a process and get the stream */
val reader = new ResultsReader(job.getResults()); // ResultReader is reader for the appication
var event:String = reader.getNextLine;
while (!isStopped || event != null) {
store(event);
event = reader.getNextLine;
}
reader.close()
} catch {
case t: Throwable =>
restart("Error receiving data", t)
}
}
}
Where did i go wrong.?
Problems
1) The job and stream reading happening after every 2 seconds and same data is piling up. So, for 60 line of data, i am getting 1800 or greater some times, in total.
Streaming Code:
val conf = new SparkConf
conf.setAppName("str1");
conf.setMaster("local[2]")
conf.set("spark.driver.allowMultipleContexts", "true");
val ssc = new StreamingContext(conf, Minutes(2));
val customReceiverStream = ssc.receiverStream(new MyReceiver)
println(" searching ");
//if(customReceiverStream.count() > 0 ){
customReceiverStream.foreachRDD(x => {println("=====>"+ x.count());x.count()});
//}
ssc.start();
ssc.awaitTermination()
Note: I am trying this in my local cluster, and with master as local[2].

Resources