There is a bunch of Mikrotik Windows Radius (NPS) AD. How can I handle statistics on VPN connections in some visual form? Mikrotik writes logs, I collect them in mikrotik syslog daemon, but I just can’t find how to process them later. At the moment, I think then hands in Excel, but this is inconvenient. NPS also writes something, but I can not find where to connect its data. The task is simple - I need to get the total time that a person spent with VPN enabled. Per day. Tell me, please, the simplest solution. thanks
Related
We have a real-time service, which gets binary messages from different sources (internal and external), then using a couple of NodeJS instances and one HAProxy instance, configured to route TCP traffic, we deliver them to our end-users and different services who consume the messages. HAProxy version is 1.8.14, NodeJS is 6.14.3, both hosted on a CentOS 7 machine.
Now we've got a complex problem with some "burst"s in the outbound interface of HAProxy instance. We are not sure whether the burst is real (e.g. some messages got stuck in Node and then network gets flooded with messages) or the problem is some kind of misconfig or an indirect effect of some other service (Both latter reasons are more likely, as sometimes we get these bursts during midnight, which we have minimal to zero load).
The issue is annoying right now, but it might get critical as it floods our outbound traffic so our real-time services experience a lag or a small downtime during working hours.
My question is, how can we track and record the nature or the content of these messages with minimum overhead? I've been reading through HAProxy docs to find a way to monitor this, which can be achieved by using a Unix socket, but we are worried about a couple of things:
How much is the overhead of using this socket?
Can we track what is going on in the servers using this socket? Or it only gives us stats?
Is there a way to "catch/echo" the contents of these messages, or find out some information about them? with minimum overhead?
Please let me know if you have any questions regarding this problem.
I have setup and Azure VM and installed a monitoring service that reaches out to various endpoints to verify a 200 response. The service is set to cycle through about 8 URL endpoints every 5 minutes or so.
We have run this service from multiple other servers outside of Azure, including virtual machines that are cheap, low end offerings.
While this machine is running on the lowest A0, it isn't doing anything else other than to run this service and call out to the various endpoints.
We are getting intermittent periods where one of the calls out of the list will fail for different periods that span 10-40 minutes at random periods several times a day.
The site or sites that fail are totally random and there is no down time from other monitor locations. We are sure that the connection problem is between Azure and the endpoints outside of Azure. There is no problem from anywhere outside of Azure.
I'm trying to figure out what could be causing this issue. It concerns me because we will be adding more services to Azure soon that use outside HTTP calls for credit card authorization and other API's.
Is this a known issue where outbound calls just don't function reliably at periods, or am I missing something in the setup or security settings?
Obviously, if the call makes it out and the response doesn't make it back, that is even worse as credit card charges would end up being pushed and the application would not register the proper response.
Anyone with some experience or insight would be greatly appreciated.
Thanks!
I find that very disturbing and hard to believe since, among a lot of other stuff, I run a service like that too... In my case I reach out to several (today, about 70) external addresses on both IPV4 and IPV6. I don't run A0s, and most of my machines are A3. I'll start a A0 to test it... if anything turns out <terminator>i'll be back</terminator> to report...
I know that there are several limitations regarding network traffic but i don't think you can reach them the way you're reporting...
My suggestion is to report that problem directly to MS via support ticket... most likely the problem is on the other side...
I have a requirement to write a load test measuring message transmission latencies. In order to simulate a large number of simultaneous uses without running into thread contention problem on one box, I'm spinning up multiple servers in Azure.
When I got my first results back, I was a little shocked to see that the results indicated the message was received before it was sent. I immediately realized that, while I had an implicit assumption that all the VMs would have their clocks synced to within milliseconds, that was clearly not the case.
I've spent several hours googling ways to resolve this, and I'm not getting anywhere. One thought was to have each VM query the time on a central server using NetRemoteTOD() using a technique similar to this NetRemoteTOD, and then establish a per-machine correction factor to be added to the time measured from the local machine's clock. However when I tried to run that method, I got a error 2184, "The service has not been started" I have verified that both the RPC service and the Windows Time service are running on the both the client and target machines, and I have not been successful in finding any information indicating what other service needs to be running (or even if the error really means what it seems to mean). (I also get the same error when running between my development desktop and a server on our corporate network. However, I can run it successfully to a PDC on the corporate network - but I can't find a PDC on Azure, since neither machine is part of a domain.)
So, does any one have either any information on what service needs to be started to get NetRemoteTOD (or the windows NET TIME command, which relies on NetRemoteTOD under the covers) working. Alternatively, does anyone have a suggestion for some other technique to get a consistent time reference across multiple VMs in Azure? (Note, I don't necessarily need their clocks synced, I just need a way to establish a consistent correction factor to reference the times to a common source. Note also, I need sub-second accuracy - probably about 100 msec will do.) Basically, I just need a windows function or shell command that will get me the time to sub-second accuracy on a given remote server.
Thanks in advance.
PS. Azure servers are running Server 2008 R2 SP1
We have a Microsoft DC R2 server running only an Interbase database application, all works fine and we can access this application via both Point to Site and Site to Site VPN.
Our transfer speeds for files is coming in at about 5Mbps which is fantastic.
When we access our software (locally) which pulls data from the server (Azure) we're seeing it clock speeds of about 125KBps.
This results in a 3-6 second wait before the dataset appears on screen within our application.
In a local environment this is done within 0.5 seconds.
I'm trying to get to the bottom of the issue as the Internet connection we are on is a 100Mbps feed.
I look at the Draytek router and assumed that this was the problem, however we have tested from multiple sites and ISPs and can't seem to get any improvement on application DB access speeds. SMB speeds remain impressive.
We're not too experienced in the Azure area but we can't work out any way of improving those speeds, if anybody has any suggestions that would be fantastic.
FYI We're using an A2 Windows deployment (approx 4Gb).
Regards,
Pottre11
Suppose that a data source sets a tight IP-based throttle. Would a web scraper have any way to download the data if the throttle starts rejecting their requests as early as 1% of the data being downloaded?
The only technique I could think of a hacker using here would be some sort of proxy system. But, it seems like the proxies (even if fast) would eventually all reach the throttle.
Update: Some people below have mentioned big proxy networks like Yahoo Pipes and Tor, but couldn't these IP ranges or known exit nodes be blacklisted as well?
A list of thousands or poxies can be compiled for FREE. IPv6 addresses can be rented for pennies. Hell, an attacker could boot up an Amazon EC2 micro instance for 2-7 cents an hour.
And you want to stop people from scraping your site? The internet doesn't work that way, and hopefully it never will.
(I have seen IRC servers do a port scan on clients to see if the following ports are open: 8080,3128,1080. However there are proxy servers that use different ports and there are also legit reasons to run proxy server or to have these ports open, like if you are running Apache Tomcat. You could bump it up a notch by using YAPH to see if a client is running a proxy server. In effect you'd be using an attacker's too against them ;)
Someone using Tor would be hopping IP addresses every few minutes. I used to run a website where this was a problem, and resorted to blocking the IP addresses of known Tor exit nodes whenever excessive scraping was detected. You can implement this if you can find a regularly updated list of Tor exit nodes, for example, https://www.dan.me.uk/tornodes
You could use a P2P crawling network to accomplish this task. There will be a lot of IPs availble and there will be no problem if one of them become throttled. Also, you may combine a lot of client instances using some proxy configuration as suggested in previous answers.
I think you can use YaCy, a P2P opensource crawling network.
A scraper that wants the information will get the information. Timeouts, changing agent names, proxies, and of course EC2/RackSpace or any other cloud services that have the ability to start and stop servers with new IP addresses for pennies.
I've heard of people using Yahoo Pipes to do such things, essentially using Yahoo as a proxy to pull the data.
Maybe try running your scraper on amazon ec2 instances. Every time you get throttled, startup a new instance (at new IP), and kill the old one.
It depends on the time the attacker has for obtaining the data. If most of the data is static, it might be interesting for an attacker to run his scraper for, say, 50 days. If he is on a DSL line where he can request a "new" IP address twice a day, 1% limit would not harm him that much.
Of course, if you need the data more quickly (because it is outdated quickly), there are better ways (use EC2 instances, set up a BOINC project if there is public interest in the collected data, etc.).
Or have a Pyramid scheme a la "get 10 people to run my crawler and you get PORN, or get 100 people to crawl it and you get LOTS OF PORN", as it was quite common a few years ago with ad-filled websites. Because of the competition involved (who gets the most referrals) you might quickly get a lot of nodes running your crawler for very little money.