So today a client of mine sent out a marketing newsletter to around 140k clients that included a link to our web app.
What happened next was my web app experienced a flood of traffic (over 9000 requests in 15 minutes) from Microsoft-owned IP addresses in the range 40.94././ requesting that specific page on my site. This took the app down for all my clients until I managed to restart it.
It seems like the scan took place regardless of whether a user clicked on the link or not, as there are no other IP addresses in the request logs for the same url during this period.
So my question is, was this Microsoft pre-emptively scanning that link as it was delivered to newsletter subscribers? Or does the scan only happen when the link is clicked - I've found conflicting information on this, and as mentioned I see no other IP address requests during this period.
And secondly, how can I stop this from happening in the future - is my only option to blacklist this IP range, or are there other strategies?
So for anyone struggling with something similar I can confirm that Microsoft pre-emptively scans the links inside a received email before it lands in the recipients inbox.
The effect of this is that if a huge newsletter is sent to hundreds of thousands of recipients, Microsoft effectively triggers a wave of traffic to your server.
It would appear the only solution is to black-list their range of IPs, or ensure you have some throttling mechanism in place.
One of the solution as mentioned in the other answer is to block the range of ip addresses that belong to Microsoft Safelink in order to prevent the scans from accessing the website.
Other solution might be to use JS Challenge such as this available in Cloudflare. With such a solution each user has to go through a website that first verifies if he/she is using a real browser and only if that is the case he/she is redirected to the target website.
Such a JS Challenge can be enabled only for those accessing website from links in the email so that anyone using browser to directly access a website won't be affected
Related
I could not find a clear answer to this question. I am looking for some up to date best-practice advice for the following two topics:
Displaying email addresses on my website (also linked via mailto:).
Having an "unprotected" contact form on my website (no captcha
etc.).
This is all for a static website served via aws S3.
I am afraid of getting hit by spammers.
How could I avoid this in an elegant way (ideally unnoticed by the user)?
Nothing you can do Nothing, you have no captcha therefore you are at the mercy of spammers.
I am working on a web application which allows users to share stuff on a web-page by clicking on an 'email to friend' link; similar to what extole is doing here
http://www.american-giant.com/mens-heavyweight-full-zip-hooded-sweatshirt-product.html
on this page if you click on the email icon near "REFER & GET $15", you will see a pop-up where you can enter your own email and a friends email and can edit the subject of the email. When you click send the data is sent to the backend as json. They are using a plain simple url to do this i.e. http://refer.american-giant.com/v2/share.
The problem for me is that somehow spammers got hold of my url (can't mention here) and now they are using it to spam others by using some sort of a script. What I did is I placed a check in the backend api to block an ip if more than 5 share requests originate from it, but it seems that the spammers have a lots of ips (more than 30,000 from what I counted in my logs) so they are still able to send lots of email. One possible solution is to use a captcha to thwart the spamming script. But I am curious that how extole is doing it. They aren't using any captchas; and they are famous too, so it is unlikely that spammers don't know about their publicly accessible api. Can any one shed some light on this?
Note:
1. I am using a third party email service to send the emails.
2. Users are not required to sign in as this defeats the purpose of sharing on a simple website
3. Users can edit the subject and body, thus these are sent to the api call and this is what allows the spammers to abuse the api with their own stuff.
I have couple Android apps on PlayStore, which use In-App purchases. I use Google App Engine for my backend. I see some users calling the APIs abnormally/repeatedly (may be to reverse engineer or hack?). I can figure out the IP address, Gmail ID, etc. How to prevent these people from accessing my API?
One suggestion is to use dos.xml
But these morons seem to constantly change the IP addresses, so it is painful to keep updating this list.
Is there a way in App Engine to black list users? or computers/devices?
If we know the google(Gmail) Ids of these ba*t*r*s, how/where do we report those? This page seems to be the right place to start, but it is not clear where to send email.
This page seems be more appropriate for vulnerabilities, but this is not such a case.
"Viewing top users in the Administration Console" section in DoS page says I should see a table of IP addresses which are using the API frequently. But I dont see such table in Admin console. Do I need to be a paid (Google App Engine) user?
Any help is greatly appreciated.
Yes, GAE allows for a blacklist, via dos.xml (dos.yaml for Python or PHP). If you don't want to have to keep updating the IP addresses, you may just have to check the user id, and serve them some message. But, that requires actually servicing the request, to check the id, etc. So, if it is a true DOS attack, it will succeed, as you have to still service the request. Using dos.xml cuts that off at the backend, so would be the best way to go.
I suggest a script to log the IP addresses in real time for those you want to ban, to make updating dos.xml less painful.
I have a dedicated server hosting a website of mine and I have about 10% of my traffic unaccounted for.
The path of the clicks is as follows
Ads on Facebok ==> My Website
I have Google Analytics script on My Website (on the bottom) and it should fire off whenever a person lands on the page.
The problems is that if I have 4000 clicks on the Ad (tracked by Facebook), Google Analytics tells me I have about 3600 people landing on My Website.
I also invested in real-time tracking software like Clicky and it gives me similar results to Google Analytics. (just in case GA is not accurate)
So I have narrowed it down to 3 scenarios:
1) The Ad clicks aren't being tracked properly by Facebook (I have made sure this is not the problem)
2) The page is taking too long for some people to load and they are hitting the back button before Google Analytics can be triggered.
3) Some connection are dropping from the Ad to My Website.
Can anyone recommend a way I can make sure 2 and 3 aren't happening? and if they are
how would I fix them.
I'm going to make an assumption that you're using Apache. It should be possible to parse the Apache logs to extract connections from unique IP addresses. Hopefully the URL requested by the client will contain some sort of path indicating that it was directed from Facebook.
Link:
Get unique visitors from apache log file
I have a site on a dedicated server with it's own IP range that has been running for a good few years. We have a notification email address (mailout#domain.com) which we use to send automated emails (activation emails when a user signs up and notification emails if something relevant to them happens, eg someone befriends them or comments on their picture etc). Users can select whether to receive these notifications or not. We have SPF and RDNS setup.
Email from all our other email accounts go to hotmail/gmail/yahoo mail etc correctly into the inbox. However any mail sent from the mailout#domain.com account (whether automatically by the server or manually via outlook) is delivered correctly to the inboxes for yahoo and gmail however goes into Junk in Hotmail (but other #domain.com addresses deliver to hotmail's inbox correctly). It says at the top of the message that MS Smartscreen marked this message as junk. I signed up for MS Smart Network Data Services to monitor the IP and it says it's not blocked but it displays Bot-like behaviour (which kind of makes sense as our notifications are kind of bot like even though they're not spam).
I can't work out what to do to prevent this from happening, we've authenticated the email, there's obviously not a general block on the IP as emails from different accounts on the same domain are going through successfully. It doesn't seem to be the format of the email either because if I send identical emails from mailout#domain.com and contact#domain then the one from contact# gets through to the inbox but the one from mailout# goes through to junk.
I can't really work out what to do and obviously trying to get MS to sort it out is never going to happen and i've used all their available tools. I can obviously try setting up a new email address (eg noreply#domain.com) and using that for notifications but i assume it will only be a matter of time before that gets blocked as well.
I would be immensely grateful for any suggestions anyone has!
Thanks so much,
Dave
You don't have many options. Try to do as many of the following as you can:
Reach out to MS support (don't discard this notion)
Implement DKIM and possibly DMARC (which are vastly more informative than SPF)
Change your IP address to something cleaner
Find and follow bulk sender best practices, e.g. M³AAWG's BCPs, perhaps the Help – I'm on a Blocklist doc