Why does my node application sends tons of get requests? - node.js

Whenever I refresh a page in my MEAN application, my terminal that's running the application outputs tons of get requests. Is this slowing down my app? Can I get rid of some of these?
GET / 304 113.645 ms - -
GET /assets/stylesheets/css/bootstrap.min.css 304 11.846 ms - -
GET /assets/stylesheets/css/animate.css 304 19.584 ms - -
GET /assets/javascripts/fancybox/jquery.fancybox.css 304 18.081 ms - -
GET /assets/javascripts/owl.carousel/owl.carousel.css 304 17.841 ms - -
GET /assets/javascripts/owl.carousel/owl.theme.css 304 17.657 ms - -
GET /assets/stylesheets/css/style.css 304 16.135 ms - -
GET /assets/stylesheets/css/responsive.css 304 5.011 ms - -
GET /assets/javascripts/jplayer/skin/jukebox/css/jplayer.jukebox.css 304 5.593 ms - -
GET /assets/stylesheets/css/main.css 304 16.013 ms - -
GET /modules/users/css/users.css 304 13.936 ms - -
GET /assets/stylesheets/css/fontawesome/font-awesome.css 304 22.547 ms - -
GET /assets/javascripts/jpreloader/jpreloader.css 304 20.911 ms - -
GET /modules/core/css/core.css 304 13.623 ms - -
GET /assets/stylesheets/css/angular-snap.min.css 304 16.814 ms - -
GET /lib/jquery/dist/jquery.min.js 304 16.388 ms - -
GET /lib/angular/angular.js 304 16.650 ms - -
GET /lib/angular-resource/angular-resource.js 304 13.443 ms - -
GET /lib/angular-cookies/angular-cookies.js 304 28.193 ms - -
GET /lib/angular-animate/angular-animate.js 304 26.984 ms - -
GET /lib/angular-ui-router/release/angular-ui-router.js 304 9.312 ms - -
GET /lib/angular-sanitize/angular-sanitize.js 304 33.775 ms - -
GET /lib/angular-bootstrap/ui-bootstrap-tpls.js 304 33.470 ms - -
GET /lib/angular-ui-utils/ui-utils.js 304 31.331 ms - -
GET /lib/lodash/lodash.min.js 304 17.803 ms - -
GET /lib/angular-touch/angular-touch.js 304 22.445 ms - -
GET /lib/angularjs-dropdown-multiselect/dist/angularjs-dropdown-multiselect.min.js 304 22.940 ms - -
GET /application.js 304 24.288 ms - -
GET /lib/angular-media-player/dist/angular-media-player.js 304 79.501 ms - -
GET /lib/snapjs/snap.min.js 304 70.739 ms - -
GET /config.js 304 83.980 ms - -
GET /modules/accounts/accounts_module.js 304 59.103 ms - -
GET /modules/artistgenres/artistgenres.client.module.js 304 86.947 ms - -
GET /modules/artisttypes/artisttypes.client.module.js 304 37.306 ms - -
GET /modules/artistinterests/artistinterests.client.module.js 304 56.973 ms - -
GET /modules/files/files.client.module.js 304 13.750 ms - -
GET /modules/cities/cities.client.module.js 304 67.768 ms - -
GET /modules/messages/messages.client.module.js 304 69.352 ms - -
GET /modules/profile/profile_module.js 304 16.140 ms - -
GET /modules/users/users.client.module.js 304 18.733 ms - -
GET /modules/core/core.client.module.js 304 83.376 ms - -
GET /modules/accounts/pages/import/import_controller.js 304 79.469 ms - -
GET /modules/home/home_module.js 304 85.333 ms - -
GET /modules/neighborhoods/neighborhoods.client.module.js 304 91.851 ms - -
GET /modules/container/container_module.js 304 24.242 ms - -
GET /modules/accounts/pages/signup/signup_controller.js 304 34.483 ms - -
GET /modules/artistgenres/config/artistgenres.client.config.js 304 41.225 ms - -
GET /modules/accounts/pages/login/login_controller.js 304 42.122 ms - -
GET /modules/artistgenres/config/artistgenres.client.routes.js 304 38.534 ms - -
GET /modules/accounts/pages/apply/apply_controller.js 304 42.867 ms - -
GET /modules/accounts/config/accounts_routes.js 304 45.520 ms - -
GET /modules/artistgenres/services/artistgenres.client.service.js 304 24.213 ms - -
GET /modules/artistinterests/services/artistinterests.client.service.js 304 13.138 ms - -
GET /modules/artistinterests/config/artistinterests.client.config.js 304 61.058 ms - -
GET /modules/artistinterests/config/artistinterests.client.routes.js 304 63.784 ms - -
GET /modules/artistgenres/controllers/artistgenres.client.controller.js 304 62.882 ms - -
GET /modules/artistinterests/controllers/artistinterests.client.controller.js 304 85.128 ms - -
GET /modules/artisttypes/config/artisttypes.client.routes.js 304 49.166 ms - -
GET /modules/cities/controllers/cities.client.controller.js 304 25.185 ms - -
GET /modules/artisttypes/controllers/artisttypes.client.controller.js 304 58.415 ms - -
GET /modules/artisttypes/config/artisttypes.client.config.js 304 57.969 ms - -
GET /modules/artisttypes/services/artisttypes.client.service.js 304 78.072 ms - -
GET /modules/cities/config/cities.client.config.js 304 53.443 ms - -
GET /modules/container/components/footer/footer_controller.js 304 51.342 ms - -
GET /modules/cities/services/cities.client.service.js 304 56.805 ms - -
GET /modules/cities/config/cities.client.routes.js 304 20.592 ms - -
GET /modules/container/components/header/header_controller.js 304 81.731 ms - -
GET /modules/container/main/main_controller.js 304 96.035 ms - -
GET /modules/container/services/nav-links-menu_directive.js 304 96.880 ms - -
GET /modules/container/splash/splash_controller.js 304 103.059 ms - -
GET /modules/container/config/container-layout_routes.js 304 109.850 ms - -
GET /modules/container/services/nav-menu_service.js 304 37.800 ms - -
GET /modules/core/controllers/header.client.controller.js 304 30.084 ms - -
GET /modules/core/config/core.client.routes.js 304 16.356 ms - -
GET /modules/core/controllers/home.client.controller.js 304 29.481 ms - -
GET /modules/core/controllers/browse.client.controller.js 304 48.258 ms - -
GET /modules/files/services/files.client.service.js 304 26.917 ms - -
GET /modules/core/services/modals/standardModal_controller.js 304 85.918 ms - -
GET /modules/core/services/menus.client.service.js 304 62.773 ms - -
GET /modules/files/controllers/files.client.controller.js 304 65.790 ms - -
GET /modules/files/config/files.client.routes.js 304 72.539 ms - -
GET /modules/home/config/home_routes.js 304 26.648 ms - -
GET /modules/home/pages/browse/browse_controller.js 304 29.926 ms - -
GET /modules/messages/config/messages.client.routes.js 304 31.505 ms - -
GET /modules/home/pages/home/home_controller.js 304 47.686 ms - -
GET /modules/messages/controllers/messages.client.controller.js 304 32.608 ms - -
GET /modules/messages/services/messages.client.service.js 304 44.331 ms - -
GET /modules/home/services/toggle-class_directive.js 304 51.178 ms - -
GET /modules/neighborhoods/controllers/neighborhoods.client.controller.js 304 48.002 ms - -
GET /modules/neighborhoods/config/neighborhoods.client.config.js 304 50.873 ms - -
GET /modules/neighborhoods/services/neighborhoods.client.service.js 304 51.807 ms - -
GET /modules/neighborhoods/config/neighborhoods.client.routes.js 304 45.290 ms - -
GET /modules/profile/config/profile_routes.js 304 33.874 ms - -
GET /modules/profile/pages/profile-edit/profile-edit_controller.js 304 36.892 ms - -
GET /modules/users/config/users.client.routes.js 304 54.851 ms - -
GET /modules/profile/pages/profile/profile_controller.js 304 59.528 ms - -
GET /modules/users/config/users.client.config.js 304 61.343 ms - -
GET /modules/users/controllers/admin.client.controller.js 304 23.830 ms - -
GET /modules/users/controllers/settings.client.controller.js 304 31.434 ms - -
GET /modules/users/services/authentication.client.service.js 304 37.039 ms - -
GET /modules/users/controllers/profile.client.controller.js 304 26.199 ms - -
GET /modules/users/controllers/password.client.controller.js 304 26.991 ms - -
GET /modules/users/controllers/authentication.client.controller.js 304 24.791 ms - -
GET /assets/javascripts/jplayer/jquery.jplayer.min.js 304 40.561 ms - -
GET /assets/javascripts/bootstrap.min.js 304 48.406 ms - -
GET /assets/javascripts/angular-snap.min.js 304 57.688 ms - -
GET /assets/javascripts/jpreloader/jpreloader.min.js 304 55.109 ms - -
GET /modules/users/services/users.client.service.js 304 23.405 ms - -
GET /assets/javascripts/jplayer/add-on/jplayer.playlist.min.js 304 15.056 ms - -
GET /assets/javascripts/sly.min.js 304 25.137 ms - -
GET /assets/javascripts/wow/wow.min.js 304 24.524 ms - -
GET /assets/javascripts/jquery.hoverdir.js 304 30.592 ms - -
GET /assets/javascripts/jquery.easing.js 304 27.058 ms - -
GET /assets/javascripts/modernizr.js 304 58.101 ms - -
GET /assets/javascripts/jquery.parallax-1.1.3.js 304 59.616 ms - -
GET /assets/javascripts/masonry.pkgd.min.js 304 24.460 ms - -
GET /assets/javascripts/isotope.pkgd.min.js 304 27.051 ms - -
GET /assets/javascripts/owl.carousel/owl.carousel.min.js 304 54.712 ms - -
GET /assets/javascripts/fancybox/jquery.fancybox.pack.js 304 52.963 ms - -
GET /assets/stylesheets/css/bootstrap.css.map 304 47.436 ms - -
GET /lib/jquery/dist/jquery.min.map 304 49.565 ms - -
GET /modules/container/main/main_template.html 304 171.919 ms - -
The page that this code came from doesn't need at least half of these...

When you launch your server and visit your main page it will load all necessary files for all the pages in your web application.
This is the power of AngularJS = No more refresh on pages. Imagine if you need to reload all files when you change to another page? Don't worry ;)
If you see some resources that you don't need, you can modify in your crud configuration file or the all.js file (on MEANJS).
Links to help you:
MEANJS(http://meanjs.org/docs.html)
MEAN.IO (http://learn.mean.io/)

Related

Reading RDD with the header and filtering it

I am using the code below to load all the log file into a RDD.
logs_rdd = sc.textFile('/user/jovyan/weblogs/')
If I look at the first 10 rows with
logs_rdd.take(10)
It is giving me this output
['3.94.78.5 - 69827 [15/Sep/2013:23:58:36 +0100] "GET /KBDOC-00033.html HTTP/1.0" 200 14417 "http://www.loudacre.com" "Loudacre Mobile Browser iFruit 1"',
'3.94.78.5 - 69827 [15/Sep/2013:23:58:36 +0100] "GET /theme.css HTTP/1.0" 200 3576 "http://www.loudacre.com" "Loudacre Mobile Browser iFruit 1"',
'19.38.140.62 - 21475 [15/Sep/2013:23:58:34 +0100] "GET /KBDOC-00277.html HTTP/1.0" 200 15517 "http://www.loudacre.com" "Loudacre Mobile Browser Ronin S1"',
'19.38.140.62 - 21475 [15/Sep/2013:23:58:34 +0100] "GET /theme.css HTTP/1.0" 200 13353 "http://www.loudacre.com" "Loudacre Mobile Browser Ronin S1"',
'129.133.56.105 - 2489 [15/Sep/2013:23:58:34 +0100] "GET /KBDOC-00033.html HTTP/1.0" 200 10590 "http://www.loudacre.com" "Loudacre Mobile Browser Sorrento F00L"',
'129.133.56.105 - 2489 [15/Sep/2013:23:58:34 +0100] "GET /theme.css HTTP/1.0" 200 12295 "http://www.loudacre.com" "Loudacre Mobile Browser Sorrento F00L"',
'217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /ronin_s4_sales.html HTTP/1.0" 200 845 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"',
'217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /theme.css HTTP/1.0" 200 738 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"',
'217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /code.js HTTP/1.0" 200 938 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"',
'217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /ronin_s4.jpg HTTP/1.0" 200 5552 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"']
The question is I want to Filter out the requests for jpg files into a new RDD. You can see the last line of the output where the GET request start. I am confused whether there is a header I did not load or do I have to create a schema for the RDD and then filter the RDD. Anything would be helpful. Thank you.
instead of using rdd i would recommend to use DataFrame as it is easy to process data with Header(column name)
Since your Data does not have schema/Header so when you load Data in a Dataframe it will automatically create a header for your data
df.show(truncate=False)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|value |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|'3.94.78.5 - 69827 [15/Sep/2013:23:58:36 +0100] "GET /KBDOC-00033.html HTTP/1.0" 200 14417 "http://www.loudacre.com" "Loudacre Mobile Browser iFruit 1"', |
| '3.94.78.5 - 69827 [15/Sep/2013:23:58:36 +0100] "GET /theme.css HTTP/1.0" 200 3576 "http://www.loudacre.com" "Loudacre Mobile Browser iFruit 1"', |
| '19.38.140.62 - 21475 [15/Sep/2013:23:58:34 +0100] "GET /KBDOC-00277.html HTTP/1.0" 200 15517 "http://www.loudacre.com" "Loudacre Mobile Browser Ronin S1"', |
| '19.38.140.62 - 21475 [15/Sep/2013:23:58:34 +0100] "GET /theme.css HTTP/1.0" 200 13353 "http://www.loudacre.com" "Loudacre Mobile Browser Ronin S1"', |
| '129.133.56.105 - 2489 [15/Sep/2013:23:58:34 +0100] "GET /KBDOC-00033.html HTTP/1.0" 200 10590 "http://www.loudacre.com" "Loudacre Mobile Browser Sorrento F00L"',|
| '129.133.56.105 - 2489 [15/Sep/2013:23:58:34 +0100] "GET /theme.css HTTP/1.0" 200 12295 "http://www.loudacre.com" "Loudacre Mobile Browser Sorrento F00L"', |
| '217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /ronin_s4_sales.html HTTP/1.0" 200 845 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"', |
| '217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /theme.css HTTP/1.0" 200 738 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"', |
| '217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /code.js HTTP/1.0" 200 938 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"', |
| '217.150.149.167 - 4712 [15/Sep/2013:23:56:06 +0100] "GET /ronin_s4.jpg HTTP/1.0" 200 5552 "http://www.loudacre.com" "Loudacre Mobile Browser MeeToo 1.0"' |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
df.printSchema()
root
|-- value: string (nullable = true)
df=spark.read.text("Path to text file")
df.filter(df.value.rlike('.jpg')).show(truncate=False)

How to report a bug of GoogleBot?

Over the last days, Google Bot tries to read one URL of our main site over and over again, leading to a DDOS attack :) Our website got very slow because of the massive requests of the Google Crawler.
Here an excerpt for the curious ones (or if a Google engineers reads this post):
66.249.76.54 - - [27/May/2019:06:31:23 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/594749/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/user/impressum HTTP/1.0" 200 32603
66.249.76.54 - - [27/May/2019:06:31:23 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/403551/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/bestusers HTTP/1.0" 200 32603
66.249.76.55 - - [27/May/2019:06:31:23 +0200] "GET /235432/tag/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/403551/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/403551/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/schreibregeln HTTP/1.0" 200 32603
66.249.76.54 - - [27/May/2019:06:31:23 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/594749/tag/chat HTTP/1.0" 200 32603
Or here (see the different IPs, so there are several bots):
66.249.76.54 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/386961/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/user/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/punkte HTTP/1.0" 200 32587
66.249.76.55 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/user/403551/agb HTTP/1.0" 200 32587
66.249.76.56 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/user/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/403551/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/qa-theme/lounge/js/lounge.min.js?v=2019-01-17 HTTP/1.0" 200 32587
66.249.76.55 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/594749/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/luckentext-zum-thema-extrema-funktionenschar-ft-x-1-2-tx-2-2-t HTTP/1.0" 200 32587
66.249.76.58 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/323274/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/323274/user/Lu HTTP/1.0" 200 32587
66.249.76.57 - - [27/May/2019:09:24:42 +0200] "GET /235432/~plot~+4x%5E2%3B+4%2Ax%5E2+++4%2A%281/32%29%2Ax+-+15%2A%281/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/154807/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/323274/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/235432/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/32)*x%20-%2015*(1/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/user/user/tag/tag/~plot~%204x%5E2;%204*x%5E2%20+%204*(1/badges HTTP/1.0" 200 32587
The false link that was leading to this problem:
~plot~ 4x^2; 4*x^2 + 4*(1/32)*x - 15*(1/32)^2; x; [[0.1]]~plot~
Where can I report a bug of GoogleBot?
There seems to be no official way how to report a bug.
Here is the link to report a crawling bug of Google Bot:
https://www.google.com/webmasters/tools/googlebot-report
Report a problem with how Googlebot crawls your site.
You can report problems only for domain-level properties (for example, "www.example.com/")
The rate at which Google crawls your page depends on many factors:
The URLs we already know about
Links from other web pages (within your site and on other sites)
URLs listed in your Sitemap.
For most sites, Googlebot shouldn't access your site more than once every few seconds on average.
Probably the report link was not easy to find since you need a Google Webmaster account and can apparently only report your own websites.

How to Protect Flask from ZMEU

Forgive me if this is a duplicate question, but I'm running a Flask app using Celery and Rabbit MQ in a Kubernetes service. The service is run as a public-facing LoadBalancer. The problem I've seen is with ZMEU scan attacks which mess up the Flask uri structure and render the app unusable:
10.240.0.4 - - [19/Apr/2018 04:48:05] "GET / HTTP/1.1" 400 -
10.240.0.4 - - [19/Apr/2018 04:48:10] "GET /index.action HTTP/1.1" 400 -
10.244.2.1 - - [19/Apr/2018 07:32:29] "GET / HTTP/1.1" 400 -
10.244.2.1 - - [19/Apr/2018 08:54:38] "GET / HTTP/1.1" 400 -
10.244.2.1 - - [19/Apr/2018 11:13:06] "GET /w00tw00t.at.blackhats.romanian.anti-sec :) HTTP/1.1" 400 -
10.240.0.4 - - [19/Apr/2018 11:13:11] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 400 -
10.240.0.4 - - [19/Apr/2018 11:13:16] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 400 -
10.244.2.1 - - [19/Apr/2018 11:13:21] "GET /pma/scripts/setup.php
...
This occurs after successful pings to the URI. I have a healthcheck occurring because this silently renders the app unreachable, though the container itself is healthy and doesn't exit.
While I am working on securing the endpoint itself, I feel that it's more of a band-aid, and the app code should be resilient to this type of attack. Is there a way for me to catch any unspecified URI (not defined by an #app.route() decorator)?
EDIT: I've made an attempt to only accept requests that have a uri header in the request by using redirects in a general uri-path http://my.flask.app:80/ to point to the proper service, but this hasn't remedied the problem

In expressjs v4, how do i put timestamps on the logs?

I have tried to use morgan with the following
var morgan = require('morgan');
app.use(morgan('combined'));
But I get a response like this
www-0 ::1 - - [15/Apr/2016:10:43:13 +0000] "GET /dasda HTTP/1.1" 404 41 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
www-0 GET / 304 4.624 ms - -
www-0 GET /font-awesome/css/font-awesome.min.css 304 1.953 ms - -
www-0 GET /css/common-561d902c.min.css 304 2.097 ms - -
www-0 GET /images/logo.png 304 3.072 ms - -
Why the logs GET / 304 4.624 ms - - and such are not affected by this? How can I put timestamps there?
Thanks
Search if any other package in your project overwrites somehow the morgan's configuration.

java.io.IOException: Too many open files kafka-rest proxy

I am using confluent kafka platform . I have a topic with 4 partition and replication factor of 2. Single zookeeper, three brokers and kafka-rest proxy server. Now I am load testing the system with siege running 1000 users with a list of api which in turn hit kafka producer. I have my producer and consumer using the rest proxy (kafka-rest). I am getting following issue:
{ [Error: getaddrinfo EMFILE] code: 'EMFILE', errno: 'EMFILE', syscall: 'getaddrinfo' }
In kafka-rest log I can see:
[2016-02-23 07:13:51,972] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 14 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,973] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 15 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,974] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 12 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,978] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 6 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,983] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 6 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,984] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 4 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,985] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 7 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,993] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 3 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,994] INFO 127.0.0.1 - - [23/Feb/2016:07:13:51 +0000] "POST /topics/endsession HTTP/1.1" 200 120 4 (io.confluent.rest-utils.requests:77)
[2016-02-23 07:13:51,999] WARN Accept failed for channel java.nio.channels.SocketChannel[closed] (org.eclipse.jetty.io.SelectorManager:714)
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processAccept(SelectorManager.java:706)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:648)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:611)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
So I went through a lot of questions related to this. Set my ec2 machine paramenters so that I dont get too many open file error. But its not solved. I have reduced the TIME_WAIT to 30 seconds. ulimit -n is 80000.
I have collected some stats and look like kafka rest proxy which is running on `localhost:8082 causing too many connections.
How do I solve this issue? Also sometimes when error is coming and then I stop my siege test but again when TIME_WAIT connections are reduced, I restart my load test with 1 user only still I see the same issue. Some issue in rest proxy wrapper for node js?
`
You need to increase the ulimit for that process. In order to check the ulimit for a particular process , run this:
sudo cat /proc/<process_id>/limits
in order to increase the ulimit for process running via supervisord, you can increase minfds in supervisord.conf

Resources