SID-02438: Block intruders

Status:	Answered	TWiki version:	6.1.0	Perl version:
Category:	CategoryAccessControl	Server OS:	Linux 3.13.0-042stab134.8 (x86_64-linux)	Last update:	6 years ago

My TWiki is under attack by Huawei. My logs show numberous attempts of IP adresses belonging to Huawei to enter my TWiki. What is best to do? Block by IP (see BlockingBasedOnIP)?

| 2020-03-24 - 15:38:39 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.161.87 |
| 2020-03-24 - 15:38:45 | guest | rdiff | TWiki.TWikiScripts | 17 1 | 114.119.162.168 |
| 2020-03-24 - 15:38:51 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.163.206 |
| 2020-03-24 - 15:38:58 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.166.75 |
| 2020-03-24 - 15:38:58 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.163.84 |
| 2020-03-24 - 15:38:59 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.164.208 |
| 2020-03-24 - 15:39:06 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.160.103 |
| 2020-03-24 - 15:39:15 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.167.214 |
| 2020-03-24 - 15:39:20 | guest | rdiff | TWiki.TWikiSystemRequirements | 43 1 | 114.119.164.169 |

-- Emiel Van Riel - 2020-03-24

Discussion and Answer

This seems to be a greedy spider, very common on TWiki.org. You can block by IP. You can also require authentication for all non-essential scripts like the rdiff you point out.

Here is the authentication configure setting of TWiki.org:

$TWiki::cfg{AuthScripts} = 'attach,edit,manage,rename,save,upload,viewauth,rdiff,rdiffauth,rest,mdrepo';

Here is part of the httpd config of TWiki.org:

BrowserMatchNoCase 80legs blockAccess
BrowserMatchNoCase Aboundex blockAccess
BrowserMatchNoCase ^Accoona blockAccess
BrowserMatchNoCase ActiveAgent blockAccess
# cut, long list

<Directory "/path/to/twiki/bin">

    RewriteEngine On
    RewriteCond %{SERVER_NAME} =www.twiki.org
    RewriteRule ^ https://twiki.org%{REQUEST_URI} [END,NE,R=permanent]

    RewriteCond %{QUERY_STRING}  !^$
    RewriteCond %{QUERY_STRING}  !^(slideshow=|note=|search=|skin=text|skin=plain|tag=|dir=|ip=|TWikiGuestCache=)
    RewriteRule view/(.*) /cgi-bin/viewauth/$1

    AllowOverride None
    Order Allow,Deny
    Allow from all
    Deny from env=blockAccess
    Deny from 35.153.56.178 107.77.245.0/24 23.254.203.154 104.131.66.0/24 # cut, long list

    Options ExecCGI FollowSymLinks
    SetHandler cgi-script

</Directory>

Add bad user agents to BrowserMatchNoCase, add bad IP addresses (& blocks) to Deny from

-- Peter Thoeny - 2020-03-24

Another method which helps against "well-behaved" spiders is to provide a /robots.txt URL, as described in Wikipedia:Robots_exclusion_standard. I see lots of different spiders in my log files, but all of them only grab that URL once per day-

So, regardless of whether you deploy the hard-blocks by TWiki, a robots.txt file is easy to write and therefore a nice additional measure which helps against all well-behaved robots which will come up in the future or haven't discovered your server yet.

-- Harald Jörg - 2020-03-24

Thanks!

-- Emiel Van Riel - 2020-03-25

I used the standard TWiki:TWiki.ApacheConfigGenerator for my setup. It seems that in $TWiki::cfg{AuthScripts} the rdiff -script is missing. So I added it yo my configuration. $TWiki::cfg{AuthScripts}='attach,edit,manage,rename,save,upload,viewauth,rdiff,rdiffauth,rest,mdrepo';

-- Emiel Van Riel - 2020-03-25

To me this looks like som serious hacking attempt:

dows/win.ini HTTP/1.1
[Tue Mar 03 02:50:09.694618 2020] [core:error] [pid 12449:tid 140516207298304] [client 81.95.5.34:48666] AH00126: Invalid URI in request GET /../../../../../../../../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:09.749315 2020] [core:error] [pid 12449:tid 140516228278016] [client 81.95.5.34:48694] AH00126: Invalid URI in request GET ../../../../../../../../../../../../windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:09.803208 2020] [core:error] [pid 12449:tid 140516312196864] [client 81.95.5.34:48734] AH00126: Invalid URI in request GET ../../../../../../../../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:09.853746 2020] [core:error] [pid 12449:tid 140516354156288] [client 81.95.5.34:48774] AH00126: Invalid URI in request GET ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\windows\\win.ini HTTP/1.1
[Tue Mar 03 02:50:09.911182 2020] [core:error] [pid 12449:tid 140516186318592] [client 81.95.5.34:48804] AH00126: Invalid URI in request GET ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\winnt\\win.ini HTTP/1.1
[Tue Mar 03 02:50:10.278759 2020] [core:error] [pid 12450:tid 140516460365568] [client 81.95.5.34:49032] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:11.336212 2020] [core:error] [pid 12449:tid 140516291217152] [client 81.95.5.34:49110] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:11.923249 2020] [core:error] [pid 12449:tid 140516259747584] [client 81.95.5.34:49910] AH00126: Invalid URI in request GET %5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5cwinnt%5cwin.ini HTTP/1.1
[Tue Mar 03 02:50:12.184192 2020] [core:error] [pid 12449:tid 140516228278016] [client 81.95.5.34:50160] AH00126: Invalid URI in request GET /././././././../../../../../windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:12.235340 2020] [core:error] [pid 12449:tid 140516312196864] [client 81.95.5.34:50354] AH00126: Invalid URI in request GET /././././././../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:12.352104 2020] [core:error] [pid 12449:tid 140516354156288] [client 81.95.5.34:50400] AH00126: Invalid URI in request GET .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:12.409784 2020] [core:error] [pid 12450:tid 140516175828736] [client 81.95.5.34:50448] AH00126: Invalid URI in request GET .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:14.668792 2020] [core:error] [pid 12449:tid 140516343666432] [client 81.95.5.34:51674] AH00126: Invalid URI in request GET /../../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:14.712247 2020] [core:error] [pid 12449:tid 140516460365568] [client 81.95.5.34:51770] AH00126: Invalid URI in request GET /./../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:14.757681 2020] [core:error] [pid 12449:tid 140516449875712] [client 81.95.5.34:51804] AH00126: Invalid URI in request GET //../../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:14.827108 2020] [core:error] [pid 12449:tid 140516291217152] [client 81.95.5.34:51836] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd HTTP/1.1
[Tue Mar 03 02:50:17.034371 2020] [core:error] [pid 12450:tid 140516333176576] [client 81.95.5.34:52612] AH00126: Invalid URI in request GET /././././././../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:17.622495 2020] [core:error] [pid 12449:tid 140516301707008] [client 81.95.5.34:36744] AH00126: Invalid URI in request GET /../../../../../../../../../../../../windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:17.746598 2020] [core:error] [pid 12449:tid 140516322686720] [client 81.95.5.34:36798] AH00126: Invalid URI in request GET /../../../../../../../../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:19.128196 2020] [core:error] [pid 12449:tid 140516270237440] [client 81.95.5.34:37052] AH00126: Invalid URI in request GET ../../../../../../../../../../../../windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:19.460218 2020] [core:error] [pid 12449:tid 140516333176576] [client 81.95.5.34:37904] AH00126: Invalid URI in request GET ../../../../../../../../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:19.784537 2020] [core:error] [pid 12450:tid 140516186318592] [client 81.95.5.34:37968] AH00126: Invalid URI in request GET ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\windows\\win.ini HTTP/1.1
[Tue Mar 03 02:50:20.894199 2020] [core:error] [pid 12449:tid 140516249257728] [client 81.95.5.34:38190] AH00126: Invalid URI in request GET ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\winnt\\win.ini HTTP/1.1
[Tue Mar 03 02:50:23.004915 2020] [core:error] [pid 12449:tid 140516449875712] [client 81.95.5.34:40320] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:23.102401 2020] [core:error] [pid 12449:tid 140516280727296] [client 81.95.5.34:40392] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:23.781053 2020] [core:error] [pid 12449:tid 140516259747584] [client 81.95.5.34:40878] AH00126: Invalid URI in request GET %5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5cwinnt%5cwin.ini HTTP/1.1
[Tue Mar 03 02:50:27.400435 2020] [core:error] [pid 12449:tid 140516217788160] [client 81.95.5.34:43298] AH00126: Invalid URI in request GET /././././././../../../../../windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:27.489781 2020] [core:error] [pid 12449:tid 140516364646144] [client 81.95.5.34:43362] AH00126: Invalid URI in request GET /././././././../../../../../winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:27.911118 2020] [core:error] [pid 12449:tid 140516343666432] [client 81.95.5.34:43442] AH00126: Invalid URI in request GET .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/windows/win.ini HTTP/1.1
[Tue Mar 03 02:50:28.040960 2020] [core:error] [pid 12449:tid 140516460365568] [client 81.95.5.34:43704] AH00126: Invalid URI in request GET .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/winnt/win.ini HTTP/1.1
[Tue Mar 03 02:50:30.915458 2020] [core:error] [pid 12449:tid 140516165338880] [client 81.95.5.34:45432] AH00126: Invalid URI in request GET /../../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:31.024317 2020] [core:error] [pid 12449:tid 140516481345280] [client 81.95.5.34:45478] AH00126: Invalid URI in request GET /./../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:31.128151 2020] [core:error] [pid 12449:tid 140516217788160] [client 81.95.5.34:45532] AH00126: Invalid URI in request GET //../../../../../../../../../../../../etc/passwd HTTP/1.1
[Tue Mar 03 02:50:32.516906 2020] [core:error] [pid 12450:tid 140516481345280] [client 81.95.5.34:46472] AH00126: Invalid URI in request GET /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd HTTP/1.1
[Tue Mar 03 02:50:34.529423 2020] [core:error] [pid 12450:tid 140516228278016] [client 81.95.5.34:47020] AH00126: Invalid URI in request GET /././././././../../../../../etc/passwd HTTP/1.1

-- Emiel Van Riel - 2020-03-25

Ah, this is hacking, sort of. You get that whenever you have a server in the internet. I'm afraid it is very hard to prevent this sort of attacks unless your provider has an intrusion detection system in place.

You could check the web server's log file for the user agent (or adjust LogFileFormat to include it). For a server I'm monitoring, I get a lot of similar stuff by robots which claim to work for some sort of internet security. But again, many of these obey robots.txt, so also check whether your server reports 404 status code for /robots.txt.

-- Harald Jörg - 2020-03-25

I am using the standard TWiki installation on my virtual host. /robots.txt is in /var/www/mytwiki.com/twiki/misc and has -wr--r--r-- rights for www-data:www-data. I think robots should be able to read this file, but I don't know how to check.

So now I made a new robot.text file in my document root /var/www/mytwiki.com/ , containing

User-agent: * 
Disallow: /twiki

and added to mytwiki.com.conf

<Location "/robots.txt">
 SetHandler None
 Require all granted
</Location>
Alias /robots.txt /var/www/mytwiki.com/robots.txt

I hope this wil do the trick. Meanwhile massive attemps of the spiders are delaying my server. I asked the operator to help me to block these intruders. But there's no reaction yet.

-- Emiel Van Riel - 2020-03-26

Verifying whether your robots.txt is served would be simple: Just point a browser to http://your.server.com/robots.txt. If (and only if) your browser can read it, robots can read it, too!

-- Harald Jörg - 2020-03-26

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.