Friday, September 5, 2014

Mitigating a DDOS attack – can you really do it?

Today I would like to smash your confidence about the security of your organization. You spent hundreds of dollars or euros on your security devices and you think you are safe. I believe there are many organizations like yours that think the same thing. Once they become victims, they realize they are not as safe as they had thought. Let’s talk about DOS/DDOS attacks. I think most people have some knowledge about them. Today we should not talk only about DOS attacks because the risk of being seriously impacted is very low. Most network devices can easily mitigate such attacks because they come from one source IP address. The situation is different when we receive illegitimate traffic from many IP addresses. There are two possibilities here: we can be flooded by traffic from a real, existing source IP address or addresses; or there is one real source IP, but the packets contain random source IPs, which means you are not able to block them easily. We can distinguish the following types of DOS/DDOS attacks:
  • DOS – one attacker who initiates an attack from one machine (one IP address)
  • DDOS – one attacker who randomly changes the source IP address
  • DDOS – traffic sent from ‘zombies’, i.e. infected machines managed by one central host. I would like to analyze these three types of attacks in order to mitigate them.  
I prepared a virtual lab with the following elements:
  • ISP – Cisco router – IOS 15.2
  • Perimeter Firewall – ASA – 8.4


 
10.0.0.10    <---------- A T T A C K ------------------------------|

  /-----\         -----             /----\                     -----------
 |WEB SRV|-------| ASA1 |----------| ISP  |--[INTERNET ]------| ATTACKER  |
  \-----/         -----             \----/                     -----------


Fig.1 Network Diagram

From the ‘Attacker’ machine I will generate traffic to the ‘Web server’, located in the DMZ. This scenario is the most common for small and medium-sized businesses (SMBs) segment. Usually these businesses manage their own firewall and have one ISP. IPS/IDS appliances are not too popular with SMBs .


Lab #1 – DOS attack 

I started the first simulation with a DOS attack on a server (10.0.0.10). I set the firewall to permit all TCP traffic from any source to the destination 10.0.0.10 on the port 80. First I checked the current configuration and utilization on all devices:
 
ISP#sh processes cpu sorted
CPU utilization for five seconds: 0%/0%; one minute: 1%; five minutes: 7%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 239       17132      254464         67  0.23%  0.26%  0.30%   0 ISG MIB jobs Man
 112        9740      127455         76  0.15%  0.14%  0.15%   0 IPAM Manager
   2        2660         874       3043  0.07%  0.04%  0.05%   0 Load Meter
  59        6960        1304       5337  0.07%  0.06%  0.07%   0 HC Counter Timer
 268        5224         877       5956  0.07%  0.07%  0.07%   0 Compute load avg
 174        3548       42094         84  0.07%  0.06%  0.07%   0 RBSCP Background
  82        7964      127454         62  0.07%  0.15%  0.15%   0 IP ARP Retry Age
   8           0           2          0  0.00%  0.00%  0.00%   0 Timers
   7          96          29       3310  0.00%  0.00%  0.00%   0 DiscardQ Backgro
  10       26904        1459      18440  0.00%  0.08%  0.15%   0 ARP Input
   6       11828         950      12450  0.00%  0.00%  0.00%   0 Pool Manager
 --More--

Fig.2 CPU usage – ISP router
 
ISP#sh run int gig0/0
Building configuration...

Current configuration : 165 bytes
!
interface GigabitEthernet0/0
 ip address 192.168.157.10 255.255.255.0
 ip access-group OUTSIDE in
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
end
!


ISP#sh run | b access-list
ip access-list extended OUTSIDE
 permit tcp any host 10.0.0.10 eq www log
!

Fig.3 Interface settings and ACL on ISP
 
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 0%; 1 minute: 0%; 5 minutes: 0%
PER-FW#

Fig.4 CPU usage – Perimeter Firewall
 
PER-FW# sh access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www (hitcnt=1) 0x5f077c5e
PER-FW#

Fig.5 Number of connection passed by the firewall

Let’s start the DOS attack using the ‘t50’ tool (part of the KALI distribution).
 
root@kali:~# t50 10.0.0.10 --flood --syn -s 192.168.157.200 --dport 80
entering in flood mode...
hit CTRL+C to break.
T50 5.4.1-rc1 successfully launched on Aug 30th 2014 22:41:39

After one minute I found almost 80,000 SYN packets:
 
ISP#sh access-lists
Extended IP access list OUTSIDE
    10 permit tcp any host 10.0.0.10 eq www log (79287 matches)
ISP#

As we see, CPU usage is close to 100%:
 
ISP#sh processes cpu
CPU utilization for five seconds: 96%/100%; one minute: 32%; five minutes: 30%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1           0           2          0  0.00%  0.00%  0.00%   0 Chunk Manager
   2        5220        1525       3422  0.07%  0.04%  0.05%   0 Load Meter
   3       83540        5068      16483  3.73%  0.78%  0.58%   0 Exec
   4           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers
   5       11812        1307       9037  0.00%  0.15%  0.12%   0 Check heaps
   6       57772        4431      13038  2.30%  1.02%  0.99%   0 Pool Manager
   7         212          67       3164  0.00%  0.00%  0.00%   0 DiscardQ Backgro
   8           0           2          0  0.00%  0.00%  0.00%   0 Timers
   9         344         397        866  0.00%  0.00%  0.00%   0 WATCH_AFS
  10       46644        4333      10764  0.07%  0.21%  0.18%   0 ARP Input
  11        4880        7878        619  0.07%  0.00%  0.00%   0 ARP Background
 --More--

Here are examples of connections initiated from the ‘Attacker’ seen on the ISP’s router:
 
ISP#sh logg

...

*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(9599) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(2627) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(8561) -> 10.0.0.10(80), 1 packet
*Aug 31 00:20:46.451: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(1329) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:17.215: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(18443) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:18.227: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(13679) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:19.247: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(9995) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:20.275: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(5977) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:21.311: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(3745) -> 10.0.0.10(80), 1 packet
*Aug 31 00:21:22.343: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 192.168.157.200(15022) -> 10.0.0.10(80), 1 packet

From the Perimeter Firewall’s point of view, the attack was not so heavy. The CPU usage was around
1-2%:
 
PER-FW# sh access-list
access-list cached ACL log flows: total 1529, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=879) 0x5f077c5e
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 2%; 5 minutes: 1%
PER-FW#

The reason for the low impact might be the number of packets which hit the PER-FW. Let’s check the counters on the ISP:

ISP#sh int gig0/0 stat
GigabitEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor      81789    4906579         17       1658
             Route cache          0          0          0          0
                   Total      81789    4906579         17       1658



ISP#sh int gig1/0 stat
GigabitEthernet1/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor          0          0       8962     538358
             Route cache        349      20242          0          0
                   Total        349      20242       8962     538358


As you can see, the number of packets sent out from the ISP router is 10 times less than it received. This could be caused by a lack of resources to process all packets. Here we have sample connections initiated by the ‘Attacker’ as seen on the PER-FW:
 
 
%ASA-6-302013: Built inbound TCP connection 12328 for outside:192.168.157.200/13402 (192.168.157.200/13402) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(23561) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12329 for outside:192.168.157.200/23561 (192.168.157.200/23561) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(29917) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12330 for outside:192.168.157.200/29917 (192.168.157.200/29917) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(17876) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12331 for outside:192.168.157.200/17876 (192.168.157.200/17876) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(14328) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12332 for outside:192.168.157.200/14328 (192.168.157.200/14328) to inside:10.0.0.10/80 (10.0.0.10/80)
%ASA-6-106100: access-list OUT permitted tcp outside/192.168.157.200(5033) -> inside/10.0.0.10(80) hit-cnt 1 first hit [0x5f077c5e, 0x0]
%ASA-6-302013: Built inbound TCP connection 12333 for outside:192.168.157.200/5033 (192.168.157.200/5033) to inside:10.0.0.10/80 (10.0.0.10/80)
<--- More --->


In this case we can easily locate the source of our problems. Let’s see how we can do this. The first problem you may notice is the ‘log’ parameter in the access-list. This is a very CPU intensive parameter and it shouldn’t be used in your production configuration:
 
 
conf t
ip access-list extended OUTSIDE
 no permit tcp any host 10.0.0.10 eq www log
 permit tcp any host 10.0.0.10 eq www 
end


Here I’ve initiatedan attack once again to check the results:
 
 
ISP#sh processes cpu sorted
CPU utilization for five seconds: 31%/100%; one minute: 61%; five minutes: 33%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 145       27216       12685       2145  1.03%  0.27%  0.10%   0 CEF: IPv4 proces
 239       38360      494717         77  0.71%  0.74%  0.44%   0 ISG MIB jobs Man
 112       20068      251110         79  0.31%  0.28%  0.20%   0 IPAM Manager
  82       15004      251100         59  0.31%  0.26%  0.19%   0 IP ARP Retry Age
  10       52028        5623       9252  0.31%  0.58%  0.27%   0 ARP Input
   2        6136        1793       3422  0.07%  0.06%  0.06%   0 Load Meter
  27        4340        8667        500  0.07%  0.02%  0.00%   0 IPC Periodic Tim
 174        9048       84571        106  0.07%  0.12%  0.10%   0 RBSCP Background
 270       66340        8950       7412  0.07%  0.67%  0.42%   0 Per-Second Jobs
   9         360         475        757  0.00%  0.00%  0.00%   0 WATCH_AFS
   7         220          76       2894  0.00%  0.00%  0.00%   0 DiscardQ Backgro
 --More--


CPU usage has dropped from 96% to 31%!
 
 
ISP#sh int gig0/0 stat
GigabitEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor        193      11307         30       2774
             Route cache     390697   23441820          0          0
                   Total     390890   23453127         30       2774

ISP#sh int gig1/0 stat
GigabitEthernet1/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor          0          0         20       2174
             Route cache        413      23954       6971     376434
                   Total        413      23954       6991     378608
ISP#


Let’s check the CPU usage of the PER-FW:
 
 
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 10%; 1 minute: 5%; 5 minutes: 2%
PER-FW# sh access-list
access-list cached ACL log flows: total 5011, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=5123) 0x5f077c5e
PER-FW#

In this case we have to isolate the traffic from the PRE-FW and the DMZ. The easiest way is to add an access-list (ACL) which blocks the traffic based on its source IP address:
 
!
interface GigabitEthernet0/0
 ip address 192.168.157.10 255.255.255.0
 ip access-group ATTACK-ACL in
 media-type gbic
 speed 1000
 duplex full
 negotiation auto
!

!
ip access-list extended ATTACK-ACL
 deny   tcp host 192.168.157.200 host 10.0.0.10 eq www
!

The negative effect of the ACL is higher CPU usage, but as you see not one packet hit the PER-FW:
 
ISP#sh processes cpu sorted
CPU utilization for five seconds: 89%/100%; one minute: 77%; five minutes: 35%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  83     1339100       99226      13495 37.87% 31.74% 11.97%   0 IP Input
   3      127132        9806      12964  1.12%  0.70%  1.22%   0 Exec
 145       38500       18525       2078  0.71%  0.18%  0.10%   0 CEF: IPv4 proces
 112       29524      366360         80  0.47%  0.19%  0.17%   0 IPAM Manager
 239       60840      722891         84  0.39%  0.37%  0.35%   0 ISG MIB jobs Man
   2        9348        2610       3581  0.31%  0.10%  0.06%   0 Load Meter
  44        3724       12985        286  0.23%  0.02%  0.00%   0 GraphIt
  59       19504        3887       5017  0.07%  0.09%  0.08%   0 HC Counter Timer
  98        8872       50501        175  0.07%  0.07%  0.02%   0 SSS Feature Time
  82       23952      366347         65  0.07%  0.13%  0.15%   0 IP ARP Retry Age
 270       88148       12990       6785  0.07%  0.49%  0.24%   0 Per-Second Jobs

ISP#
 
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 1%
PER-FW# sh access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=0) 0x5f077c5e
PER-FW#


The next option is to use Policy Based Routing (PBR) for traffic coming from the ‘Attacker’:
 
 
!
access-list 1 permit 192.168.157.200
!
route-map ATTACK permit 10
 match ip address 1
 set interface Null0
!
interface gig0/0
 ip policy route-map ATTACK
!

ISP#sh route-map ATTACK
route-map ATTACK, permit, sequence 10
  Match clauses:
    ip address (access-lists): 1
  Set clauses:
    interface Null0
  Policy routing matches: 598255 packets, 35895300 bytes
ISP#

It should be processed by the hardware, so it will not impact the CPU on the ISP’s router.

Lab #2 – DDOS attack 

Due to technical limitations I will only test a scenario in which an attack comes from one physical machine, but the packets contain spoofed source IP addresses. I removed all security features applied in lab #1, so now only http traffic is allowed from any source to the destination 10.0.0.10. I started this attack using random IP sources :
 
root@kali:~# t50 10.0.0.10 --flood --syn --dport 80
entering in flood mode...
hit CTRL+C to break.
T50 5.4.1-rc1 successfully launched on Aug 30th 2014 22:41:39

As expected CPU usage on the ISP’s router is very high:
 
ISP#sh processes cpu sorted
CPU utilization for five seconds: 94%/100%; one minute: 31%; five minutes: 14%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  83     1474212      106143      13888 26.77%  9.08%  3.87%   0 IP Input
 270       93876       16754       5603  2.39%  0.49%  0.20%   0 Per-Second Jobs
   6       91056        7058      12901  1.83%  0.96%  0.42%   0 Pool Manager
   3      145300       11877      12233  0.95%  0.51%  0.53%   0 Exec
 174       15604      160192         97  0.39%  0.09%  0.08%   0 RBSCP Background
 112       33524      479984         69  0.39%  0.18%  0.16%   0 IPAM Manager
 239       70372      949141         74  0.39%  0.36%  0.32%   0 ISG MIB jobs Man
  50       27368        6837       4002  0.39%  0.10%  0.06%   0 Net Background
  82       27472      479970         57  0.15%  0.13%  0.13%   0 IP ARP Retry Age
  11        8368       17366        481  0.15%  0.02%  0.00%   0 ARP Background
   5       24248        2830       8568  0.07%  0.12%  0.12%   0 Check heaps

The number of hits is very similar to test #1:
 
ISP#sh access-lists OUTSIDE
Extended IP access list OUTSIDE
    10 permit tcp any host 10.0.0.10 eq www log (72422 matches)
ISP#
ISP#sh int gig0/0 stats
GigabitEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor      72550    4357936         36       4074
             Route cache          0          0          0          0
                   Total      72550    4357936         36       4074
ISP#sh int gig1/0 stats
GigabitEthernet1/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor          0          0       8852     532396
             Route cache          0          0          0          0
                   Total          0          0       8852     532396
ISP#

As you see in the following log, the connections came from different source IPs:
 
ISP#sh logging
Syslog logging: enabled (0 messages dropped, 1549 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)


*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 220.199.63.3(155) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 164.55.130.104(28412) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 83.17.127.33(2912) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 80.202.164.15(7333) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 35.163.122.116(14173) -> 10.0.0.10(80), 1 packet
*Aug 31 02:52:06.902: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 160.76.89.101(23452) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:33.038: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 150.234.98.10(30593) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:34.082: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 50.7.245.9(8186) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:35.098: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 32.64.62.34(8789) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:36.114: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 197.138.244.20(31482) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:37.126: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 77.36.118.8(24865) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:38.174: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 103.95.142.100(11182) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:39.194: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 64.230.212.54(24549) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:40.218: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 210.25.64.40(23580) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:41.218: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 130.11.182.87(3643) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:42.230: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 181.182.64.60(14103) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:43.242: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 92.8.226.119(24074) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:44.250: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 34.72.28.48(22367) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:44.566: %SEC-6-IPACCESSLOGRL: access-list logging rate-limited or missed 13666 packets
*Aug 31 02:54:45.254: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 25.9.251.30(23508) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:46.258: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 201.27.106.117(26452) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:47.278: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 190.52.74.50(6904) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:48.286: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 169.147.17.89(12970) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:49.294: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 196.174.184.49(24059) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:50.326: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 118.237.206.110(1664) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:51.326: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 132.72.239.123(22164) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:52.358: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 26.238.113.114(12075) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:53.366: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 200.78.129.114(27771) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:54.374: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 44.69.75.92(25733) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:55.402: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 185.39.35.54(9501) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:56.406: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 149.238.124.68(689) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:57.458: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 215.40.122.24(14267) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:58.534: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 141.225.192.103(24388) -> 10.0.0.10(80), 1 packet
*Aug 31 02:54:59.542: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 65.67.18.17(7899) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:00.542: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 57.135.236.19(3134) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:01.610: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 240.55.235.108(23735) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:02.726: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 52.202.105.109(23049) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:03.738: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 226.5.29.67(12367) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:04.766: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 1.160.121.91(10350) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:05.786: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 254.144.152.15(28777) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:06.802: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 90.84.122.90(31084) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:07.806: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 67.119.134.11(24975) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:08.830: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 48.195.167.60(31799) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:09.834: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 117.232.51.94(29094) -> 10.0.0.10(80), 1 packet
*Aug 31 02:55:10.854: %SEC-6-IPACCESSLOGP: list OUTSIDE permitted tcp 83.219.21.118(24424) -> 10.0.0.10(80), 1 packet
 --More

The CPU on the PER-FW is not very high:
 
PER-FW# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 1%

PER-FW# sh access-list
access-list cached ACL log flows: total 732, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list OUT; 1 elements; name hash: 0xcd7d0798
access-list OUT line 1 extended permit tcp any host 10.0.0.10 eq www log informational interval 300 (hitcnt=732) 0x5f077c5e
PER-FW#


PER-FW# sh int gig0 stats
Interface GigabitEthernet0 "outside", is up, line protocol is up
  Hardware is Linux Ethernet Dev, BW 100 Mbps, DLY 100 usec
        (Full-duplex), (100 Mbps)
        Input flow control is unsupported, output flow control is unsupported
        MAC address 00ab.cd92.5200, MTU 1500
        IP address 192.168.158.11, subnet mask 255.255.255.0
        7814 packets input, 443038 bytes, 0 no buffer
        Received 0 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        8 packets output, 534 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (0/0)
        output queue (blocks free curr/low): hardware (0/0)
  Traffic Statistics for "outside":
        918 packets input, 38467 bytes
        0 packets output, 0 bytes
        92 packets dropped
      1 minute input rate 0 pkts/sec,  16 bytes/sec
      1 minute output rate 0 pkts/sec,  0 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 2 pkts/sec,  106 bytes/sec
      5 minute output rate 0 pkts/sec,  0 bytes/sec
      5 minute drop rate, 0 pkts/sec
PER-FW# sh int gig1 stats
Interface GigabitEthernet1 "inside", is up, line protocol is up
  Hardware is Linux Ethernet Dev, BW 100 Mbps, DLY 100 usec
        (Full-duplex), (100 Mbps)
        Input flow control is unsupported, output flow control is unsupported
        MAC address 00ab.cd92.5201, MTU 1500
        IP address 10.0.0.1, subnet mask 255.255.255.0
        1329 packets input, 88071 bytes, 0 no buffer
        Received 0 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        0 L2 decode drops
        7139 packets output, 385491 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (0/0)
        output queue (blocks free curr/low): hardware (0/0)
  Traffic Statistics for "inside":
        393 packets input, 19227 bytes
        830 packets output, 33200 bytes
        393 packets dropped
      1 minute input rate 0 pkts/sec,  11 bytes/sec
      1 minute output rate 0 pkts/sec,  0 bytes/sec
      1 minute drop rate, 0 pkts/sec
      5 minute input rate 1 pkts/sec,  46 bytes/sec
      5 minute output rate 2 pkts/sec,  97 bytes/sec
      5 minute drop rate, 1 pkts/sec
PER-FW#

As you can see below, the half-open connections are kept only for 30 seconds:
 
PER-FW# sh conn
388 in use, 2542 most used
TCP outside 132.16.242.95:4437 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 45.180.196.105:7374 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB

TCP outside 139.45.38.60:29808 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 251.45.52.118:9456 inside 10.0.0.10:80, idle 0:00:00, bytes 0, flags SaAB
TCP outside 120.180.46.9:32555 inside 10.0.0.10:80, idle 0:00:02, bytes 0, flags SaAB
TCP outside 190.97.134.67:15763 inside 10.0.0.10:80, idle 0:00:02, bytes 0, flags SaAB

TCP outside 93.216.144.56:12253 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 79.73.193.9:28466 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 115.141.225.36:2679 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 202.235.0.52:18265 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB
TCP outside 46.100.213.111:3357 inside 10.0.0.10:80, idle 0:00:03, bytes 0, flags SaAB

TCP outside 52.54.191.123:30871 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 171.140.183.72:13459 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 146.14.23.38:10416 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 29.105.29.26:25413 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
TCP outside 134.132.159.121:1246 inside 10.0.0.10:80, idle 0:00:13, bytes 0, flags SaAB
….
TCP outside 207.26.176.50:18211 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB
TCP outside 46.13.236.42:20774 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB
TCP outside 184.58.42.97:28843 inside 10.0.0.10:80, idle 0:00:20, bytes 0, flags SaAB

TCP outside 16.185.54.105:12733 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 157.244.239.122:11022 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 130.80.98.111:2086 inside 10.0.0.10:80, idle 0:00:27, bytes 0, flags SaAB
TCP outside 102.155.65.57:9322 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 79.180.252.112:20785 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 112.112.154.108:18489 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 191.31.218.24:2074 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 213.125.40.95:19220 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
TCP outside 65.75.170.103:8192 inside 10.0.0.10:80, idle 0:00:29, bytes 0, flags SaAB
PER-FW#

After 30 seconds they are removed:
 
PER-FW# sh logg
Syslog logging: enabled
    Facility: 20
    Timestamp logging: disabled
    Standby logging: disabled
    Debug-trace logging: disabled
    Console logging: disabled
    Monitor logging: disabled
    Buffer logging: level informational, 22071 messages logged
    Trap logging: disabled
    Permit-hostdown logging: disabled
    History logging: disabled
    Device ID: disabled
    Mail logging: disabled
    ASDM logging: disabled
nside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3419 for outside:61.84.231.56/5802 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout

%ASA-6-302014: Teardown TCP connection 3503 for outside:92.159.9.91/21461 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3504 for outside:175.111.170.38/10465 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3505 for outside:58.116.52.96/8622 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN 

%ASA-6-302014: Teardown TCP connection 3522 for outside:95.189.67.74/17573 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3523 for outside:150.0.190.70/2709 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3524 for outside:211.185.70.62/32324 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN Timeout
%ASA-6-302014: Teardown TCP connection 3525 for outside:204.68.234.108/26687 to inside:10.0.0.10/80 duration 0:00:30 bytes 0 SYN 

Below you can find the CPU usage of the WEB-SRV:
 
WEB-SRV#sh processes cpu sorted
CPU utilization for five seconds: 12%/2%; one minute: 4%; five minutes: 2%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  83      227296       34712       6548  4.79%  1.78%  0.81%   0 IP Input
   3       11328        1288       8795  3.43%  0.48%  0.36%   0 Exec
 239       21852      907335         24  0.47%  0.32%  0.29%   0 ISG MIB jobs Man
 150       20708       11147       1857  0.31%  0.18%  0.05%   0 TCP Timer
 270        7356       14662        501  0.15%  0.03%  0.00%   0 Per-Second Jobs
  82        8544      453167         18  0.15%  0.15%  0.13%   0 IP ARP Retry Age
 112       10512      453168         23  0.15%  0.16%  0.15%   0 IPAM Manager
  59       11256        4395       2561  0.07%  0.08%  0.07%   0 HC Counter Timer
 173         184        1463        125  0.07%  0.00%  0.00%   0 QoS stats proces
 174        3136      145816         21  0.07%  0.07%  0.07%   0 RBSCP Background
 268       11416        2934       3890  0.07%  0.08%  0.07%   0 Compute load avg

WEB-SRV#

WEB-SRV#sh int gig0/0
GigabitEthernet0/0 is up, line protocol is up
  Hardware is i82543 (Livengood), address is ca02.10e8.0008 (bia ca02.10e8.0008)
  Internet address is 10.0.0.10/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1000Mbps, link type is auto, media type is SX
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:08:13, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:11:26
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     830 packets input, 44820 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     435 packets output, 30199 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

WEB-SRV#sh int gig0/0 stats
GigabitEthernet0/0
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor        830      44820        436      30259
             Route cache          0          0          0          0
                   Total        830      44820        436      30259
WEB-SRV#

Let’s think a while about how we can mitigate the attack or how we can minimize its impact on our network. As you can see, we can’t apply an access list with the source of the attacker’s IP address because in this case every connection comes from a different IP address. We can’t use policy base routing (PBR), because we can’t identify the ‘bad’ traffic. We can deny all connections to our web server, but it is not a good solution the server needs to be available to our clients. This is a list of some possible solutions:
  • Limit the number of embryonic connections – you have to know your ‘normal’ traffic: how many connection you see on average and what is the maximum number of these connections. Based on this information you can apply connection limits. To do this, you can use one of the following features:
a) Context-Based Access Control (CBAC):
 
!CBAC TCP/UDP Intercept Feature

R3:
ip inspect max-incomplete low 500
ip inspect max-incomplete high 700
ip inspect one-minute low 90
ip inspect one-minute high 100
ip inspect tcp synwait-time 15
ip inspect tcp max-incomplete host 50 block-time 5
!
ip inspect name PROTECTDOS tcp
ip inspect name PROTECTDOS udp
ip inspect name PROTECTDOS icmp
!
ip access-list extended ATTACK
deny ip any any log
!
interface FastEthernet 0/0
ip inspect PROTECTDOS out
ip access-group ATTACK in

b) Zone Base Policy Firewall (ZBPF)
 
parameter-map type inspect ATTACK_CONTROL
 audit-trail on
 alert off
 max-incomplete low 200
 max-incomplete high 300
 udp idle-time 50
 dns-timeout 5
 tcp idle-time 60
 tcp synwait-time 5

zone security outside
zone security dmz


interface GigabitEthernet0/1
 zone-member security dmz
!
interface GigabitEthernet0/0
 zone-member security outside


class-map type inspect match-any DMZ-protocols
 match protocol http

policy-map type inspect outside_to_dmz

 class type inspect DMZ-protocols
  inspect ATTACK_CONTROL

 class class-default
  drop

zone-pair security OUT>DMZ source outside destination dmz
 service-policy type inspect outside_to_dmz

c) ASA:
 
class-map ATTACK
match any

policy-map ATTACK
class ATTACK
set connection conn-max 500 embryonic-conn-max 100
set connection timeout idle 2:0:0 embryonic 0:40:0 half-closed 0:20:0 dcd

service-policy CONNS interface outside
  • Keep the timeout of half open connections as low as possible, it will delete all not-completed TCP sessions faster.
  • Contact your ISP and ask them what kind of protections he can guarantee during a DDOS attack. Every ISP offers different services but for some of them you may have to pay more. This information should be included in your “Business Recovery Plan”. Don’t wait to do this until you are under attack, as it will be too late.
  • If you need your WEB Server available for your customers, you should prepare a backup plan. It can be a backup server located in a Data Center, for example Amazon (Elastic Compute Cloud EC2). You pay only for what you use, and if you need the backup service only occasionally, it should not ruin your budget.
  • If your server architecture is very complex, or due to regulations you can’t keep your data out of your company/country, you can consider using companies like Prolexis (now Akamai), but it is rather for the enterprise segment. They have many Data Centers around the world and your content can be easily moved around. They also work closely with ISPs to track down attackers.
  • The next option for the medium to large segment is to use the following appliances that can mitigate DOS/DDOS attacks: DDOS Protector (Checkpoint), BIG-IP ASM and/or BIG-IP LTM (F5). You can also buy security services from AT&T or IBM, but as I said earlier, this is rather for the enterprise segment.
The number of DOS/DDOS attacks will grow as it is the easiest way to stop someone’s business. The motivations may be different, but the effect is the same: your service is not available. You should prepare your business for these kinds of attacks. Maybe you will never use this plan, but it’s definitely better to be prepared. You can’t think ‘I have a firewall, so I’m safe!’ It isn’t true, even Cisco openly says: “Although firewalls play a critical role in any organization’s security solution, they are not purpose-built DDoS prevention devices.”

No comments:

Post a Comment