More improvements and bugfixes for the shield

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

More improvements and bugfixes for the shield

user8446
Administrator
Here are some more improvements and bugfixes:

Local Startup (rc.local) :

..snip..
ethtool -K eth0 gro off
ethtool -K eth0 gso off
ethtool -K eth0 tx-nocache-copy off
ifconfig eth0 up
sleep 1
ethtool -s eth0 autoneg off
sleep 1
ethtool -s eth0 autoneg on
sleep 1
ethtool -K eth1 gro off
ethtool -K eth1 gso off
ethtool -K eth1 tx-nocache-copy off
ifconfig eth1 up
sleep 1
ethtool -s eth1 autoneg off
sleep 1
ethtool -s eth1 autoneg on
sleep 1
ethtool -K eth2 gro off
ethtool -K eth2 gso off
ethtool -K eth2 tx-nocache-copy off
ifconfig eth2 up
..snip..

Here we are replacing "lro off" for the 3 interfaces with "tx-nocache-copy off". LRO is fixed off already on the interfaces so it can't be changed. "tx-nocache-copy off" is default on, and people have had it mangle packets if on and it uses less processor clock cycles when off.

Snort config:

Replace your line in the config with this line:
config detection: search-method ac-nq split-any-any search-optimize max-pattern-len 18 no_stream_inserts

Here we are truncating rules to no longer than 18 vs. the 20 in the fast pattern matcher. This saves over ~3mb in the cache and ~1000 additional rules truncated over the 20 character limit.

Here's the difference:

18 bytes:
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: [ Port Based Pattern Matching Memory ]
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: +- [ Aho-Corasick Summary ] -------------------------------------
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Storage Format    : Full
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Finite Automaton  : DFA
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Alphabet Size     : 256 Chars
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Sizeof State      : Variable (1,2,4 bytes)
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Instances         : 121
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     1 byte states : 106
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     2 byte states : 15
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     4 byte states : 0
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Characters        : 133829
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | States            : 89542
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Transitions       : 6457880
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | State Density     : 28.2%
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Patterns          : 10177
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Match States      : 10326
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: | Memory (MB)       : 48.27
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |   Patterns        : 1.03
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |   Match Lists     : 2.97
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |   DFA
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     1 byte states : 0.70
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     2 byte states : 43.36
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: |     4 byte states : 0.00
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: +----------------------------------------------------------------
Thu Oct 27 23:49:45 2016 daemon.notice snort[25052]: [ Number of patterns truncated to 18 bytes: 3049 ]

20 bytes:
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: [ Port Based Pattern Matching Memory ]
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: +- [ Aho-Corasick Summary ] -------------------------------------
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Storage Format    : Full
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Finite Automaton  : DFA
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Alphabet Size     : 256 Chars
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Sizeof State      : Variable (1,2,4 bytes)
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Instances         : 121
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     1 byte states : 106
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     2 byte states : 15
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     4 byte states : 0
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Characters        : 139625
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | States            : 94837
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Transitions       : 6823476
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | State Density     : 28.1%
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Patterns          : 10177
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Match States      : 10497
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: | Memory (MB)       : 51.02
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |   Patterns        : 1.04
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |   Match Lists     : 3.09
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |   DFA
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     1 byte states : 0.73
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     2 byte states : 45.96
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: |     4 byte states : 0.00
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: +----------------------------------------------------------------
Thu Oct 27 23:38:22 2016 daemon.notice snort[24339]: [ Number of patterns truncated to 20 bytes: 2071 ]

Next, disable ALL preprocessors except for SSL. In /usr/lib/snort_dynamicpreprocessor just delete all except for the three libsf_ssl_preproc*. You can always add back later if you want. They're in the itusshield repo on github. Preprocessors have nothing to do with the normal signature rules. However, you want to leave the SSL one on so snort will ignore the encrypted packets, increasing performance.

Now comment out all of those preprocessors in the snort config. SMTP, SIP, FTP, etc. Here's mine I'm running now as an example: snort_bridge.conf
You may have adjusted your stream5 memcap, prune_log_max, max_queued_segs, and max_queued_bytes to match your pipe so it may be different than mine.

Optional bugfix if needed:
With some modems, if it resets or it's power cycled on it's own your intenet goes down and you loose connection, even after snort restarts. If you have this issue go to /etc/config/network and update the following:

config interface 'wan'
        option proto 'none'
        option enable '1'
        option _orig_ifname 'eth0'
        option _orig_bridge 'true'
        option ifname 'eth0 eth2'
        option auto '1'

config interface 'wan6'
        option proto 'none'
        option enable '1'
        option _orig_ifname 'eth2'
        option _orig_bridge 'true'
        option ifname 'eth2'
        option auto '1'

Your modem can now get the DHCP lease out to your router properly because eth0 stays up.


I have noticed that the load on the processor is lower with these fixes which should mean faster throughput. Also, this may be a coincidence but I have had no resets in weeks. My pipe is only 50/5 so hopefully you guys on faster links can test.
And as always, you always want to eliminate rules that are not needed for your network. The lower the rule count the better the performance.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
This post was updated on .
Hi, this is great, thanks.

Does this apply regardless of the mode you are running? (i.e. Router or Bridge?)

I'm running Bridge mode, I've applied the hotfixes to v1.51 SP1 + Hotfix Mar 9, and done all the config changes per your post above (except the very last one, doesn't seem to be a problem for me).  I can confirm the SNORT rules are firing and logging hits, internet speed test is 50-58Mbps down (out of 200Mbit pipe which is pretty much the same as before.  However, for me it's the reliability of the connection which is important, my wife works from home and it normally drops a couple of times a day.  I'll monitor it using a bash script for the next 48 hours, see if it drops.

Thanks again for your continued support of the Shield.
Running v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
So far so good. Zero internet drops, even after the ips update ran at 1am, impressive so far!
Running v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

user8446
Administrator
Yes these apply for both router & bridge except for the optional one (those on router remember there are 2 instances snort7 & snort 8)

Thanks for testing... I've been 3 weeks now without a reset so it definitely squashed some bugs.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Wisiwyg
Applied changes yesterday and things seemed to work. Today internet down.

Sorting through, something is deleting the /etc/snort/rules/snort.rules file. Also, my fw_upgrade file was erased. I replaced with a version on my pc.

Running /etc/fw_upgrade and watching the output indicates it runs well until the cleanup stage. Will have to look deeper when I have more time.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

breda
In reply to this post by Turrican
Hi, Turrican did you internet speed get better with the updated?  I have not updated yet but I sill only get  50 Mbps down in  Bridge mode and we have 105 Mbps line coming in  user8446 has been helping be but we still can't seem to get past 50 Mbps range

http://itus.accessinnov.com/Internet-speed-slower-in-bridge-mode-td1123.html

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Roadrunnere42
In reply to this post by Wisiwyg
Hi Wisiwyg

At the end of fw-upgrade script is a section that checks if these  words are in the tmp/ogfile.log
FATAL ERROR

If thesewords are present then it renames the snort rule file and  deletes the snort rule, this is what I think is happening
mv -f /etc/snort/rules/snort.rules /etc/snort/rules/snort.rules.old
rm -f /etc/snort/rules/snort.rules

The reason I put this into the script was if after downloading the new snort rules an error occurred, this would stop the Shield connecting to the Internet.  New snort rules are downloaded and checks again for errors, this is done 3 times if it still fails then human intervention is required to sort the problem.

Have a look in the tmp/logfile.log for the words FATAL ERROR, I'm thinking that you may have had another error which the fw_upgrade script had seen and thus run the cleanup process. If this is the case then the script will have to be changed somehow to only trigger when it's the snort rules that are causing the fatal error

As for the fw_upgrade script being delete i have no idea.

Hope this helps

Roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
In reply to this post by breda
Hi Breda

As I mentioned above, I am getting around 50Mbps down.  My pipe is 200Mpbs which I acheive (exceed actually, sometimes I get 220Mbps) without the Shield in place.

With regards to Internet drops (my main gripe) I checked my log output from my connectivity test script and the results were interesting. The ONLY internet drops since I started monitoring it (on the 2nd) were on 2nd and 3rd Nov between 14:24:30 and 14:25:51 (my script tests every 30 seconds) - both days, same time-frame.


Wed 2 Nov 14:24:31 GMT 2016 Internet UP
Wed 2 Nov 14:25:21 GMT 2016 Internet DOWN
Wed 2 Nov 14:25:51 GMT 2016 Internet UP

Thu 3 Nov 14:24:30 GMT 2016 Internet UP
Thu 3 Nov 14:25:20 GMT 2016 Internet DOWN
Thu 3 Nov 14:25:51 GMT 2016 Internet UP


I'm attaching my system log if anyone can find any clues?

Thanks

Sys_Log.txt   
Running v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

user8446
Administrator
Right off the bat I see you're hitting your memcap and max_queued_bytes. Each network is different depending on what's going across your wire (streaming, large files, etc.). I would bring then up some. Try this in your snort config:

..snip..

preprocessor stream5_global: track_tcp yes, \
   track_udp yes, \
   track_icmp yes, \
   max_tcp 10000, \
   max_udp 10000, \
   memcap 10388608, \
   max_active_responses 2, \
   min_response_seconds 5, \
   prune_log_max 3579067

preprocessor stream5_tcp: policy windows, detect_anomalies, require_3whs 180, \
   overlap_limit 10, small_segments 3 bytes 150, timeout 180, \
   max_queued_bytes 3550531, \
   max_queued_segs 3621, \

..snip..
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Wisiwyg
In reply to this post by Roadrunnere42
samtec55@gmail.com wrote
Hi Wisiwyg

At the end of fw-upgrade script is a section that checks if these  words are in the tmp/ogfile.log
FATAL ERROR

If thesewords are present then it renames the snort rule file and  deletes the snort rule, this is what I think is happening
mv -f /etc/snort/rules/snort.rules /etc/snort/rules/snort.rules.old
rm -f /etc/snort/rules/snort.rules

The reason I put this into the script was if after downloading the new snort rules an error occurred, this would stop the Shield connecting to the Internet.  New snort rules are downloaded and checks again for errors, this is done 3 times if it still fails then human intervention is required to sort the problem.

Have a look in the tmp/logfile.log for the words FATAL ERROR, I'm thinking that you may have had another error which the fw_upgrade script had seen and thus run the cleanup process. If this is the case then the script will have to be changed somehow to only trigger when it's the snort rules that are causing the fatal error

As for the fw_upgrade script being delete i have no idea.

Hope this helps

Roadrunnere42
Thank you for the info Roadrunnere42! It does help!

Yes, found the rm at the end and realized it was whacking the snort.rules. I've just copied the snort.rules.old to snort.rules and everything is working until fw_upgrade runs again. So I've renamed fw_upgrade to prevent it from running until I want it to.

Unfortunately, there's no logfile.log in /tmp. Offhand, do you know if it gets deleted as a final cleanup? I'll look through the script tomorrow to see if I can locate something that deletes it. I'll try to catch it before deletion to see what is happening. There was a snort rule that had "fatal error" not "FATAL ERROR" as part of the rule description. I commented that ruleset out and ran again without receiving the "fatal error" line, but it didn't change the outcome.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Roadrunnere42
I think i know whats going on, just done a factory reset and copied all updates and fixes all well

Copied over the new snort  changes as mentioned above rerun fw_upgrade and bang the snort rules get deleted, checked in the system log via gui and see the following

FATAL ERROR: /etc/snort/snort_bridge.conf(119) => Invalid Stream TCP policy option

so when fw_upgrade runs it See's the FATAL ERROR and so deletes the snort rules, but in facts it another problem.

I think I have  messed somehow the process mention above maybe deleted a TCP policy?

Roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Roadrunnere42
Just an upgrade copied over the snort_bridge.conf file from the post to my shield restarted, run fw_upgrade and all worked ok, so i had messed up  the snort_bridge.conf somehow.

great work user8446 will see if it drops the Internet connection

as a side note and i know snort is a beast but what are
preprocessor.rules
decoder.rules
sensitive-data.rules

tried looking but hard to fine a simple answer.

Roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
In reply to this post by user8446
User8446

Thanks for the suggestion, for 3 days in a row, internet dropped at 14:25. I applied the changes to my short config last night and today it didn't go down at all, no drops.  Speed is still around 50mbps, but at least it's stable.  Great stuff. Thanks
Running v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Wisiwyg
In reply to this post by Roadrunnere42
samtec55@gmail.com wrote
I think i know whats going on, just done a factory reset and copied all updates and fixes all well

Copied over the new snort  changes as mentioned above rerun fw_upgrade and bang the snort rules get deleted, checked in the system log via gui and see the following

FATAL ERROR: /etc/snort/snort_bridge.conf(119) => Invalid Stream TCP policy option

so when fw_upgrade runs it See's the FATAL ERROR and so deletes the snort rules, but in facts it another problem.

I think I have  messed somehow the process mention above maybe deleted a TCP policy?

Roadrunnere42
By jove! I think you've got it!

Seriously, thank you for your sleuthing work. Oddly enough, everythings working again. Snort.rules gets created and stays there. I ran for a day with fw_upgrade renamed so it couldn't execute. Then renamed it when I implemented your previous advice and it has now worked for an overnight update. Seems to be fixed, but I could'nt point to any one thing that fixed it.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Wisiwyg
In reply to this post by Roadrunnere42
samtec55@gmail.com wrote
Just an upgrade copied over the snort_bridge.conf file from the post to my shield restarted, run fw_upgrade and all worked ok, so i had messed up  the snort_bridge.conf somehow.

great work user8446 will see if it drops the Internet connection

as a side note and i know snort is a beast but what are
preprocessor.rules
decoder.rules
sensitive-data.rules

tried looking but hard to fine a simple answer.

Roadrunnere42
I'm no expert, but I believe the sensitive-data.rules are rules that look for things like social security numbers, bank account numbers, credit card numbers, etc. The stuff you don't want to be posting out in the clear or that you don't want other apps sending across in the clear.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
This post was updated on .
spoke too soon.  Lost internet today again at the same time!

Mon 7 Nov 14:24:28 GMT 2016 Internet UP
Mon 7 Nov 14:25:18 GMT 2016 Internet DOWN
Mon 7 Nov 14:25:49 GMT 2016 Internet UP

Seems like it was fine over the weekend.  I've checked my Virgin Media router (in Bridge, Modem-Only mode) and there is no connectivity loss.  Also checked my router which is plugged into my Shield, nothing odd in the logs and internet connetion shows its been up since 5th Nov (last time I rebooted it).

I guess the Ultimate test will be to remove the Sheild from the circuit and see if it happens tomorrow but I'm fairly sure at this point its the shield, question is why that time?  There's nothing scheduled at that time in CRON and system time is accurate:

# daily at 03:01 - Clear Logs
1 3 * * * > /tmp/snort/alert.fast
# daily at 02:00 - upgrade IPS rules
0 1 * * * sh /sbin/fw_upgrade
# daily at 00:00 - sync clock
0 0 * * * /usr/sbin/ntpclient -s -p 123 -h 0.us.pool.ntp.org || /etc/init.d/ntpclient restart

Any ideas gratefully received

Log attached, again :)

Shield_Sys_Log.txt

This is my script I'm using to test the Internet is up/down

#!/bin/bash

while true
do
    wget -q --tries=10 --timeout=20 -O - http://google.com > /dev/null
    if [[ $? -eq 0 ]]; then
        echo $(date) "Internet UP" | tee -a log.csv
    else
        echo $(date) "Internet DOWN" | tee -a log.csv
    fi
    sleep 30
done
Running v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Roadrunnere42
In reply to this post by user8446
Hi
I'm running in router mode and my original modified line was as suggest sometime ago for snort7 and snort8 files

config detection: search-method ac-split search-optimize max-pattern-len 20 no_stream_inserts

Now you say modify it to

config detection: search-method ac-nq split-any-any search-optimize max-pattern-len 18 no_stream_inserts

is the setting ac-nq correct for running in router mode or should it be

config detection: search-method ac split-any-any search-optimize max-pattern-len 18 no_stream_inserts

thanks

Roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

user8446
Administrator
Yes, you would want the nq (no queue) for both snort7 & 8:

config detection: search-method ac-nq split-any-any search-optimize max-pattern-len 18 no_stream_inserts

Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

user8446
Administrator
In reply to this post by Turrican
Your logs show it was Snort restarting. Odd how it's at the same time. Is there anything on your network that repeats daily at that time?
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: More improvements and bugfixes for the shield

Turrican
Nothing which repeats on my network.   Strange indeed, it's stayed up since my last post!
Running v2 Firmware
12