Bridge mode bugfix and performance improvement

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Bridge mode bugfix and performance improvement

user8446
Administrator
This post was updated on .
All,

I've been working on this for a few weeks now and wanted to release what I have. I have decreased latency (ping) on my network about 20%. Maybe someone with a fast connection can check to see if it increases throughput. Unfortunately, my connection is < 50mbps. Here is an updated /etc/snort/snort_bridge.conf or you can even copy/paste in the GUI:

Config file if you are NOT running the Trojan rules:
snort_bridge.conf

Config file if you ARE running the Trojan rules:
snort_bridge1.conf

---> PLEASE NOTE I HAD TO NAME THE SECOND FILE snort_bridge1.conf , it needs to be snort_bridge.conf . I can't upload two different files with the same name with this forum so change the name <---

You then want to create a folder/directory in /usr/lib/snort_dynamicpreprocessor and name it Disabled. Move these into Disabled:

libsf_smtp_preproc.so.0.0.0
libsf_smtp_preproc.so.0
libsf_smtp_preproc.so
libsf_sip_preproc.so.0.0.0
libsf_sip_preproc.so.0
libsf_sip_preproc.so
libsf_sdf_preproc.so.0.0.0
libsf_sdf_preproc.so.0
libsf_sdf_preproc.so
libsf_reputation_preproc.so.0.0.0
libsf_reputation_preproc.so.0
libsf_reputation_preproc.so
libsf_pop_preproc.so.0.0.0
libsf_pop_preproc.so.0
libsf_pop_preproc.so
libsf_modbus_preproc.so.0.0.0
libsf_modbus_preproc.so.0
libsf_modbus_preproc.so
libsf_imap_preproc.so.0.0.0
libsf_imap_preproc.so.0
libsf_imap_preproc.so
libsf_gtp_preproc.so.0.0.0
libsf_gtp_preproc.so.0
libsf_gtp_preproc.so
libsf_dnp3_preproc.so.0.0.0
libsf_dnp3_preproc.so.0
libsf_dnp3_preproc.so

(EDIT: There's a script below to move these files for you, thanks Hans!)

What all of this does is turn off preprocessors that are not used. Snort has to cycle packets through all of the preprocessors, so the more you have the longer it takes to process. Itus had preprocessors on that we're not even needed or used. I've also increased the stream5 queue and cache. When these are exceeded the stream has to be flushed out. This is one of the reasons people we're having their internet stop, the stream5 errors in their logs. You would think that you should just increase these to a large amount but the trick is to increase it to only what is needed otherwise you'll INCREASE latency. I used the largest that I saw on my network over the last few weeks. I have not had snort restart in weeks now. The performance optimization for the rules pattern matcher and the 64k log size rotation is also in there.
There will be more to come as testing and time allows as well as the router mode. Enjoy!
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
Good work user8446

Can these be user in router mode by  just copying into snort 7 and snort 8 ?

roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
The router config is slightly different. If you don't mind posting it I'll get to it quicker as I have limited time to take my unit offline to go into router mode. I'll make the changes and post it up.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
Here's the router files snort7 snort8
snort7.confsnort8.conf

roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
How many rules are you running? It matters on the pattern matcher.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
i am running at present 8942 rules but list could go up to 10000, how does this affect the pattern matcher?

roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
This post was updated on .
Because ac-full is actually the fastest pattern matcher but if you run a lot of rules snort will crash with an out of memory error since we only have 1gb of RAM. You can only use that with ~6500 rules or lower.

Here you go:

snort7.conf
snort8.conf

Don't forget to disable those preprocessors as shown above.

Everyone else: Roadrunnere42 changed his HOME_NET to match his network so change it to yours if you want to run this config. This is optimized for a large amount of  rules... remove "split-any-any" if you aren't.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Wisiwyg
Thank you for digging into this! It would proove very helpful!

I ran through the update last night. After reboot, Shield didn't come up. I took a quick look through system log and saw a couple of errors about dnsmasq not restarting. Before I could sort it out my family came home. Can't have internet down when they're home....

So I replaced everything the way it was and rebooted - came up as usual.

I was going to run a speedtest before and after to try to document throughput improvement. Without Shield 120, with Shield 98-100'ish, with improved _____?

I'll try to figure it out and fill in the last bit. During the evening, I was thinking through what might not have worked. I have trojan enabled on my Snort config. That might have done it if the trojan preprocessor was needed but not there. I'll disable trojan rules and retry the fix above when I can - when no one is wanting internet.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Hans
Administrator
I will try it soon too in Bridge mode

This should help in moving the files to Disabled

snort_preproc_move.sh

#!/bin/sh

if [[ ! -d "/usr/lib/snort_dynamicpreprocessor/Disabled" ]] ; 
	then 
	mkdir /usr/lib/snort_dynamicpreprocessor/Disabled 
fi

mv /usr/lib/snort_dynamicpreprocessor/libsf_smtp_preproc.so.0.0.0 /usr/lib/snort_dynamicpreprocessor/Disabled
mv /usr/lib/snort_dynamicpreprocessor/libsf_smtp_preproc.so.0 /usr/lib/snort_dynamicpreprocessor/Disabled
...
Using Shield Pro v1, Chaos Calmer, FW 1.51 SP1, Bridge Mode

2nd Shield as Sandbox, Chaos Calmer, FW 1.51 SP1 + hotfixes
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
Wisiwg -

If you are running the trojan rules replace this line in your cofig:
config detection: search-method ac-nq search-optimize max-pattern-len 20 no_stream_inserts

with this one:
config detection: search-method ac-nq split-any-any search-optimize max-pattern-len 20 no_stream_inserts

You're over the ~4500 rule limit for the ac full pattern matcher.

Reference: http://manual.snort.org/node16.html


Hans -

Thanks for throwing together that little script for us!


Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
Hi user8446

I followed your instruction running router mode and copied over the two modified snort rules 7 / 8, disabled processes as instructed reboot and it failed to connect to internet. Looked at sysrem log and found
Thu Mar 31 15:56:35 2016 daemon.notice netifd: Network device 'eth0' link is down
Thu Mar 31 15:56:35 2016 daemon.notice netifd: Interface 'wan' has link connectivity loss
Thu Mar 31 15:56:35 2016 daemon.notice netifd: wan (3340): Received SIGTERM
Thu Mar 31 15:56:36 2016 daemon.err snort[3323]: FATAL ERROR: /etc/snort/snort8.conf(120) Unknown rule type: prune_log_max.
Thu Mar 31 15:56:36 2016 user.emerg procd: Cannot change large-receive-offload
Thu Mar 31 15:56:38 2016 daemon.notice netifd: Network device 'eth0' link is up
Thu Mar 31 15:56:38 2016 daemon.notice netifd: Interface 'wan' has link connectivity
Thu Mar 31 15:56:38 2016 daemon.notice netifd: Interface 'wan' is setting up now
Thu Mar 31 15:56:38 2016 kern.notice kernel: [   75.354303] eth0: 1000 Mbps Full duplex, port 0
Thu Mar 31 15:56:38 2016 daemon.notice netifd: wan (3983): udhcpc (v1.23.2) started
Thu Mar 31 15:56:38 2016 daemon.notice netifd: wan (3983): Sending discover...
Thu Mar 31 15:56:39 2016 daemon.err snort[3391]: FATAL ERROR: /etc/snort/snort7.conf(120) Unknown rule type: prune_log_max.
Thu Mar 31 15:56:39 2016 user.emerg procd: Cannot change large-receive-offload
Thu Mar 31 15:56:41 2016 kern.notice kernel: [   78.304013] eth2: Link down


did find this in snort 7 about prune_log_max

preprocessor stream5_global: track_tcp yes, \
   track_udp yes, \
   track_icmp yes, \
   max_tcp 10000, \
   max_udp 10000, \
   max_active_responses 2, \
   min_response_seconds 5
   prune_log_max 1120810preprocessor stream5_global: track_tcp yes, \
   track_udp yes, \
   track_icmp yes, \
   max_tcp 10000, \
   max_udp 10000, \
   max_active_responses 2, \
   min_response_seconds 5
   prune_log_max 1120810
 any idea what to do to  or is then because i'm running 8942 rules?

reverted back at present.

Keep up the good work

roadrunnere42





Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
In reply to this post by Hans
Hans

Great script does make the job easier

roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
In reply to this post by Roadrunnere42
Found it! Small syntax error for snort7, snort8... the new config files are in the same post.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Roadrunnere42
update the snort files and all seems to work fine will continue to test

roadrunnere42
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Wisiwyg
In reply to this post by user8446
Already did the memory enhancement to enable trojan rules. It all was working pretty well before with trojan rules enabled. Just tried with the perf optimization.
user8446 wrote
If you are running the trojan rules replace this line in your cofig:
config detection: search-method ac-nq search-optimize max-pattern-len 20 no_stream_inserts
Also, the syntax change noted above - did that affect the bridge.conf setting? Could that have been part of the issue?
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
This new config and removing the preprocessors will still give you an improvement over just the memory optimization I posted a few weeks back. For several weeks now I have not had Snort reboot like it has in the past and I have reduced latency. To make it less confusing I posted both config files in the original post for with and without the trojan rules.
The syntax error was not with bridge, only with the router config files but now fixed.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Wisiwyg
This post was updated on .
That did it...
I used the revised snort.bridge1.conf and repeated the instructions.

Note: while in there I saw that I have both snort.conf and snort_bridge.conf. I renamed both the original files with a .orig extension, then duplicated the contents for snort_bridge.conf into a new snort.conf. The system came up quickly after reboot.

edit: I rebooted again and timed it - bootup and internet passthrough at 1:54 min! Very nice!
I repeated speedtest and see improvement: ping 10ms, DL 99 with peak @ 110mbps, UL 11.8 with peak 12.2mbps. Response time seems snappier as well, but not measured.

Thank you for putting this together.
Shield Pro v1, Chaos Calmer, FW 1.51 SP1, v8.3.2, Bridge Mode
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Gnomad
In reply to this post by user8446
Thanks guys, applied the snort7 & 8 conf files (after tweaking HOME_NET) and it seems to be running fine.  To clarify for others reading this thread, these particular files from user8446 contain the following line:
config detection: search-method ac-nq split-any-any search-optimize max-pattern-len 20 no_stream_inserts
This is an instruction to be used only if you have 6000+ snort rules, as described here.  You can count your snort rules by running:
wc -l /etc/snort/rules/snort.rules
  If you want to incorporate trojan rules into snort, uncomment the following line within /sbin/fw_upgrade
curl -k -1 -m 40 -o /mnt/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules
__________________________

user8446, I notice you've commented out the decoder rules.  I remember you recommended it in this thread in order to unblock Apple iTunes, but I found that wasn't needed after excluding sid 58.  Are we opening ourselves up to much risk by disabling all the decoder rules?
Router 1.51 SP1, fw_upgrade v8.3.6
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

user8446
Administrator
This post was updated on .
The decoder is for watching packets to try to detect anomalies such as packet size, header length, experimental stuff, etc. It has nothing to do with the normal detection rules. It would be useful if you had someone watching the logs and traffic full time like in an enterprise setting to try to identify a rogue endpoint. Itus even left it default off in bridge mode, I'm not sure why it was on in the router config. Given the networks the shield boxes are on, opt-in should be the default instead of opt-out.
Nice that you went one step farther and only turned off the offending SID.
Running in bridge mode, 1.51 SP1 fw
Reply | Threaded
Open this post in threaded view
|

Re: Bridge mode bugfix and performance improvement

Gnomad
Thanks for the tip.

For other readers, I've since spotted the snort FAQ on decoder and preprocessor rules here.  Note that although our decoder.rules are all configured as "alerts" by default - i.e., intended to log warnings only - some other setting (or a bug) must be upgrading them to drops.  Otherwise commenting them out/excluding them wouldn't have the affect on iTunes Store connectivity that it seems to.
Router 1.51 SP1, fw_upgrade v8.3.6
12