Internet speed slower in bridge mode

If it's the Shield thats under powered then when i run speed.net  download test and look at the shield cpu utilization window I would expect the cpu to be at full speed.

In my tests it does not go above 50%, I have seen in the pass when doing heavy network traffic it being above 200%

I can't see Itus building this box with an under powered cpu as it's main job was as an IDS system, I think Itus  got the shields out working and then were going to do the optimizing later, but as we know the film got screwed by the suppliers.

As some will remember it was released with firmware rc1 then sp1.51 which was a great improvement but still had rough edges, I'm sure if the firm was still running it would have improved it greatly, that said it's not going to help us now so lets try and find where the bottle neck is.

Roadrunnere42  

Hi Roadrunnere42, you're absolutely right.

I recall a discussion on how questions on using multiple cores to perform traffic filtering, and the discussion on 200% usage was 2 cores @ 100% each. So 50% usage is probably 1 core at half load due to overhead and the other core idle. Just speculation. I'll try this too later.

Thanks user8446 got the file back but now  getting an different errors and no internet


Tue Nov 15 18:15:21 2016 kern.info kernel: [  105.514369] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Tue Nov 15 18:15:21 2016 user.notice firewall: Reloading firewall due to ifup of wan6 (eth2)
Tue Nov 15 18:15:22 2016 daemon.crit dnsmasq[5126]: illegal repeated keyword at line 13 of /var/etc/dnsmasq.conf
Tue Nov 15 18:15:22 2016 daemon.crit dnsmasq[5126]: FAILED to start up
Tue Nov 15 18:15:48 2016 authpriv.info dropbear[3229]: Early exit: Terminated by signal
Tue Nov 15 18:15:48 2016 authpriv.info dropbear[5181]: Not backgrounding
Tue Nov 15 18:15:48 2016 authpriv.warn dropbear[5180]: Failed listening on '22': Error listening: Address already in use
Tue Nov 15 18:15:48 2016 authpriv.info dropbear[5180]: Early exit: No listening ports available.
Tue Nov 15 18:15:48 2016 authpriv.warn dropbear[5183]: Failed listening on '22': Error listening: Address already in use
Tue Nov 15 18:15:48 2016 authpriv.info dropbear[5183]: Early exit: No listening ports available.
Tue Nov 15 18:15:48 2016 authpriv.warn dropbear[5182]: Failed listening on '22': Error listening: Address already in use
Tue Nov 15 18:15:48 2016 authpriv.info dropbear[5182]: Early exit: No listening ports available.
Tue Nov 15 18:15:58 2016 daemon.emerg procd: recvfrom: Connection refused
Tue Nov 15 18:15:58 2016 user.notice root: Successful NTP clock adjust (0.us.pool.ntp.org).
Tue Nov 15 18:15:58 2016 daemon.info procd: - init complete -
Tue Nov 15 18:16:20 2016 kern.info kernel: [  164.626334] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.

Hi,  user8446 the internet is back up and getting  105 Mbps  with the shield   bridge mode but see few errors  and looking at the /etc/snort/rules the snort file looks empty  I ran the fw script via SSH





http://www.speedtest.net/my-result/5801621087

Tue Nov 15 18:53:20 2016 kern.info kernel: [ 2384.604389] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Tue Nov 15 18:53:20 2016 daemon.notice netifd: Interface 'wan6' is now up
Tue Nov 15 18:53:20 2016 user.notice firewall: Reloading firewall due to ifup of wan6 (eth2)
Tue Nov 15 18:53:20 2016 daemon.crit dnsmasq[7147]: illegal repeated keyword at line 13 of /var/etc/dnsmasq.conf
Tue Nov 15 18:53:20 2016 daemon.crit dnsmasq[7147]: FAILED to start up
Tue Nov 15 18:53:22 2016 daemon.notice netifd: Network device 'eth0' link is up
Tue Nov 15 18:53:22 2016 daemon.notice netifd: Interface 'wan' has link connectivity
Tue Nov 15 18:53:22 2016 daemon.notice netifd: Interface 'wan' is setting up now
Tue Nov 15 18:53:22 2016 daemon.notice netifd: Interface 'wan' is now up
Tue Nov 15 18:53:22 2016 kern.notice kernel: [ 2386.624517] eth0: 1000 Mbps Full duplex, port 0
Tue Nov 15 18:53:22 2016 kern.info kernel: [ 2386.624540] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Tue Nov 15 18:53:22 2016 user.notice firewall: Reloading firewall due to ifup of wan (eth0)
Tue Nov 15 18:53:22 2016 daemon.crit dnsmasq[7276]: illegal repeated keyword at line 13 of /var/etc/dnsmasq.conf
Tue Nov 15 18:53:22 2016 daemon.crit dnsmasq[7276]: FAILED to start up
Tue Nov 15 18:58:08 2016 authpriv.info dropbear[7337]: Child connection from 192.168.1.3:49967
Tue Nov 15 18:58:15 2016 authpriv.notice dropbear[7337]: Password auth succeeded for 'root' from 192.168.1.3:49967

Thats great your back up and running,and with105 Mbps in  bridge mode .

Try just adding one rule in fw_upgrade, rerun script then run speed.net to see your speed and while doing this looking a cpu utilization widow to see how much cpu is being used whist downloading. If it's a horse power issue with the shield  then the more rules being used the more cpu usage should go up in the cpu utilization window

Does any one know why in router mode there are 2 snort config files snort 7 and snort 8, and only one in bridge? .My theory could router mode be using both cpu core whist checking rules whist bridge mode only using one core. I can't test this theory as i only have max 50mb isp.

Roadrunnere42

Hi Roadrunnere42

I have a 200mpbs pipe at homre and a spare shield.  Happy to help if you can provide instructions :)

Hi, Roadrunnere42 I notice after the 1:00 AM snort update my speed drop down to 32 Mbps  can you  walk me thru how to Try just adding one rule in fw_upgrade,

Thanks for the help

Here is screen shot of CPU Utilization when doing speed test

looks like I was getting 105 Mbps  when the snort.rules file was empty

Thanks for the help

breda wrote

Here is screen shot of CPU Utilization when doing speed test

So @ 50% utilization it probably means only 1 core is engaged in Snort 'ing. Anyone know how to engage 2 cores? Or whether my assumption is incorrect and there are already 2 cores engaged?

breda

open the fw_upgrade.sh script which is in the folder /sbin and scroll down till you see the following lines
just add a # at the begin of each line that you don't want downloaded and updateed, then save and run fw_upgrade.sh  ( sh /sbin/fw_upgrade.sh )  try only selecting
a few and test speed just in case it's only one rule set thats slowing the Shield.

        curl -k -1 -m 40 -o /tmp/ramdisk/botcc.portgrouped.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.portgrouped.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/botcc.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/ciarmy.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-ciarmy.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/compromised.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-compromised.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/dshield.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-dshield.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-exploit.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-exploit.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-malware.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-mobile_malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-mobile_malware.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-user_agents.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-user_agents.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-web_client.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_client.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-worm.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-worm.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/emerging-current_events.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-current_events.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules
#      curl -k -1 -m 40 -o /tmp/ramdisk/drop.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-drop.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-web_specific_apps.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_specific_apps.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-scan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-scan.rules

# new rule site as suggested SSL Blacklist (SSLBL) is a project maintained by abuse.ch. The goal is to provide a list of "bad" SSL certificates identified
# by abuse.ch to be associated with malware or botnet activities. SSLBL relies on SHA1 fingerprints of malicious SSL certificates
# and offers various blacklists that can found in the SSL Blacklist section.
        curl -k -1 -m 40 -o /tmp/ramdisk/abuse-sslbl.rules https://sslbl.abuse.ch/blacklist/sslipblacklist.rules
       

Roadrunnere42

Thanks, Roadrunnere42 I will give it a try and report back to you so just to confirm  # at the begin of each line then run /fw_upgrade the via SSH?  Also should I upgrade to fw_upgrade 8.3.3 release? Do you have update file >? I think Wisiwyg said there was something missing?

http://itus.accessinnov.com/fw-upgrade-8-3-3-release-td1279.html

Hi, Roadrunnere42 found that if I put # curl -k -1 -m 40 -o /tmp/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules  my speeds go up to 96 Mbps but they seem to be combination  of files I trying to disable some of the rules to see if I can find them all one of them is definitely the  curl -k -1 -m 40 -o /tmp/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules is slowing my speed down


Thanks

Hi, Roadrunnere42 I can get 100 Mbps if I have this setting I'm using   VERSION NUMBER 1.51 - 8.3.2


        echo "Starting SNORT rule download..."
        curl -k -1 -m 40 -o /tmp/ramdisk/botcc.portgrouped.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.portgrouped.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/botcc.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/ciarmy.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-ciarmy.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/compromised.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-compromised.rules
        curl -k -1 -m 40 -o /tmp/ramdisk/dshield.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-dshield.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-exploit.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-exploit.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-malware.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-mobile_malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-mobile_malware.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-user_agents.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-user_agents.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-web_client.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_client.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-worm.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-worm.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-current_events.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-current_events.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/drop.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-drop.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-web_specific_apps.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_specific_apps.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-scan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-scan.rules

also before I disabled  some of the rules  I noticed that the flowing rules in the fw_upgrade  were  disabled but I did not turn them off  should they be off ?


# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-trojan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-trojan.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/drop.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-drop.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-web_specific_apps.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_specific_apps.rules
# curl -k -1 -m 40 -o /tmp/ramdisk/emerging-scan.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-scan.rules

100mps not bad so it's the rules slowing the shield down, we need to see if the Shield (snort) is using both cpu core when checking rules.

As for the rules that were commented out, these were as Itus had it set in the original sp1.51 release  here is what user8446 said months ago in the forum  about the rules.
Hope this help

Roadrunnere42


emerging-web_specific_apps.rules

these rules are only useful if you're hosting a web server and have ports on your firewall open. If not,
it would be a real waste of time for your Shield to be firing on all these rules that will never make it through your firewall anyway.

Removing this entire set of rules will allow your shield to download the rules faster, boot / reset snort faster, and just run faster in general.

I agree that this rule set is a good one to exclude out-of-the-box. If somebody is running a web server, they would also probably have enough knowledge to enable this specific rule set and configure HTTP_SERVERS / HTTP_PORTS in snort.conf. For the general Shield user, they can forget about it.

--

emerging-scan.rules

As the name implies, this rule set focuses on scanning for open ports and corresponding applications. Again, if you don't run anything that is exposed to the web your firewall will be (should be) completely closed down - nothing is exposed / nothing is open. So, although it's kind of nice to see that your shield is logging all sorts of port scanning that's happening, I agree that there is no real benefit in knowing this (beyond that your shield is working) and it could be a drain on the shield's resources.

Again, I agree that this rule set is a good one to exclude. However, if you like to see this stuff to know that your shield is working (port scanning is happening ALL the time so you'll see it in your logs), then include it and see if it slows down your Shield in any way.

--

drop.rules

I haven't looked closely, but this seems to be just another set of rules based on bad IPs - sort of like a blacklist. The important thing to understand is how a particular blacklist is created and maintained. without knowing this it's hard to say if the rules are useful or not.

--

emerging-trojan.rules

I noticed these rules were missing because of an IPS test I usually do against microsofi.org - a site set up specifically for testing your IPS. DNS and HTTP activity to microsofi.org triggers a trojan detection and is an easy way to check if your shield is running. By excluding this rule set the test obviously doesn't work.


It would be nice to hear from someone regarding why this rule set was commented out / isn't necessary.

That's good information, I did wonder why microsofi.org no longer generated an alert in the Snort Log.  I might try turning on some of these, particularly emerging-trojan.rules  see what that does to performance.

Thank Roadrunnere42   looking at my system log it seems to  be me running the update snort rules every hour

here is my Scheduled Tasks


# daily at 01:00 - upgrade IPS rules
0 1 * * *  sh /sbin/fw_upgrade
# daily at 00:00 - sync clock
0 0 * * * /usr/sbin/ntpclient -s -p 123 -h 0.us.pool.ntp.org || /etc/init.d/ntpclient restart


and my system log

System_Log.txt

It seems to happen when I go to some websites  can it be something  think with HTTPS ?


Tue Nov 22 08:49:43 2016 daemon.notice snort[16070]: +++++++++++++++++++++++++++++++++++++++++++++++++++
Tue Nov 22 08:49:43 2016 daemon.notice snort[16070]: Initializing rule chains...
Tue Nov 22 08:49:43 2016 daemon.notice snort[16070]: WARNING: /etc/snort/rules/snort.rules(1235) threshold (in rule) is deprecated; use detection_filter instead.

Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]: 6504 Snort rules read
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]:     6504 detection rules
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]:     0 decoder rules
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]:     0 preprocessor rules
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]: 6504 Option Chains linked into 2498 Chain Headers
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]: 0 Dynamic rules
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]: +++++++++++++++++++++++++++++++++++++++++++++++++++
Tue Nov 22 08:49:56 2016 daemon.notice snort[16070]:

breda

Change then to

28 03 * * * sh /sbin/fw_upgrade
40 3 * * 0 /usr/sbin/ntpclient -s -p 123 -h 0.us.pool.ntp.org || /etc/init.d/ntpclient restart

this should sort them out.

Roadrunnere42