How long will the eMMC flash memory last on our shields?

As we all know, flash memory has a finite number of write cycles before bad blocks start to appear. I have no idea if the controller on the shield has wear leveling or not. That being said, if you are not using the web filter on your box I would comment out the updates for that as the whole thing gets downloaded and overwritten every time the update runs, decreasing the life of the memory. The whole IPS ruleset gets downloaded too instead of just the changes. I've attached the /sbin/fw_upgrade with the web filtering commented out saving memory writes and it runs much quicker too, about a minute and a half to finish and Snort is only offline for about 10 seconds (in bridge).

fw_upgrade.fw_upgrade

Thank you for this... I'm not using the web filter functions on mine in bridge - relying on built-in functions on the router making use of Yandex free web filter services.

edit: I ran this last night - replaced the standard fw_upgrade with this modified script. The Status pages indicates IPS signatures updated Feb 10, Web Filter updated Feb 9, and Shield Update Last Run Feb 9. So it looks like the IPS and WF date edits are working, but the last update run isn't. I'll see if I can figure it out.

Otherwise, thank you again for these edits!  

On another note, I haven't cracked it open to look, without opening it - does anyone know if the eMMC is soldered or socketed?

Hi user8446

That's a good point, what we need a someone who's good at scripting  to change the fw_upgrade script to,

1) download say the snort rules but only to ram
        curl -ko /tmp/botcc.portgrouped.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.portgrouped.rules
        curl -ko /tmp/botcc.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-botcc.rules
        curl -ko /tmp/ciarmy.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-ciarmy.rules
        curl -ko /tmp/compromised.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-compromised.rules
        curl -ko /tmp/dshield.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-dshield.rules
        curl -ko /tmp/emerging-exploit.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-exploit.rules
        curl -ko /tmp/emerging-malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-malware.rules
        curl -ko /tmp/emerging-mobile_malware.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-mobile_malware.rules
        curl -ko /tmp/emerging-user_agents.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-user_agents.rules
        curl -ko /tmp/emerging-web_client.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-web_client.rules
        curl -ko /tmp/emerging-worm.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-worm.rules
        curl -ko /tmp/emerging-current_events.rules https://rules.emergingthreats.net/open/snort-edge/rules/emerging-current_events.rules

then run the 3 command below on the file that s in ram

sed -i 's/alert /drop /' alert.list
sed '/^\#/d' alert.list >> temp.rules
sed '/^$/d' temp.rules > snorttemp.rules

then check if the line exists in snort.rules that’s saved on the shield,if it does check the next line and so on. If the line does not then add it to snort.rules.

this could also be done for the ads and malicious rules, at present it have
206857 asd rules
76843 mailcious rules
4388 snort rules
these change daily sometimes going up then sometimes going down.
In the long team it will surely help save the eMMC.

Also the fw_update script should have a version number added so people would know what version they are using and what was changed, again the the short team it's fine but as time goes on it becomes hard to track with a version number.

Andy

CONTENTS DELETED

Hi Ronniem1

All that script does is not download any rules for Ads and Malicious rules, It still downloads the snort rules as i believe user8446 uses something else  to block ads and malicious site.

Andy

I've made a small change to this fw_upgrade script, see http://itus.accessinnov.com/Update-script-fw-upgrade-td43.html

Also my cron job to update these rules are not daily anymore, I've set it to weekly for now.

Hans
Don't now if you have seen the massive drop in ads rules in last nights download, i have been keeping track of how many lines in the rules,

Ads
206857 going up and down slightly but last night it drop to 157555, the malicious and snort rules say about the same.

I use
wc -l /etc/itus/lists/ads
wc -l /etc/itus/lists/malicious
wc -l /etc/snort/rules/snort.rules

Have run fw_upgrade thinking maybe it stop half way through downloading but no it the right amount of rules.

Andy

If I run

wc -l /etc/itus/lists/ads /etc/itus/lists/malicious /etc/snort/rules/snort.rules

then i get

   157555 /etc/itus/lists/ads
    27778 /etc/itus/lists/malicious
     4400 /etc/snort/rules/snort.rules
   189733 total

same as you

I have:

0 /etc/itus/lists/ads
0 /etc/itus/lists/malicious
4421 /etc/snort/rules/snort.rules
4421 total

Go into /etc/itus/lists/  and clear out social, racism, proxies, porn, piracy, malicious, illegal, gambling, drugs, dating, blasphemy, and ads to free up memory

Another good GUI command line that I use is: /etc/init.d/log restart

Clear out your system log.

CONTENTS DELETED

The update script or the command line commands?

Regarding the eMMC flash memory, It is just possible that Rhino Labs could be helpful in some way. It seems that Rhino manufactured the Shield for ITUS. They also offer for sale a big brother to the Shield, an enterprise-class version:
http://www.rhinolabsinc.com/rhino-utm8-networking-appliance/

And check out the one sitting on the top. It's the shield in a new skin!  http://www.rhinolabsinc.com/sdna-7890/

user8446 wrote

And check out the one sitting on the top. It's the shield in a new skin!  http://www.rhinolabsinc.com/sdna-7890/

It is a RED shield - what does this mean for us?

It would be nice to get a hold of the OpenWRT image on it that takes advantage of the processor offloading. We can then load snort on there and all of our changes and hotfixes on top of it.

user8446,

Since Rhino Labs' contact page appears to want enquiries from companies, not individuals, would you be willing to contact them -- perhaps wearing your company hat?

Determine what it would take for Rhino to adopt us Shield orphans under their software support umbrella.

Done, email sent. I'll post what they respond.

user8446,

Possible answer to your question: "...I have no idea if the controller on the shield has wear leveling or not..."

The Wikipedia entry for OpenWrt says this about the file system:

"A writable root file system, enabling users to add, remove or modify any file. This is accomplished by using overlayfs[23] to overlay[24] a read-only compressed SquashFS file system with a writable JFFS2 file system in a copy-on-write fashion. JFFS2 supports flash wear leveling."

https://en.wikipedia.org/wiki/OpenWrt