[FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Next Topic
 
classic Classic list List threaded Threaded
168 messages Options
123456 ... 9
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Well, if your Shield isn't readily working or not in use scanning your traffic, then absolutely you could take a go at the test image.  If your machine (desktop/laptop) has a wired port, you can set the Shield between just that machine and your switch/router.  In a home environment, you probably aren't using a 10.0.0.0 network, so it would be transparent - at least outbound anyway.

Since all my testing is being done on the slot Itus used for the Gateway image, it won't even break whatever you have got going on your Shield's router or bridge mode. If your interested, let me know and I'll get the image and info to you. I'd rather not put an image out that removes the Gateway image publicly until I know for sure the replacement is solid.

I can certainly understand the frustration part. I'm still fighting the network interfaces not working at times, even with a link light and the system seeing the removal and insertion of the cable.  I'm wondering if it's a timing issue, but I've not been able to track it down yet.
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
New repo added to my Github.  https://github.com/Grommish/OpenWrt_Feeds_Packages.git

This allows me to make local changes to the OpenWrt feed for packages so they can be altered to work the way I need them do.  They are updated from the upstream and I merge them with the changes I have to make.

I'll eventually probably end up doing the same for the other feeds, but until then, I'm going to leave the submodules for the luCi, telephony, and routing pointing at OpenWrt
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Well, this is interesting!

With the recently kernel and compile changes that I've made, the speed of the Shield seems to have increased.
I ran a time on Snort loading with just over 15k rules inline afpacket mode on both br-lan and eth0.  I ran it only until the interfaces entered promiscuous mode, then bailed on it.

I also have seen massive updates in network speed, although I'm not sure if it was due to the changes or the fact I recently swapped out a network cable.



15389 Snort rules read
    14955 detection rules
    153 decoder rules
    281 preprocessor rules
15389 Option Chains linked into 1085 Chain Headers


        Command being timed: "snort -Q -i br-lan:eth0 -c /etc/snort/snort.conf"
        User time (seconds): 22.04
        System time (seconds): 1.12
        Percent of CPU this job got: 65%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 35.38s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1692096
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 149188
        Voluntary context switches: 65
        Involuntary context switches: 129
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


I didn't get a chance to run a timer on the previous load and I'm not going to go back to try it, but subjectively it seems to be a lot quicker..
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Hey Road,

Any interest in doing some Lua for luCi?

I took at look at the snort.lua you did, and I'm really no good with Lua.

I took a slightly different approach to how to handle the multiple config files Snort was setup with (config1, config2, etc)

If you check out this init.d/snort commit, I actually cat the 4 files together (config5 just seemed to be extra since the rules files were defined in config4) into a since /etc/snort/snort.conf when the service starts.  This way, we can maintain the individual tabs Itus had and on the next restart, it'll make those the "current conf".
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Question time.

I've already put out there that I really don't know Snort, so I'm checking to see if what I'm thinking is actually correct.

In Router mode, where there 2 snort instances running concurrently - one for the WAN, one for the LAN - rather than 1 instance that just covers both?  This is what it looks like from what I can see in the codebase.  Is this so you can have multiple rule sets, one less restrictive between eth1/eth2 (br-lan) and a heavier barrier over eth0?  Can anyone explain why this setup might be better than a single instance of snort covering both br-lan and eth0 at the same time?  Since a properly setup HOME_NET limits the IP scope (I'm currently using ipvar HOME_NET [10.0.0.0/8,172.16.0.0/12,192.168.1.0/16] instead of ipvar HOME_NET any, for example, with EXTERNAL_NET being !$HOME_NET)
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Roadrunnere42
Hi Grommish
I was under the impression that snort could not do multi tasking, so itus has one instant of snort running on each core, I could be wrong.

With regards to doing Lua for luCi i did look into it a few years back but could not work out do i learn Lua or luCi. I could not find any learning docs for luCi.

How are you doing testing with openwrt on the Shield, do you work things out in a vm running openwrt and then copy to the Shield?

I also ran ipvar HOME_NET [10.0.0.0/8,172.16.0.0/12,192.168.1.0/16] instead of ipvar HOME_NET any on my Shield, but left it out on any upgrade because it had a side effect which messed things up for some people, but you right not to use  ipvar HOME_NET any.

Do i have to build a firmware image from github and then copy over a you said at the beginning of these emails.

Roadrunnere42


On Wed, 26 Jun 2019 at 01:41, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
Question time.

I've already put out there that I really don't know Snort, so I'm checking to see if what I'm thinking is actually correct.

In Router mode, where there 2 snort instances running concurrently - one for the WAN, one for the LAN - rather than 1 instance that just covers both?  This is what it looks like from what I can see in the codebase.  Is this so you can have multiple rule sets, one less restrictive between eth1/eth2 (br-lan) and a heavier barrier over eth0?  Can anyone explain why this setup might be better than a single instance of snort covering both br-lan and eth0 at the same time?  Since a properly setup HOME_NET limits the IP scope (I'm currently using ipvar HOME_NET [10.0.0.0/8,172.16.0.0/12,192.168.1.0/16] instead of ipvar HOME_NET any, for example, with EXTERNAL_NET being !$HOME_NET)



If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1778.html
To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Hey Roadrunnere42,

My test versions are running Snort 2.13.9 (latest non-beta version).  I'm not sure about previous versions, but my current test is running a single Snort Config with DAQ afpacket (nfq isn't working right at the moment, I'm looking into it) in inline mode covering both eth0 and br-lan.  I have noticed that doing this, I can no longer use SSH to github, so I think there is a rule blocking outbound ssh.  This is why I was asking about the concurrently running Snort (one for WAN, one for LAN).  snort7/snort8 was setup this was, right?  Any idea which one was which?

I asked about you and Lua because I saw that most of the commits on Github were credited to you :)

I do my testing on the Shield itself.  Because I use the Gateway slot, I can leave the other two slots alone.  This allows me to do a recovery if I have to without going through the tftp process.  I'll point out that the only difference between the 3 modes is the image uboot picks on boot.  Programmatically, there isn't anything that keeps "Gateway" from being "Router", for example.

If you want a test build, I'll email you a send.firefox.com link and you can follow the directions in the first post.

The question is, do you want a version that doesn't change the mmcblk1p3 mount (changes don't survive reboots) or one that does (changes will survive reboots).  If you're not using the Gateway spot, the one that changes the mmcblk1p3 is best, since it allows you to experiment more.  Just let me know.

If you WANT to build out from source, you should be able to by getting the github source.  The only thing that might cause issues in building out the source yourself is the fact that I haven't kept track of what host packages are required, so you'd probably have to work through the build dependencies when they error.  My ubuntu box serves no purpose other than development, so I have no issues with throwing any packages on the system that I need .

Anyway, let me know..



Roadrunnere42 wrote
Hi Grommish
I was under the impression that snort could not do multi tasking, so itus
has one instant of snort running on each core, I could be wrong.

With regards to doing Lua for luCi i did look into it a few years back but
could not work out do i learn Lua or luCi. I could not find any learning
docs for luCi.

How are you doing testing with openwrt on the Shield, do you work things
out in a vm running openwrt and then copy to the Shield?

I also ran ipvar HOME_NET [10.0.0.0/8,172.16.0.0/12,192.168.1.0/16] instead
of ipvar HOME_NET any on my Shield, but left it out on any upgrade because
it had a side effect which messed things up for some people, but you right
not to use  ipvar HOME_NET any.

Do i have to build a firmware image from github and then copy over a you
said at the beginning of these emails.

Roadrunnere42
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Roadrunnere42
Hi Grommish
I have attached some files which allow snort to be upgraded to snort  2.9.9.0-2 but i found that when i upgrade it would fail, this was tracked down to snort deleting /usr/lib/daq/daq_nfq.so.
I put a script together which allowed my to upgrade the Shield easier, as you will see i just copied daq_nfg.so to a temp location, then upgraded snort and copied file back and snort worked fine. Hope this helps with (nfq isn't working right at the moment) , I had no idea that the two snort files acted on different  lan's .

As i have a spare Shield so lets got for the one that changes the mmcblk1p3 so allow more experimenting.

I await you email, I going away for 2 weeks Sunday so will not be able to reply to any emails.

Roadrunnere42

On Wed, 26 Jun 2019 at 22:30, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
Hey Roadrunnere42,

My test versions are running Snort 2.13.9 (latest non-beta version).  I'm not sure about previous versions, but my current test is running a single Snort Config with DAQ afpacket (nfq isn't working right at the moment, I'm looking into it) in inline mode covering both eth0 and br-lan.  I have noticed that doing this, I can no longer use SSH to github, so I think there is a rule blocking outbound ssh.  This is why I was asking about the concurrently running Snort (one for WAN, one for LAN).  snort7/snort8 was setup this was, right?  Any idea which one was which?

I asked about you and Lua because I saw that most of the commits on Github were credited to you :)

I do my testing on the Shield itself.  Because I use the Gateway slot, I can leave the other two slots alone.  This allows me to do a recovery if I have to without going through the tftp process.  I'll point out that the only difference between the 3 modes is the image uboot picks on boot.  Programmatically, there isn't anything that keeps "Gateway" from being "Router", for example.

If you want a test build, I'll email you a send.firefox.com link and you can follow the directions in the first post.

The question is, do you want a version that doesn't change the mmcblk1p3 mount (changes don't survive reboots) or one that does (changes will survive reboots).  If you're not using the Gateway spot, the one that changes the mmcblk1p3 is best, since it allows you to experiment more.  Just let me know.

If you WANT to build out from source, you should be able to by getting the github source.  The only thing that might cause issues in building out the source yourself is the fact that I haven't kept track of what host packages are required, so you'd probably have to work through the build dependencies when they error.  My ubuntu box serves no purpose other than development, so I have no issues with throwing any packages on the system that I need .

Anyway, let me know..



Roadrunnere42 wrote
Hi Grommish
I was under the impression that snort could not do multi tasking, so itus
has one instant of snort running on each core, I could be wrong.

With regards to doing Lua for luCi i did look into it a few years back but
could not work out do i learn Lua or luCi. I could not find any learning
docs for luCi.

How are you doing testing with openwrt on the Shield, do you work things
out in a vm running openwrt and then copy to the Shield?

I also ran ipvar HOME_NET [10.0.0.0/8,172.16.0.0/12,192.168.1.0/16] instead
of ipvar HOME_NET any on my Shield, but left it out on any upgrade because
it had a side effect which messed things up for some people, but you right
not to use  ipvar HOME_NET any.

Do i have to build a firmware image from github and then copy over a you
said at the beginning of these emails.

Roadrunnere42



If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1780.html
To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML

libdaq_2.0.6-1_octeon.ipk (109K) Download Attachment
libpcre_8.41-1_octeon.ipk (115K) Download Attachment
zlib_1.2.11-1_mips64_octeon.ipk (58K) Download Attachment
libpcre_8.40-2_octeon.ipk (116K) Download Attachment
zlib_1.2.11-1_octeon.ipk (59K) Download Attachment
snort_2.9.9.0-2_octeon.ipk (1M) Download Attachment
install ipk packages and fw_upgrade.sh (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
OpenWrt's DAQ source disables daq_nfq so I had to turn it back on (see: https://github.com/Grommish/OpenWrt_Feeds_Packages/commit/94cdb3b37e1227a4930a9d242e5a6ff805fe5606)

It compiles and installs onto Shield correctly, the error happens when it Snort starts (I'll have to get the error later).

I'll send you an email with the link here in a bit.  I want to do one more compile and cold install to test.

Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Does anyone know which is which?  snort7 and snort8, which is internal version vs external?

Or rather, does anyone have an suggestions on how it should be setup or how we want it to be setup?  At this point, I should be able to set it up for any configuration.  I just need ideas..
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Roadrunnere,

I'm beginning to suspect your assumption on why there were 2 snorts might be right.  The configurations seem identical, which you wouldn't suspect would be the case if they were actually different.

I did find some interesting things though.

If I switch from nfq to afpacket, I can span the interfaces (eth0:br-lan) and do a spread.

      PACKET_FANOUT (since Linux 3.1)
              To scale processing across threads, packet sockets can form a fanout group.  In this mode, each matching packet is
              enqueued onto only one socket in the group.  A socket joins a fanout group by  calling  setsockopt(2)  with  level
              SOL_PACKET  and  option  PACKET_FANOUT.  Each network namespace can have up to 65536 independent groups.  A socket
              selects a group by encoding the ID in the first 16 bits of the integer option value.  The first packet  socket  to
              join  a  group implicitly creates it.  To successfully join an existing group, subsequent packet sockets must have
              the same protocol, device settings, fanout mode and flags (see below).  Packet sockets can leave  a  fanout  group
              only by closing the socket.  The group is deleted when the last socket is closed.

              Fanout supports multiple algorithms to spread traffic between sockets, as follows:
              *  The  default mode, PACKET_FANOUT_HASH, sends packets from the same flow to the same socket to maintain per-flow
                 ordering.  For each packet, it chooses a socket by taking the packet flow hash modulo the number of sockets  in
                 the group, where a flow hash is a hash over network-layer address and optional transport-layer port fields.

              *  The load-balance mode PACKET_FANOUT_LB implements a round-robin algorithm.

              *  PACKET_FANOUT_CPU selects the socket based on the CPU that the packet arrived on.

              *  PACKET_FANOUT_ROLLOVER processes all data on a single socket, moving to the next when one becomes backlogged.

              *  PACKET_FANOUT_RND selects the socket using a pseudo-random number generator.

              *  PACKET_FANOUT_QM  (available  since  Linux 3.14) selects the socket using the recorded queue_mapping of the re‐
                 ceived skb.

              Fanout modes can take additional options.  IP fragmentation causes packets from the same flow  to  have  different
              flow  hashes.   The flag PACKET_FANOUT_FLAG_DEFRAG, if set, causes packets to be defragmented before fanout is ap‐
              plied, to preserve order even in this case.  Fanout mode and options are communicated in the second 16 bits of the
              integer  option value.  The flag PACKET_FANOUT_FLAG_ROLLOVER enables the roll over mechanism as a backup strategy:
              if the original fanout algorithm selects a backlogged socket, the packet rolls over to the next available one.

I was going to specify config daq_var: fanout_type=lb and config daq_var: fanout_flag=defrag, although some of the other ones might be worth looking at.  

Does anyone have any way to stress-test Snort to test for throughput?
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Roadrunnere42
Hi
It's starting to go above my knowledge level but found this git hub page about afpacket hash fault on some linux kernels, may be worth a look before spend time with afpacket-fanout

just had a look at the snort conf files and when the Shield is in bridge mode (snort_bridge.conf) it uses afpacket
config daq: afpacket
config daq_dir: /usr/lib/daq/
config daq_mode: inline
config daq_var: buffer_size_mb=300

but when in router mode (snort7.conf and snort8.conf) it uses  nfq
config daq: nfq
config daq_dir: /usr/lib/daq/
config daq_mode: inline
config daq_var: queue=7

Not sure why Itus used the two different methods, but I was told by one of there tech guys that they needed to get the Shield delivered and then sort things out with software updates.

Roadrunnere42


On Thu, 27 Jun 2019 at 18:01, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
Roadrunnere,

I'm beginning to suspect your assumption on why there were 2 snorts might be right.  The configurations seem identical, which you wouldn't suspect would be the case if they were actually different.

I did find some interesting things though.

If I switch from nfq to afpacket, I can span the interfaces (eth0:br-lan) and do a spread.

      PACKET_FANOUT (since Linux 3.1)
              To scale processing across threads, packet sockets can form a fanout group.  In this mode, each matching packet is
              enqueued onto only one socket in the group.  A socket joins a fanout group by  calling  setsockopt(2)  with  level
              SOL_PACKET  and  option  PACKET_FANOUT.  Each network namespace can have up to 65536 independent groups.  A socket
              selects a group by encoding the ID in the first 16 bits of the integer option value.  The first packet  socket  to
              join  a  group implicitly creates it.  To successfully join an existing group, subsequent packet sockets must have
              the same protocol, device settings, fanout mode and flags (see below).  Packet sockets can leave  a  fanout  group
              only by closing the socket.  The group is deleted when the last socket is closed.

              Fanout supports multiple algorithms to spread traffic between sockets, as follows:
              *  The  default mode, PACKET_FANOUT_HASH, sends packets from the same flow to the same socket to maintain per-flow
                 ordering.  For each packet, it chooses a socket by taking the packet flow hash modulo the number of sockets  in
                 the group, where a flow hash is a hash over network-layer address and optional transport-layer port fields.

              *  The load-balance mode PACKET_FANOUT_LB implements a round-robin algorithm.

              *  PACKET_FANOUT_CPU selects the socket based on the CPU that the packet arrived on.

              *  PACKET_FANOUT_ROLLOVER processes all data on a single socket, moving to the next when one becomes backlogged.

              *  PACKET_FANOUT_RND selects the socket using a pseudo-random number generator.

              *  PACKET_FANOUT_QM  (available  since  Linux 3.14) selects the socket using the recorded queue_mapping of the re‐
                 ceived skb.

              Fanout modes can take additional options.  IP fragmentation causes packets from the same flow  to  have  different
              flow  hashes.   The flag PACKET_FANOUT_FLAG_DEFRAG, if set, causes packets to be defragmented before fanout is ap‐
              plied, to preserve order even in this case.  Fanout mode and options are communicated in the second 16 bits of the
              integer  option value.  The flag PACKET_FANOUT_FLAG_ROLLOVER enables the roll over mechanism as a backup strategy:
              if the original fanout algorithm selects a backlogged socket, the packet rolls over to the next available one.

I was going to specify config daq_var: fanout_type=lb and config daq_var: fanout_flag=defrag, although some of the other ones might be worth looking at.  

Does anyone have any way to stress-test Snort to test for throughput?



If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1785.html
To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Thanks Road, i will take a look at it.  OpenWrt doesn't support the Go language, so I'd need to make it work in order to even test.

Subjectively, I can tell you that since I switched to the FANOUT Round-Robin (which spreads it across all cores in a round-robin fashion), the system load has been at about 30%-50% under load, which highly suggests that all the cores are being hit to process packets.  

I'm still running into problems where the RC2 snort.rules are blocking outbound SSH.  I wanted something to test that dropped packets and I never bothered to update to 1.51SP1 on the Shield, so I grabbed the RC2 snort.rules.  Unfortunately, when snort is running, I can't use SSH to connect to github when trying to push my commits! .  The moment I turn it off, however, it works fine.  I wish we still had contact with Daniel from Itus, since his name is all over the files.  It would be nice to get some input from him.
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
I finally got Snort to play nice with the log files.  Since it defaults to /var/log/snort/, I understand folks were running into issues where it was filling up the RAM and crashing the system.

I want to handle this by setting up a crontabs job to move the alert.fast.* files to the MMC card and then delete the binary snort.log.* files outright.  Then, I want to look into logrotate on the moved log files.

Anyone have any idea where the log files should be stashed on the / filesystem?  /etc/snort/log?  
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
https://github.com/Grommish/Itus_Shield_v2/blob/master/files/etc/snort/rotatelogs

This is the script i ended up going with.  It'll save the newest 5 alert.fast.xxxxx by timestamp in /etc/snort/logs and kill the rest off, but only after pulling any Priority 1 alerts/drops.

I know Road is on the road without email for the next 2 weeks, but he'll catch up..  @Gnomad and @user8446, currently it only pulls the Priority 1 logs from the existing alert.fast.xxxx before it culls them, but it's triggered every 30 minutes.  Should the file results be appended or overwritten (currently it's overwriting them)?  I'm thinking appending, since I'd hate for any Priority 1 logs to be lost.  of course, it's sitting in RAM, so if it reboots, it'll be gone.. Maybe it should be written to disk instead.  this way a DDNS/RCE crash bug can't just clear the history..
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Gnomad
There were previously concerns about excessive writing shortening the SSD life, so I'd vote logging to memory is fine for now.  If you're unlucky enough to get hit by a DOS attack or similar, these logs could get big quick!

So could leave implementing an optional output path as future work?
Could even provide a field in luci so users can spec it themselves, if you wanted to take it that far..

On Tue, 2 Jul 2019 at 05:15, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
https://github.com/Grommish/Itus_Shield_v2/blob/master/files/etc/snort/rotatelogs

This is the script i ended up going with.  It'll save the newest 5 alert.fast.xxxxx by timestamp in /etc/snort/logs and kill the rest off, but only after pulling any Priority 1 alerts/drops.

I know Road is on the road without email for the next 2 weeks, but he'll catch up..  @Gnomad and @user8446, currently it only pulls the Priority 1 logs from the existing alert.fast.xxxx before it culls them, but it's triggered every 30 minutes.  Should the file results be appended or overwritten (currently it's overwriting them)?  I'm thinking appending, since I'd hate for any Priority 1 logs to be lost.  of course, it's sitting in RAM, so if it reboots, it'll be gone.. Maybe it should be written to disk instead.  this way a DDNS/RCE crash bug can't just clear the history..


If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1789.html
To unsubscribe from [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*, click here.
NAML
OpenWrt SNAPSHOT, r10391-3d8d528939
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
One of the reasons I was looking into the F2FS (Flash Friendly File System) was because it incorporates wear-leveling, which helps with the destructive writes shortening the life.

Currently, I am moving the alert.fast.xxxx files (1mb each) to /etc/snort/logs, pulling out the Priority 1 logs and saving them on disk, then removing everything but the last 5 logs (by timestamp).  I'm crontab'ing this every half an hour.  You're right about the logs getting get many and large quickly, and I understand the concern.

What about this..

Every 30 minutes, pulling the Priority 1 alerts/drops out and storing that on disk, but instead of moving/removing the alert.fast.xxxx files, just removing all but the "newest" 4 (not counting the active alert.fast file) to keep the RAM from getting filled up.

This way, we can remove almost all of the MMC writes AND free up RAM.

Thoughts on this strategy?
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Gnomad
Sounds reasonable!  The original shield luci had a page showing all the custom crons - is that something you've pulled over?
If so, people can always tweak the timing to their own requirements anyway.  

Note that (from memory) not all shield models have the MMC slot, and for other users an MMC card might not be inserted - so you'll want an availability detection check upfront regardless.

On Tue, 2 Jul 2019 at 15:05, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
One of the reasons I was looking into the F2FS (Flash Friendly File System) was because it incorporates wear-leveling, which helps with the destructive writes shortening the life.

Currently, I am moving the alert.fast.xxxx files (1mb each) to /etc/snort/logs, pulling out the Priority 1 logs and saving them on disk, then removing everything but the last 5 logs (by timestamp).  I'm crontab'ing this every half an hour.  You're right about the logs getting get many and large quickly, and I understand the concern.

What about this..

Every 30 minutes, pulling the Priority 1 alerts/drops out and storing that on disk, but instead of moving/removing the alert.fast.xxxx files, just removing all but the "newest" 4 (not counting the active alert.fast file) to keep the RAM from getting filled up.

This way, we can remove almost all of the MMC writes AND free up RAM.

Thoughts on this strategy?


If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1791.html
To unsubscribe from [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*, click here.
NAML
OpenWrt SNAPSHOT, r10391-3d8d528939
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Grommish
Administrator
Every Shield has a embedded Flash card.. It's what the MMC (Multi-Media Card) in /dev/mmcblk0 means :D

It's an embedded Flash card, basically..  I wasn't referring to the front-panel SD card slot.

I'm not sure if stock luCi has the cron page, but I can certainly pull it if it doesn't.  i did notice that the cron service was disabled by default, I had to enable and start it. I'll need to figure out how to make sure it's started by default in the image.

The front-panel SD slot *should* work out of the box if anyone ever wanted to use it, but I've never bothered to check.  The drivers are there for it though.

Gnomad wrote
Sounds reasonable!  The original shield luci had a page showing all the
custom crons - is that something you've pulled over?
If so, people can always tweak the timing to their own requirements
anyway.

Note that (from memory) not all shield models have the MMC slot, and for
other users an MMC card might not be inserted - so you'll want an
availability detection check upfront regardless.
Running Itus Shield v2 Firmware
Reply | Threaded
Open this post in threaded view
|

Re: [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*

Gnomad
My mistake, yes, was thinking of the SD slot.
Sounds good.

On Tue, 2 Jul 2019 at 15:17, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
Every Shield has a embedded Flash card.. It's what the MMC (Multi-Media Card) in /dev/mmcblk0 means :D

It's an embedded Flash card, basically..  I wasn't referring to the front-panel SD card slot.

I'm not sure if stock luCi has the cron page, but I can certainly pull it if it doesn't.  i did notice that the cron service was disabled by default, I had to enable and start it. I'll need to figure out how to make sure it's started by default in the image.

The front-panel SD slot *should* work out of the box if anyone ever wanted to use it, but I've never bothered to check.  The drivers are there for it though.

Gnomad wrote
Sounds reasonable!  The original shield luci had a page showing all the
custom crons - is that something you've pulled over?
If so, people can always tweak the timing to their own requirements
anyway.

Note that (from memory) not all shield models have the MMC slot, and for
other users an MMC card might not be inserted - so you'll want an
availability detection check upfront regardless.



If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Networks-Shield-Firmware-Upgrade-WIP-tp1726p1793.html
To unsubscribe from [FIRMWARE] Itus Networks Shield Firmware Upgrade *WIP*, click here.
NAML
OpenWrt SNAPSHOT, r10391-3d8d528939
123456 ... 9