which file holds the Web Filter setting

Hi
On the web filter page you and select which categories you want to block, can anyone tell me which file holds this information on which filter to apply?

I'm thinking that every night the shield  downloads all the categories, say you only have porn blocked it would be better if the fw_upgrade script checked which filters where selected and download only the blocked ones, i can see that if you decided to block drugs the rules would not be there but a script could be put together to sort this out.

That is probably

/etc/config/e2guardian

config e2guardian 'e2guardian'
        option content_ads '1'
        option content_blasphemy '0'
        option content_dating '0'
        option content_illegal '1'
        option content_malicious '1'
        option content_piracy '0'
        option content_porn '0'
        option content_proxies '0'
        option content_racism '0'
        option content_social '0'
        option content_drugs '1'
        option content_gambling '1'

Now that you mentioned it and I never paid attention because I don't use the filtering on the shield but the fw_upgrade script only updates ads and malicious. All of the other categories are NOT updated and are static lists in /etc/itus/lists. However, you can selectively choose "ads" or "malicious" to update or not.

user8446 wrote

Now that you mentioned it and I never paid attention because I don't use the filtering on the shield but the fw_upgrade script only updates ads and malicious. All of the other categories are NOT updated and are static lists in /etc/itus/lists. However, you can selectively choose "ads" or "malicious" to update or not.

the UCI system has some quiet neat features that we can use. With uci get e2guardian.e2guardian.content_<VAR> you can read the e2guardian config file.

so i've added to the fw_upgrade script these lines:

# update ads rules
# To prevent the ads rules from updating put # in front (# uupdate_ads_rules)
	if [ $(uci get e2guardian.e2guardian.content_ads) = 1 ]; then
		update_ads_rules
		sleep 1
	fi

# update malicious sites rules
# To prevent the malicious rules from updating put # in front (# update_malicious_rules)
	if [ $(uci get e2guardian.e2guardian.content_malicious) = 1 ]; then
		update_malicious_rules	
		sleep 1
	fi

 
Who knows how the other lists are updated? If I check /etc/itus/lists I notice that the other files also have a recent time stamp.

I remember Itus dropping E2Guardian because of a lot of problems with it so they we're writing their own. It looks like it starts with /etc/init.d/dnsmasq :

...
        sh /etc/update_blacklist.sh
        sh /etc/itus/write-categories.sh
        sed -i '/^$/d' /etc/itus/lists/white.list
        sed -i '/^$/d' /etc/itus/lists/black.list
        sh /etc/itus/lists/whitelist.sh /etc/itus/lists/white.list
        sh /etc/itus/lists/blacklist.sh /etc/itus/lists/black.list

 sh /etc/update_blacklist.sh :

#!/bin/sh
#set +x
# Get block redirect domain ip address
ip_regex="[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+"
blockdomain_ip=$(uci get network.blockdomain.ipaddr)
blacklist_ip=`head -1 /etc/itus/lists/ads | cut -d'/' -f3`

if [[ `echo $blockdomain_ip | grep -o $ip_regex` && `echo $blacklist_ip | grep -o $ip_regex` && "$blockdomain_ip" != "$blacklist_ip" ]]
then
        #Process blacklist in parallel to increase performance
        blacklist=`echo "porn drugs gambling proxies dating blasphemy racism malicious piracy social ads illegal"`
        for list in ${blacklist}
        do
                sed -i -E "s/\/[0-9]+.[0-9]+.[0-9]+.[0-9]+$|\/$/\/$blockdomain_ip/g" /etc/itus/lists/$list &
        done

        # Wait for the last process to complete before exiting
        wait
        logger -s "update_blacklist" -t "Updated redirect ip address: $blockdomain_ip"
fi

this will update all list files (see var blacklist) no matter the settings in Service>Web Filter>Content filtering -and- all the read/writes with it.


Please check this changed script:

#!/bin/sh
#set +x
# Get block redirect domain ip address
ip_regex="[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+"
blockdomain_ip=$(uci get network.blockdomain.ipaddr)
blacklist_ip=`head -1 /etc/itus/lists/ads | cut -d'/' -f3`

if [[ `echo $blockdomain_ip | grep -o $ip_regex` && `echo $blacklist_ip | grep -o $ip_regex` && "$blockdomain_ip" != "$blacklist_ip" ]]
then
        #Process blacklist in parallel to increase performance
        #        blacklist=`echo "porn drugs gambling proxies dating blasphemy racism malicious piracy social ads illegal"`
        #        for list in ${blacklist}
        blacklist=`grep content_ /etc/config/e2guardian | grep \'1\' | cut -d "_" -f 2 | cut -d ' ' -f 1`
        for list in $blacklist
        do
                sed -i -E "s/\/[0-9]+.[0-9]+.[0-9]+.[0-9]+$|\/$/\/$blockdomain_ip/g" /etc/itus/lists/$list &
        done

        # Wait for the last process to complete before exiting
        wait
        logger -s "update_blacklist" -t "Updated redirect ip address: $blockdomain_ip"
fi

based on /etc/itus/write-categories.sh.

It looks like it now goes out to see what categories are selected first in the e2guardian config?

user8446 wrote

It looks like it now goes out to see what categories are selected first in the e2guardian config?

Right, with

blacklist=`grep content_ /etc/config/e2guardian | grep \'1\' | cut -d "_" -f 2 | cut -d ' ' -f 1`

it pulls the e2guardian config - which matches the LuCI > Services > Web Filter selections. Before it would simply create all the files, no matter the  selection that you made.

user8446 wrote

I remember Itus dropping E2Guardian because of a lot of problems with it so they we're writing their own. It looks like it starts with /etc/init.d/dnsmasq :

...
        sh /etc/update_blacklist.sh
        sh /etc/itus/write-categories.sh
        sed -i '/^$/d' /etc/itus/lists/white.list
        sed -i '/^$/d' /etc/itus/lists/black.list
        sh /etc/itus/lists/whitelist.sh /etc/itus/lists/white.list
        sh /etc/itus/lists/blacklist.sh /etc/itus/lists/black.list

would it be worth the writes to use the RAMDISK setup from roadrunner here?

# check to see of /mnt/ramdisk is mounted if not will create the ramdisk in memory.
        if mount | grep /mnt/ramdisk > /dev/null; then
                echo "yes mounted"
        else
                echo "creating ramdisk"
                mount -t tmpfs -o size=50000k tmpfs /mnt/ramdisk
        fi

	sh /etc/update_blacklist.sh                                             # <--- create lists in ramdisk
	sh /etc/itus/write-categories.sh                                       # <-- create ITUS_DNS.tmp in ramdisk
	sed -i '/^$/d' /etc/itus/lists/white.list                               # <-- update in ramdisk
	sed -i '/^$/d' /etc/itus/lists/black.list                               # <-- update in ramdisk
	sh /etc/itus/lists/whitelist.sh /etc/itus/lists/white.list          # etc
	sh /etc/itus/lists/blacklist.sh /etc/itus/lists/black.list           # etc
...

keep in mind that you have to update your backup lists for all files that you've touched.

ANY writes we can keep off of the fs will add life to our boxes. I just have those scripts and sorting commented out in /etc/init.d/dnsmasq as I don't have a need for them.

It's just a thought but could we put a cheap sd card  and us that for all the writing and when that fails just replace

roadrunnere42

Roadrunnere42 wrote

It's just a thought but could we put a cheap sd card  and us that for all the writing and when that fails just replace

roadrunnere42

Hi all

i've updated the other scripts to work in ramdisk until the final file is created.
Please check http://itus.accessinnov.com/Hotfix-160301-BETA-ONLY-td157.html item 12) for details.

Hi!

firstly, so glad I found this forum, was disappointed to see ITUS go down the gurgler.

e2Guardian does have a few issues, particularly with performance on the shield.

I changed over to using OpenDNS which allows for the same level of blocking at a DNS level, no load on the Shield, and then use DDNS with DNS-O-Matic to update my dynamic IP so the custom rules I apply follow.

All I have to do now is figure out how to intercept all DNS and forward to OpenDNS incase a local client decides to use a custom DNS eg 8.8.8.8 and therefore bypass the blocks :)

I am pretty sure I can do this with a firewall redirect in iptables.

#Redirect DNS requests to go through router
iptables -t nat -A PREROUTING -i br-lan -p udp --dport 53 -j REDIRECT --to-port 53