Itus Networks Owners Forum › Technical Discussion

[FIRMWARE] Itus Shield v2

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

128 messages Options

1234567

Grommish

A side idea for the Shield?

Administrator

Does anything else thing the Shield would be a bang-up SIP/VoIP PBX server?

https://openwrt.org/docs/guide-user/services/voip/asterisk

Hmmmm.. hehe

Running Itus Shield v2 Firmware

Grommish

A small status update..

Administrator

Hey all!

I've been working on some things.. Mostly, I'm working on getting the Itus Shield stuff sorted out so I can attempt to submit the request to OpenWrt for official inclusion and automated builds..

I've had a few things in the packages stuff accepted, so, we will find out what bar is set for the device itself. Complicating issues is the fact I have to make it play nice with the Ubiquiti Edgerouter/Edgeroute Lite which is part of the same Octeon arch (although Octeon+ rather than Octeon3).

I've successfully used Github as a opkg repo and whatnot.. I'm working on building out two modes, but the Bridge mode will take a back seat to the Router.

I'm currently stuck on getting the upgrade system to work properly.. It's close, but not quite there yet.

A point to ask for options on..

Currently, the Shield has root partitions for the images at 850MB.. My image currently only uses 400MB and I'm not sure if it can be expanded. I'm looking at other ways of doing the upgrade, but.. meh..

It's coming :)

Running Itus Shield v2 Firmware

Grommish

Re: A small status update..

Administrator

Ok..

Well, this is getting more and more interesting..

First: I've made definitions for both the Router and Bridge modes. I've decided to scrap the Gateway mode and turn it into an emergency de-brick boot (everything will be kept in RAM like a LiveCD. I hate TFTP as much as anyone).

I'm working on trying to build out different formats for each, but keep it combined in OpenWrt. At the end of this, I'm going to really try and get it set as an official device so it'll get regular builds. The first step is figuring out the build system and how the target directories are laid out. I've got to re-arrange the Edgerouter/Edgeroute Lite stuff and create a generic octeon device build, in addition to the Itus builds.

I've got it set to build both Router and Bridge images at the same time (yay).. and build out all the packages as well.. Then, I can push them to the Github repo so everyone can use luCi and/or opkg to install files..

I'm trying to figure out the sysupgrade system, as well.. Currently, I'm building out sysupgrade images that contain the kernel image and a root image. Currently, the rootfs is creates is only 104MB in size, and calling the dd write only writes 104MB, which limits the available space from 850Mb to 104Mb This is.. sub-optimal.. Granted, it's WAY more than routers usually get, but..

When I increase the rootfs size, I need to be able to fit it expanded into RAM before dd'ing it to the device. I also need to be careful not to stall the device while it does the dd, because then the watchdog kicks in and kills the system. I'm ALSO generating a rootfs.tar.gz file that contains the files that I can roll out, but it doesn't put the rootfs.tar.gz and the kernel in the same file ;/ If I can get it to this, then the upgrade system will be complete (for real.. you can even send it thru luCi! :) ) And, it will preserve existing configurations... Or should.. testing has to be done.

After that, it's a matter of trying to get the separate configurations for each mode, if I can..

If I cannot, or can't easily, then both images will be set to act as a router by default.. Since you have to add packages yourself, you can set the network however you need it.. This will at least prevent any chance of IP conflicts on a given network though it'll create a higher barrier to get it setup..

I am seriously looking at using f2fs over ext4 for the filesystem.. f2fs is FAR better and gentler on the eMMC, but it makes recovery a pain if the image doesn't know f2fs (see above about the gateway slot).

Does anyone have any objections regarding f2fs?

Running Itus Shield v2 Firmware

Roadrunnere42

Re: A small status update..

Anything that makes life easier for the emmc must be better, but as you say recovery would be a pain. How would you go about recovering an image?

On Wed, 8 Jul 2020 at 13:33, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:

Ok..

Well, this is getting more and more interesting..

First: I've made definitions for both the Router and Bridge modes. I've decided to scrap the Gateway mode and turn it into an emergency de-brick boot (everything will be kept in RAM like a LiveCD. I hate TFTP as much as anyone).

I'm working on trying to build out different formats for each, but keep it combined in OpenWrt. At the end of this, I'm going to really try and get it set as an official device so it'll get regular builds. The first step is figuring out the build system and how the target directories are laid out. I've got to re-arrange the Edgerouter/Edgeroute Lite stuff and create a generic octeon device build, in addition to the Itus builds.

I've got it set to build both Router and Bridge images at the same time (yay).. and build out all the packages as well.. Then, I can push them to the Github repo so everyone can use luCi and/or opkg to install files..

I'm trying to figure out the sysupgrade system, as well.. Currently, I'm building out sysupgrade images that contain the kernel image and a root image. Currently, the rootfs is creates is only 104MB in size, and calling the dd write only writes 104MB, which limits the available space from 850Mb to 104Mb This is.. sub-optimal.. Granted, it's WAY more than routers usually get, but..

When I increase the rootfs size, I need to be able to fit it expanded into RAM before dd'ing it to the device. I also need to be careful not to stall the device while it does the dd, because then the watchdog kicks in and kills the system. I'm ALSO generating a rootfs.tar.gz file that contains the files that I can roll out, but it doesn't put the rootfs.tar.gz and the kernel in the same file ;/ If I can get it to this, then the upgrade system will be complete (for real.. you can even send it thru luCi! :) ) And, it will preserve existing configurations... Or should.. testing has to be done.

After that, it's a matter of trying to get the separate configurations for each mode, if I can..

If I cannot, or can't easily, then both images will be set to act as a router by default.. Since you have to add packages yourself, you can set the network however you need it.. This will at least prevent any chance of IP conflicts on a given network though it'll create a higher barrier to get it setup..

I am seriously looking at using f2fs over ext4 for the filesystem.. f2fs is FAR better and gentler on the eMMC, but it makes recovery a pain if the image doesn't know f2fs (see above about the gateway slot).

Does anyone have any objections regarding f2fs?

Running Itus Shield v2 Firmware

If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Shield-v2-tp2014p2079.html

To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML

Grommish

Re: A small status update..

Administrator

I create a Initramfs image in the Gateway spot. It effectively acts as a "OpenWrt" Linux LiveCD. As long as that image is compiled with f2fs, there would be no issues with using f2fs.. The only issue would be with trying to recover data from the partition if it bricks.. though at that point, you're better off starting over, but the security side of my personality won't let me not bake in a way to try and get stuff back.

The initramfs image is basically the entire image (kernel + rootfs) in a single compressed squashfs ELF binary image. It's loaded the exact same way as the other images, but it maintains its rootfs in RAM, so it can't be changed, but you also can't screw it up heheh

I use that for the times I hose the other images..

Running Itus Shield v2 Firmware

Roadrunnere42

Re: A small status update..

In reply to this post by Roadrunnere42

I say go for f2fs, so the emmc will last longer.

On Wed, 8 Jul 2020 at 14:07, Roadrunnere42 [via Itus Networks Owners Forum] <[hidden email]> wrote:

Anything that makes life easier for the emmc must be better, but as you say recovery would be a pain. How would you go about recovering an image?

On Wed, 8 Jul 2020 at 13:33, Grommish [via Itus Networks Owners Forum] <[hidden email]> wrote:
Ok..

Well, this is getting more and more interesting..

First: I've made definitions for both the Router and Bridge modes. I've decided to scrap the Gateway mode and turn it into an emergency de-brick boot (everything will be kept in RAM like a LiveCD. I hate TFTP as much as anyone).

I'm working on trying to build out different formats for each, but keep it combined in OpenWrt. At the end of this, I'm going to really try and get it set as an official device so it'll get regular builds. The first step is figuring out the build system and how the target directories are laid out. I've got to re-arrange the Edgerouter/Edgeroute Lite stuff and create a generic octeon device build, in addition to the Itus builds.

I've got it set to build both Router and Bridge images at the same time (yay).. and build out all the packages as well.. Then, I can push them to the Github repo so everyone can use luCi and/or opkg to install files..

I'm trying to figure out the sysupgrade system, as well.. Currently, I'm building out sysupgrade images that contain the kernel image and a root image. Currently, the rootfs is creates is only 104MB in size, and calling the dd write only writes 104MB, which limits the available space from 850Mb to 104Mb This is.. sub-optimal.. Granted, it's WAY more than routers usually get, but..

When I increase the rootfs size, I need to be able to fit it expanded into RAM before dd'ing it to the device. I also need to be careful not to stall the device while it does the dd, because then the watchdog kicks in and kills the system. I'm ALSO generating a rootfs.tar.gz file that contains the files that I can roll out, but it doesn't put the rootfs.tar.gz and the kernel in the same file ;/ If I can get it to this, then the upgrade system will be complete (for real.. you can even send it thru luCi! :) ) And, it will preserve existing configurations... Or should.. testing has to be done.

After that, it's a matter of trying to get the separate configurations for each mode, if I can..

If I cannot, or can't easily, then both images will be set to act as a router by default.. Since you have to add packages yourself, you can set the network however you need it.. This will at least prevent any chance of IP conflicts on a given network though it'll create a higher barrier to get it setup..

I am seriously looking at using f2fs over ext4 for the filesystem.. f2fs is FAR better and gentler on the eMMC, but it makes recovery a pain if the image doesn't know f2fs (see above about the gateway slot).

Does anyone have any objections regarding f2fs?

Running Itus Shield v2 Firmware

If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Shield-v2-tp2014p2079.html

To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML

If you reply to this email, your message will be added to the discussion below:
http://itus.accessinnov.com/FIRMWARE-Itus-Shield-v2-tp2014p2080.html

To start a new topic under Technical Discussion, email [hidden email]
To unsubscribe from Itus Networks Owners Forum, click here.
NAML

Grommish

Re: A small status update..

Administrator

Gotcha, will do! Check this out..

Kernel 5.4, working Ethernet driver..

Running Itus Shield v2 Firmware

Grommish

Re: A small status update..

Administrator

For those who haven't seen it, I've publicly released v2..

http://itus.accessinnov.com/Itus-Networks-Shield-Firmware-v2-Released-tp2084.html

Running Itus Shield v2 Firmware

Grommish

v2 continuing development

Administrator

I'm going to continue this thread for general discussion and updates on the continued development going forward.

I've successfully updated everything to the Linux 5.4 kernel (I think), though I'm still working on some other things.

If you are feeling froggy, you can try the new Kernel level.

This is the correct branch for the 5.4 kernel files:
https://github.com/Grommish/shield_opkgs/blob/r13863+81-9f3415d30b/

You can find the files for 5.4 here: https://github.com/Grommish/shield_opkgs/tree/r13863%2B81-9f3415d30b/targets/octeon/itus

If you haven't already upgraded to the 4.19 v2, you'll follow the same directions as in the release thread, as you'll still need to create the f2fs partition for the storage.

If you have already upgraded to the 4.19 v2 that was released a few days ago, you can just run the sysupgrade-itusbridge.tar.gz or sysupgrade-itusrouter.tar.gz directly from luCi or sysupgrade CLI. Please do NOT save the existing configuration!

You'll need to check/change the distfeeds.conf as well, since I'm still working out the process for these things and didn't bake it in first (it'll be fixed in future revisions):

src/gz openwrt_core https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/targets/octeon/itus/packages
src/gz openwrt_base https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/base
src/gz openwrt_freifunk https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/freifunk
src/gz openwrt_luci https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/luci
src/gz openwrt_packages https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/packages
src/gz openwrt_routing https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/routing
src/gz openwrt_telephony https://github.com/Grommish/shield_opkgs/raw/r13863+81-9f3415d30b/packages/mips64_octeon3/telephony

Running Itus Shield v2 Firmware

user8446

Re: v2 continuing development

Administrator

That's great, you may be ahead of the OpenWRT team. On their main page, it says "The OpenWrt 19.07 series focuses on bringing all supported targets to Linux kernel version 4.14 and introducing initial device tree based ath79 support."

Are they really still on 4.14 for their newest release?

Running the latest OpenWrt stable release

Grommish

Re: v2 continuing development

Administrator

I don't think any device is using 4.14, unless it's a small flash device (they are dropping support for less than 8MB flash size devices, for example)... Most are 4.19 or 5.4. I'm not sure about the static Official builds like 17.x, 18.x, 19.07, etc. I know when I did a pull on the 19.07 branch, it was outdated. But, it's supposed to be, which is why it's considered stable :)

I started with just the master branch and built on that. Before the release, I'd just pull every few days on the main repo, but now I suppose I need to start doing it properly with branches.

I was having issues with the ethernet driver under 5.4, but it was fine under 4.19, so i went with 4.19. When I updated the tools and toolchains (gcc to 8.x to 10.1, musl to 1.2.0, uClibc) because they were behind in the core, it seemed to fix the issue. I didn't want to confuse anyone potentially doing the upgrade with multiple kernel versions and links and I've not fully tested it out.

Running Itus Shield v2 Firmware

Grommish

Re: v2 continuing development

Administrator

Remember when i said I was still playing with things?

I've got squashfs and overlayfs mostly working!

https://hastebin.com/azefujakav.coffeescript

This will simplify the upgrade process, if I can figure out why the kernel file gets changed when moved.. https://hastebin.com/amutifubup.rb

If I copy the kernel file in the sysupgrade file directly, it works.. if I use sysupgrade, it bricks.. Not sure why yet..

Running Itus Shield v2 Firmware

Gnomad

Re: v2 continuing development

Well I had to look up what those fs'es were, but looks very promising - nice work Grommish!!
I guess you're thinking a squashfs'd base image, with overlayfs to hold user configs & customisations?

I'll try give your 5.4 update a whirl today, report back how it goes :)

OpenWrt SNAPSHOT, r10391-3d8d528939

Grommish

Re: v2 continuing development

Administrator

Gnomad wrote

Well I had to look up what those fs'es were, but looks very promising - nice work Grommish!!
I guess you're thinking a squashfs'd base image, with overlayfs to hold user configs & customisations?

I'll try give your 5.4 update a whirl today, report back how it goes :)

Yes.. Because it'll still allow for the f2fs (Flash-Friendly File System), which is MUCH better for the eMMC than standard ext3/ext4, while simplifying the update process. With ext4, I was having to either short-size the partition to fit (because shoving a 850MB root file to flash into RAM wasn't an option), or hack up the sysupgrade system like I did previously.

using Squashfs and the Overlayfs means the upgrade file will be the size of the initramfs originally, but will scale.. - Or.. should ;D

If you run into issues, hit me up on Google Hangouts or Gmail.

Running Itus Shield v2 Firmware

Gnomad

Re: v2 continuing development

Up and running with 5.4 from https://github.com/Grommish/shield_opkgs/tree/r13871-546e140382, with AdBlock configured per your recommendations
(thanks to Grommish for help via Hangouts!)

Will keep you posted re: any stability issues, but all looking much cleaner than Itus v1.5 so far!

Now.. to pick what else to install..

OpenWrt SNAPSHOT, r10391-3d8d528939

Gnomad

Re: v2 continuing development

Grommish, you may/not be aware yet of Github Actions?
https://github.com/features/actions

Basically, you can setup a pipeline to automatically build stuff (via containers to your own spec) whenever you push updates to the repo. This could save you a heap of desktop build effort, maybe even allow you to automate whenever the upstream OpenWrt master branch changes.

Looks like some people have already worked on similar for OpenWrt - e.g.
a readme and file to define the action workflow
https://github.com/P3TERX/Actions-OpenWrt/blob/master/.github/workflows/build-openwrt.yml

You'd just need to copy & adapt that yaml for the Itus device, drop it into the same named folder `.github/workflows`..

OpenWrt SNAPSHOT, r10391-3d8d528939

Grommish

Re: v2 continuing development

Administrator

Nope, was not aware, but I am now :)

I'll look at that. My primary goal is to get the device introduced as an official device for OpenWrt, so we can let the people maintaining the big servers deal with building things out :)

I am also learning about HOW their build system works.. For example, the issue you want into with banip and the kernel "version". The code didn't change, I didn't do an upstream merge or anything, i simply removed a package (or tried to) and rebuilt.. the result? Kernel mismatch on only certain things.

FYI, doing a sysupgrade with the bridge sysupgrade file will fix your issue, but you'll have to reinstall and setup adblock.. You can see what I'm looking for help wise here. it's the only thing holding this up.. I cannot get /dev/mmcblk1pX to mount r/w so I can move the sysupgrade.tgz file there (so it'll be applied on the next boot)..

Running Itus Shield v2 Firmware

Grommish

Need Your Input...

Administrator

Well.. in talking with some of the OpenWrt dev team, they want to concat three images into one in order to be considered for inclusion into the mainline.

This means making the mode switch irrelevant and having only a single image-mode. At that point, we either lose out on 1.7Gb of space by staying with the 850Mb partition for the image, or repartition..

Thoughts? Is being included in the mainline worth it? I can see their point, but without a lot of work I'm not willing to undertake right now, I don't know how to create a single image that will work in all three spots without far too much hassle.

Running Itus Shield v2 Firmware

Turrican

Re: Need Your Input...

You know, I'm kind of ok with that notion...

I guess you could reconfigure the device within OPENWRT if you wanted to make it operate in a different mode than router right?

Running v2 Firmware

Grommish

Re: Need Your Input...

Administrator

You could do anything you wanted, yes. Nothing on the switch enables or disables anything hardware wise..

The question is, how important was the multi-mode ability when you purchased the Shield?

Just to be clear, I'm not talking about abandoning the project, just not getting it included in the openwrt mainline

Running Itus Shield v2 Firmware

1234567