I Over-Complicated my Home Network for fun

So, first and foremost, the characters involved:

1. Router: my trusty Mikrotik RB2011 - This is an old boy now, still performs better than any consumer routers and offers far more customizationthan your average wifi router. I don't use it's 2.4G WIFI 4 radio anymore. However, rest of it is still promising. 5 Gigabit ethernet ports for your high speed needs and 5 10/100 Fast ethernet ports for the old Raspberry pi 2 clusters. Also comes with a micro-usb port to get an LTE dongle hooked as a failover.
2. LTE: decomissioned JioMiFi 4G hotspot - This too is essentially 4 Years old at this point and since then, I've bought a more modern 4G hotspot from jio for my on the go needs. Since this was lying around along with a spare 4G sim, I dediced to go with it. The device performs great with acceptable 20 Up/5 Down speeds over its USB port which is enough when my Primary internet is down.
3. Raspberries: decomissioned Raspberry Pi 2B(s) - I use these for things like Pi-Hole and wireguard along with some basic monitoring tools for my cloud infra. This isn't the most ideal of situations but gets the job done and I have control over the monitoring infra and methods. As they say, gets the job done.

Enough overview, Now let's get to the main story:

While spring cleaning, I came across my stash of devices that are not used anymore. These devices were retired of their normal duties due to them being replaced by better alternatives at work or at home. These devices were collecting dust for a really long time and I was experiencing a first world problem of ISP level outages for the entire lockdown period. I couldn't do much about it since no other ISP is willing to serve our area and the only ISP I have right now is better than those shady run-down-the-street style ISPs. This one at least offers FTTx connectivity to their service box and terminates connections as standard ethernet cable. Due to ongoing construction and lack of maintenance of the infrastructure, their fiber backhaul tends to get damaged and once again, due to pandemic, the lack of staff and the social distancing norms that come with it. They were consistently failing to keep their services running smooth. The problem wasn't too bad for a normal day, The link only goes down once or twice a month for a couple of hours. However, when your and your family's livelyhood depends on it. You have to take some steps to make things better.

Now, as the only certified internet fixer wizard of my family, I had to do something to keep up my brownie points and to keep the peace in house. It struck my mind that I have these spare items lying around which could be put to use as a failover so that no outage could cause disruption for long. I started investigating the possibilities and found out that Mikrotik had added support for various Jiofi hotspots to their RouterOS and that started my hunch to get the 4G hotspot to become the Internet Saviour.

My existing WIFI situation was satisfactory with a set of Netgear orbi RBK13 placed strategically to cover all the internet soft-spots. The main router of the orbi kit was my primary source of internet and it worked well for the most part. However, It didn't had any option to set up an automatic failover. first off, there are only two ethernet ports. One is marked WAN and hence can not be used as a switch and the other LAN port is a loner, there is no USB port on this device either to plug in the WIFI hotspot as a secondary link. This is where my trusty RB2011 comes into play. It has what it takes to get me going. Plus, it eliminates the need of having my raspberry Pi(s) plugged into a cheap tenda switch. I can finally have dedicated subnets for my home WIFI and IoT devices along with a separate guest subnet to keep the visitors away from my internal infrastructure while still giving them internet access. I brought in an old netgear router and made it the guest access point and placed it in a good location that gives enough strength in the common area. This allowed us to place a QR code installed in a photo frame that lets anyone connect to the guest wifi when scanned.

For the rest of system, the process was fairly straightforward. Mikrotik provides very detailed instructions on how to configure their products for various scenarios and their configuration is generally universal i.e. any device that has enough ports can adapt to the same config without a tussle. So, I plugged in my Primary internet to Mikrotik, followed by the JioMiFi hotspot into it's usb port. I had to take out the battery and use a special powered USB OTG cable to give enough juice for the hotspot to run on USB power instead of destroying the battery. One thing I learned here is that the adapter supplied with hotspot doesn't have enough AMPs to keep the hotspot running on USB power. I had to connect it to a different 5v 2A adapter (it needs 1.2A for normal ops but a little extra won't hurt) which got it running and I didn't have to bother about it anymore. Next, I created a bridge of the 5 10/100 Fast Ethernet ports and plugged in my Raspberry Pi cluster to the ports. These Pis run adblocker (Pi-Hole) and a few network monitoring tools which I'll talk about in a different article. For this one, Let's continue with the rest of details.

Next, I set up the routes and other system specific configurations to make sure that everything seamlessly switches to 4G when Ether1 is down and switches back to Ether1 when it is available once again. Once that was sorted, I established two DHCP servers on Ether2 and Ether3. Ether2 is connected to the orbi system to get internet access for the rest of home, can't keep them away from internet or else they all go grumpy. Once that was done, I connected Ether3 to my other AP for guest WIFI. once all of it is done, All that was left was to test if everything is working as expected. I disconnected the Ether1 through winbox since I'm lazy and didn't want to walk to the router and physically unplug it. Nevermind, It all went smooth until . . . (insert a really long moment of silence) I plugged it back in and it didn't switch back to my primary. I just couldn't fathom what went wrong. Did everything once again and it still didn't work. Turns out, I messed up the routing distance and setting up correct distance was enough to get things back in order.

Nonetheless, It all went smooth at the end and now whenever our main link is down, everything switches to 4G and when its back online, everything magically switches back to primary. Now, For the faint of heart, I would like to clarify that the aforesaid 4G is part of a pooled data family plan and due to this fact, it generally has ~200GB of data at our disposal. Even if we continue using data as if nothing happened, usage trends show that our internet outages don't last so long to take up all the 200GBs out of the plan. and even if that happens, getting more GBs isn't too expensive if need be.