ZeroTier SDWAN

I played around with ZeroTier open source SDWAN solution last year, but forgot about it until the other day. I was wondering if there was an easier way than having to VPN into my home network when I'm away. Thought I'd give ZT a shot. Turns out, it was a great choice, as it took me all of 30 minutes to install on all my devices and configure.

ZT is a distributed network hypervisor built atop a cryptographically secure global peer to peer network that provides advanced network virtualization and management capabilities on par with an enterprise SDN switch, and similar to VXLAN Ethernet virtualization. It has two conceptually separate but closely coupled layers in the OSI model sense: VL1 and VL2. VL1 is the underlying peer to peer transport layer, the “virtual wire,” while VL2 is an emulated Ethernet layer that provides operating systems and apps with a familiar communication medium. Also, what's really cool is how broadcast is handled. Since IPv4 ARP is built on simple Ethernet broadcast and scales poorly on large or distributed networks, ZT generates a unique multicast group for each IPv4 address detected on its system and then transparently intercepts ARP queries and sends them only to the correct group. This converts ARP into effectively a unicast or narrow multicast protocol (like IPv6 NDP). Awesome! See this manual link if you're interested on further details. Now onto my setup.

I used ZeroTier's 10.10.0.0/16 subnet range, and then for each device I assigned the same last two octets as my LAN subnet range. So for instance, if a device was 10.1.1.100/24 on my LAN, my ZeroTier address would be 10.10.1.100/16, thus making it easy to just remember the second octet's change and all the other octets are identical. The setup is extremely quick. I installed it on my PiHole, NAS, Plex server, and other mobile devices. Each one got an address from ZeroTier. Then when I need to connect to any of these remotely, I just toggle the ZeroTier app on my phone, and boom I can talk to any of those devices. For my NAS/Plex server, I have to to send a magic packet to wake as I have it go to sleep after 15 minutes since I don't use it everyday. I tried sending WOL packet via my phone's LTE but it didn't work even though I have broadcast enabled in the config. Need some further troubleshooting, but the work around is to just to SSH into my PiHole (Odroid XU4 running Ubuntu and the ZeroTier client) and send a "wakeonlan <MAC address of NAS>" and I'm live. Additionally, I have MiXplorer installed on my OnePlus 6T as well as other home mobile devices. With this app I have a FTP server role that I can toggle on, which enables me to quickly share files or pictures from my phone to other devices in my ZT network.

Totally blown away at how quick and easy this was. This would be a great solution for a SMB to get a SDWAN up and running for free, as ZT enables 100 devices for free! Even the paid tiers are very fairly priced considering the cost of the other big name solutions on the market.

I never cease to have fun with networking, and these are the tools that are make the possibilities endless. There's just something so satisfying putting all the Lego pieces together and have that magic moment where everything is connected and doing what you intended.

Unifi doesn't have QoS, LLDP or CoS

I'd like to touch on 2 separate topics relating to Ubiquiti products, but they seem to get confused on the interwebs:

1. QoS vs. Smart Queues 
2. LLDP + CoS

IMHO, Smart Queues != QoS. The reason why is SQ is just generic traffic shaping based on CoDel. It only drops the fattest packets. It doesn't prioritize a specific type of traffic over others, which is what QoS is for.

Secondly, it appears that on the forums, people think LLDP is some form of QoS. This is definitely not the case. Essentially LLDP is merely a way to identify types of devices on a switchport and assign them a VLAN and other similar resources, among other things. Heck, even a known tech YouTuber made a video "trying" to explain Unifi QoS, but then starts talking about CoS and LLDP! Ridiculous, CoS != QoS. The latter is layer 3, the former layer 2. So CoS will not help any routed traffic on your WAN link, only switched locally. Then he makes a comment at the end about Unifi having QoS on par with Cisco. Again, ridiculous! He appeared to not even understand what he was saying as he fumbled with the EdgeOS backend commands, and muttering random comments as he read the running config claiming. "they've got things happening under the hood", whatever that means. I don't mind people that don't know and are willing to admit it. I do it all the time. But the people that pretend to know but don't.....nobody got time for that nonsense!

Moving on. So a major application for LLDP is VoIP. See what happens in EdgeOS:

show auto-voip oui-table
Telephony OUI  Status         Description
-------------  ----------     -----------
00:01:E3       Default        SIEMENS
00:03:6B       Default        CISCO1
00:12:43       Default        CISCO2
00:0F:E2       Default        H3C
00:60:B9       Default        NITSUKO
00:D0:1E       Default        PINTEL
00:E0:75       Default        VERILINK
00:E0:BB       Default        3COM
00:04:0D       Default        AVAYA1
00:1B:4F       Default        AVAYA2
00:04:13       Default        SNOM

As you can see, a limited list. So here is how you add the prefix for the MAC address for Sangoma:

auto-voip oui 00:50:58 oui-desc Sangoma

However it doesn't stick after a reboot because of the controller overwrite so I think it's useless.

Finally, Ubiquiti has a CoS command, but again, it's only available in the EdgeSwitch, so it's still useless for Unifi.

So Unifi has no QoS, no LLDP, and no CoS. There, I said it. Fight me. IHMO, Ubiquiti really needs to push all of their EdgeSwitch features into the Unifi controller and just have one product line, as well as implement actual QoS. But they need to do alot of things in their Unifi line. So whatever.

FreePBX/Asterisk backup/restore

I have a lot of sequential content I'd like to publish in the Sangoma PBXact deployment tips series, but I thought I'd make an out of order, short post explaining FreePBX backups and restore. Pretty simple, but a couple things to note. First, the backup process is self explanatory. You'll want to customize to include all the things you want. I always just create my own and drop everything in it. Save. Run. Secondly, this is all going to change in FreePBX 15. Not only will you have more control over specific modules, but you'll be able to restore from different versions. Right now, I'm waiting on this very thing so I can move from 13 to 15, since the upgrade script broke when trying to go from 13 to 14.

Now, if you want to ever grab that file, you go to the "Restore" section and then select the file, and download. I generally do this on a semi annual basis, just in the event I have a complete storage failure, my new install and restore will be somewhat complete.

To restore, it's a little trickier, especially if it's a complete wipe and reinstall. You'll need WinSCP (https://winscp.net/eng/index.php). This will enable you to FTP files back and forth. The directory is /var/spool/asterisk/backup. 

To restore your file from another device, Open WinSCP and drop the tgz in the same directory as the others.
The key is for some reason, the GUI won't see it if it's inside /var/spool/asterisk/backup/<specific_folder>.

....so put it in the parent directory:
/var/spool/asterisk/backup

Then, just head to the GUI, find the file, and hit restore. (or you can use the CLI)

Below is a screenshot of the directory of my custom backup, and I'm just downloading it via WinSCP instead of the GUI.

Raspberry Pi + Darkice + Icecast + FreePBX MOH streaming

A recent client requirement was to be able to use their own audio source for music on hold. This source was a box that played a monthly CD they received from corporate with seasonal ads/music and pushed out via analog component cables to the amp for the overhead speakers. Since there is no analog input on the PBXact, I had to figure out a way to take the analog audio and convert it to IP. After a bit of research, I found extensive mention of Darkice to capture, encode and then push to another program called Icecast that would then publish a network stream.

First off, you'll need some form of computer. I decided on an SBC since this is the only job it will be doing and for the small form factor+price+low power. The Raspberry Pi is the obvious choice since it's so ubiquitous and well supported with the kernel and applications. Second, a USB audio input. I used this one from Amazon: https://amzn.com/B00IRVQ0F8

More specific for my client was the fact that they used this audio system for overhead music as well, which required me to get a left/right component cable splitter, that then necked down with another adapter to the 3.5mm for the input into the Pi. This enabled both the amp/overhead speakers and the phone system to pull the same stream of audio.

The rest was done in software.

To start install Icecast:
sudo apt install icecast2

Go through the install prompts. Make sure whatever password you use for the "source" is what you put in the darkice.cfg file below. Default is "hackme"

Next install Darkice:
sudo apt install darkice

Create/etc/darkice.cfg in your home dir and add:
[general]
duration        = 0           # encoding duration, 0 = forever
bufferSecs      = 5           
reconnect       = yes       

[input]
device          = plughw:1,0  # USB audio adapter
sampleRate      = 44100       
bitsPerSample   = 16        
channel         = 2           # 1 = mono, 2 = stereo

[icecast2-0]
bitrateMode     = abr         # average bit rate
format          = mp3       
bitrate         = 128
server          = localhost  
port            = 8000      
password        = hackme    
mountPoint      = mic       
public          = yes       

To start Darkice on boot:

• crontab -e
• @reboot sleep 30 && sudo darkice -c darkice.cfg

Couple things to mention:

• The USB sound adapter I used worked the first time with no issues. YMMV. You can find articles troubleshooting this if you have issues. As a hint to see if it's a Darkice issue or the USB adapter issue, try: 
• arecord -D plughw:1,0 temp.wav  ----> for 5 seconds, then:
• aplay temp.wav 
• If the file plays back your sound/music, you know the USB adapter is working fine.
• To find out which USB adapter you need to specify in the darkice.cfg, enter the command 
• lsusb
• You might need to play with the microphone gain depending on the source signal strength/volume. Use:
• alsamixer

To test, go to the Icecast local page and you should see your mountpoint and stream stats (if a blank page, something's not configured correctly so go back and troubleshoot):
http://<ip address of raspi>:8000

To play, you can click the link to the stream, or open VLC and add a network stream:
http://<ip address of raspi>:8000/mic

Last step is to tell FreePBX how to find the audio stream. Go the "Music on Hold" module, and put the following in the application field:
/usr/bin/mpg123 -q -s --mono -r 8000 -f 8192 -b 1024 http://<ip address of raspi>:8000/mic

That's it! A fully functional, auto starting, cost effective, and simple way to convert analog audio into a network stream for your PBX.

FreePBX basics

Now that have covered some basics of VoIP in our "Sangoma PBXact deployment tips" series, lets clarify the OS of FreePBX. First is CentOS. This is the Linux distro everything is based on, with some tweaks. Then there is the backend of the server, Asterisk. It handles all of the functions needed to operate calls and databases. Then the GUI, FreePBX, that interacts with the previous but in a way that makes configuration simple. As a note, Asterisk is open sourced, and managed by Digium. FreePBX is also open source, but managed by Sangoma. Sangoma recently purchased Digium, bringing both products under one roof. This should help the teams work more closely together to streamline the product. Quick note to OS versions, at of the time of this post, FreePBX 14 is the latest stable release. v15 is right around the corner, with some important improvements. I will touch on this a later post.

So, moving on. FreePBX has modules, which can be purchased separately, or in the PBXact series that Sangoma offers. This platform includes the base OS that includes Asterisk and the GUI FreePBX, plus a large amount of commercial modules that expand the capabilities. I prefer PBXact to a basic FreePBX box. The reason is mainly due to client needs. Even if the client doesn't initially need all of the features, being able to have those already included resources later to offer as solutions is a great experience for the them (and me as the admin). If you you want just basic call capabilities, or want to load the OS on your own hardware/VM, then FreePBX can be a great way to go. There's even a way to convert from FreePBX to PBXact afterwards, or just purchase the small handful of modules that are specific to your needs.

Now might be a good time to mention the main modules I use, with some brief descriptions of their purpose. Here is a comprehensive picture of all the modules in a PBXact system. I am not going to cover every one, but in will give you an idea of what's capable in these servers.

Here are the big ones:

• Inbound routes - Incoming calls to your DIDs need to get routed somewhere in the PBX.
• Time Groups - How you build business hours, holidays ect.
• Time Conditions - You route calls to these. They have an separate destinations depending on the whether they match the time group they're linked to.
• IVR (Interactive Voice Response) - This is the menu that you can route callers to after qualifying via Time Conditions. "press 1 for ___, press 2 for ___..." A very useful module.
• Extensions & User Management - Very obvious, but a lot of options inside both of these.
• EPM - Builds templates to mass provision phones. Very important module.
• Ring Groups - Add extensions to a group so they all ring.
• VM Blast - Similar to the above, but just for Voicemail
• Parking - Similar to the POTS idea of "lines". Basically a place to park a call till another extension can pick it up.
• SipStation - Sangoma's SIP trunking service module. This enables you to pop in a large key that is unique to your account, and it will build all the trunks and import your DIDs. This is what we use and have been happy with the price and reliability. 

I will probably add to this list as I think of use cases, but for now this will get us started. I will be pushing out posts describing more in detail the use cases of the modules, and will link them.

Thanks for reading!