Dili Village Telco
From DiliTelco
Contents |
Introduction
This project is building a 100 node Village Telco mesh network in Dili, Timor Leste. This will simultaneously provide a low cost local telephony service and a metropolitan IP backbone. The project is a collaboration between the Free Telephony Project and Fongtil.
This Wiki documents the project management and technical parts of the project. It is hopeful the information will be useful for other Village Telco deployments.
Funding and Support
The project is being kindly funded from two sources:
1/ The Information Society Innovation Fund ISIF.
2/ This Project was made possible in part through a donation from the Internet Society (ISOC) Community Grants Program ISOC.
Special thanks to Atcom who are manufacturing a batch of Mesh Potatoes just for this project.
Background
Dili is the largest city in Timor Leste, one of the poorest countries in Asia. Mobile and fixed phone service is available but simply too expensive for the average Timorese.
There is almost no local Internet infrastructure. For example it is impossible to send an IP packet from one side of Dili to the other without sending the packet overseas using a VSAT link or a dedicated point to point Wifi link. The cost of connecting to the Internet via VSAT discourages local Internet content – as all Internet traffic must be fetched from overseas via expensive and slow VSAT pipes.
Connectivity is typically provided via small ISPs which often distribute Internet from a VSAT using a 10-20 node point-multipoint Wifi networks. These networks require tall, expensive, and often unsafe masts. Each ISP re-crosses the same path as neighbouring ISPs which is a waste of resources and an inefficient use of spectrum.
A Village Telco is built from low cost, rugged Wifi telephony devices (the Mesh Potato). Each Mesh Potato provides a single telephone land-line to the end user, and is connected to other Mesh Potatoes via a mesh Wifi network. Mesh Potatoes are robust to developing world environmental conditions (e.g. accidental abuse, weather, static damage, poor electricity supply) and are designed for low power consumption. While telephony is the primary goal, the mesh network can also carry data.
Dili Village Telco
The Dili Village Telco is one of the first Village Telco roll outs in the world.
The Mesh Network is be a public resource, managed by Fongtil, such that anyone in Dili can have fair access to the bandwidth. This provides low cost access to local IP traffic. Applications include VOIP, web and as a pipe for ISP traffic.
Goals
1. Train Timorese to roll out a Village Telco network and in associated technologies (mesh Wifi, VOIP, mesh node installation and maintenance).
2. Deploy a 100 node Village Telco mesh network to build a local call telephone network.
3. Use the Mesh Wifi network to provide a community IP backbone across metropolitan Dili to encourage local IP traffic and local content.
Status
August 17, 2010: All Mesh Potatoes and associated hardware have been delivered to Dili, and are already being deployed (WP4000). A total of 80 nodes will be deployed in Dili over the next few months. We have also decided to build 10 node networks in the country towns of Baucau and Ermera.
August 2, 2010: The 10 node pilot network has been in daily use for 2 months. Ninety more Mesh Potatoes are en route to Dili. These nodes will be deployed (WP4000) over the next two months.
May 25 2010: Ten node Pilot network has been deployed, WP3000 is complete. Deployment of next 90 nodes (WP4000) starting in June.
April 29 2010: We are currently executing WP3000 "April Workshop and Pilot network". We are have completed the April 2010 Workshop and the 10 node pilot network is currently being installed by the Fongtil team.
Press, Media and Blog Posts
This project has generated a lot of interest with the media! Here are some pods casts and media articles. They make nice introductions to the Village Telco.
Media Stories:
1. March 2010, Latin Radical pod cast which has many interesting pod casts on Timor Leste.
2. April 2010, David was interviewed on Radio Australia's Tech Stream program.
3. April 20, 2010 Radio Australia Connect Asia Program. Similiar material as (2), but a different radio program.
4. April 20, 2010 Radio Australia Pacific Beat Program. Similiar material as (2), but a different raidio program.
5. April 20, 2010: Komunikasaun Alternativa foun Baratu ba Timor oan Timor Expose news story (Timorese news site in Tetum).
Blog Posts documenting the project:
1. April 20, 2010: Dili Village Telco Part 1 David and Rosemary arrive in Dili, talk about the plans for the project, and what it's like to visit Timor Leste.
2. April 22, 2010: Dili Village Telco Part 2 Training day.
3. April 24, 2010: Dili Village Telco Part 3 Assembling mesh nodes.
4. April 24, 2010: Dili Village Telco Part 4 Problem with interference.
5. April 25, 2010: Dili Village Telco Part 5 Installing mesh nodes at the University.
6. April 25, 2010: Dili Village Telco Part 6 Training on testing mesh links.
7. April 29, 2010: Dili Village Telco Part 7 Ermera mesh network and Batman over Ethernet.
8. April 29, 2010: Dili Village Telco Part 8 Lessons learned from April 2010 Trip.
9. May 25, 2010: Dili Village Telco Part 9 Ten node Pilot Network and WP3000 complete.
9. August 31, 2010: Dili Village Telco Part 10 The Dili Village Telco rolls out.
Photos:
Photos from the Dili Village Telco from Steve's Fickr account.
Presentations and Training Courses:
1. A non-technical overview of the Dili Village Telco Project OO in presentation form.
2. Introductory technical course on the Village Telco, Mesh Potato, and mesh networking.
Contact
email: David Rowe <david_at_rowetel_dot_com>, Lemi Soares <ict4ngotimor_at_gmail_dot_com>
IRC..: irc.freenode.net #villagetelco channel
Project Plan
WP1000 Management
Write project plan and detailed task list, prepare sub contracts with Fongtil, set up budget lines manage project, measure outcomes, monitor progress.
WP2000 Kilkenny Mesh
Set up a small mesh network in David's suburb of Kilkenny, Adelaide, that models the Dili Village Telco. Gain experience in roof-top installations, Mesh Wifi performance measurement, and debugging problems. Set up server functions such as Supernode and Afrimesh. Performing this set up in a first world environment is good preparation for the Dili Village Telco roll out.
WP3000 April Workshop
A workshop Dili, Timor Leste, scheduled for April 2010, and presented by Rowetel. A training course will be presented to the Village Telco and Mesh Potato, and a 10 node Pilot network will be installed.
The training course will cover Village Telco introduction, mesh Wifi, the Mesh Potato operation and an overview of it's hardware and software components. Practical work will include configuring the Mesh Potato, reflashing and demonstrations of various problems such as interference and line of sight issues.
The installation part of the Worskhop will include assembly of the Mesh Potatoes into weatherproof boxes, building power cables, and Power over telephone Line (PoTL) adaptors. The nodes will be installed by the Fongtil team, and VTE server functions brought up. A gateway will be installed to connect local calls to the GSM/PSTN networks. Performance monitoring software will be installed to monitor the 10 node Pilot network.
Rowetel will work with Fongtil to debug and correct problems, gathering valuable experience for the Village Telco projects in the process.
WP4000 100 Node Village Telco
Using the experience gained with the Pilot network, the Fongtil team will extend the network to 100 nodes over a period of several months between July and October 2010. The completed network will supply local telephony and Internet backbone services. Fongtil will manage the day to day operation of the Dili Village Telco.
WP5000 Evaluation Visit
Rowetel will visit Dili to evaluate the project and troubleshoot any problems. Data will be gathered, and end users interviewed to see how effective the local telephony and IP backbone functions have been to the people of Dili. Reports will be published on the web to disseminate the project results as widely as possible. Selected training and installation information will be translated into Tetun and Bahasa Indonesian.
Network Design and Device Configuration
This section describes the IP network design used to implement the Dili Village Telco, including hints on configuration and testing. The goals of the network design are:
- To allow phone calls between Mesh Potatoes.
- To allow phone calls from Mesh Potatoes to the IP0X Asterisk server, which can gateway phone calls to the GSM/PSTN networks.
- To allow laptops to connect to the Mesh network via normal (non batman) Wifi Access Points (APs).
- Implement a "flat", no NAT network. Any computer on the network should be able to talk to any other computer on the network. For example a laptop connected to Wifi AP should be able to download content from a web server connected to the Ethernet port of a Mesh Potato on the other side of Dili. This supports local Timorese content and the "IP backbone" goal of the Dili Village Telco.
Network Diagram
The figure below shows the basic design, including the IPs of various subnets: Batman routes IP packets between the subnets, for example from 10.130.1.8 to 10.30.1.4. Batman also handles routing of IP packets to the Internet, via the Supernode. Once roll out is complete, there will be 100 Mesh Potatoes (only two are shown below). Mesh Potato 8 is an example of a Mesh Potato that has a Wifi Access Point (AP) connected. This allows regular Linux and Windows laptops to connect to the mesh network using normal (non Batman) Wifi. Mesh Potato 8 runs a DHCP server that allocates IPs to the local laptops. The DIR-300 could be any Wifi AP that can run in Access Point(bridged) mode, it is not running OpenWRT or Batman.
Notes and Common problems
1. The 192.168.1.0/24 and 10.30.1.x/24 networks share the same physical LAN (sample cables and Ethernet switch). This is because it is a very bad idea to route 192.168.1.0/24 packets to the mesh network. The 192.168.1.0/24 IP range is very common for LANs so we do not want to configure Batman to use this IP range, it would lead to confusion. To implement the "flat network" design we need subnets connected to the LAN to have a unique IP range on the 10.0.0.0/8 network. So we use 10.30.1.0/24 for the LAN connected to the 10.130.1.1 Mesh Potato, and 10.30.8.0/24 for the LAN connected to the 10.130.1.8 Mesh Potato.
2. Batman uses Policy routing, so routes to batman won't show up in the normal route table on Mesh Potatoes. The following example shows the ping/route command running on Mesh Potato 10.130.1.8. It has nothing connected to it's Ethernet (eth0) port. You can see we have an Internet connection (ping to Google works), but the route table shows the wrong gateway (192.168.1.20 on eth0). Batman hides the routing.
root@OpenWrt:/# ping google.com -c 3 PING google.com (150.101.98.211): 56 data bytes 64 bytes from 150.101.98.211: seq=0 ttl=60 time=31.679 ms 64 bytes from 150.101.98.211: seq=1 ttl=60 time=31.268 ms 64 bytes from 150.101.98.211: seq=2 ttl=60 time=30.167 ms root@OpenWrt:/# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 172.31.255.252 0.0.0.0 255.255.255.252 U 0 0 0 eth0 10.130.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ath0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.30.8.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 192.168.1.20 0.0.0.0 UG 0 0 0 eth0
3. If you connect a computer to the Ethernet port of a Mesh Potato you need a route from that computer back to the 10.130.1.0/24 batman network. For example on a Linux computer plugged into 10.30.1.0/24 LAN:
# ifconfig eth0:0 10.30.1.3 netmask 255.255.255.0 # route add -net 10.130.1.0/24 gw 10.30.1.1 # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.130.1.0 10.30.1.1 255.255.255.0 UG 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 10.30.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0 0.0.0.0 192.168.1.22 0.0.0.0 UG 0 0 0 eth0
The route command tells the Linux computer to use 10.30.1.1 (Supernode) as a gateway to the 10.130.1.0/24 network. Without the route command packets can't find the mesh network. This Linux machine also has an eth0 interface on the 192.168.1.0/24 network.
Reading Further
This page documents the most important information for the Dili Village Telco. There is much more information on the Mesh Potato Howto section of the Village Telco Wiki.
Supernode
The Supernode is a Ubiquity Nanostation 2 which is similar to a Mesh Potato but does not have a FXS port. The Supernode provides the following functions:
- Gateways the mesh network to the 192.168.1.x network and hence the Internet.
- Gateways the mesh network to the 10.30.1.x network.
- Is the DNS server for the mesh network.
- Runs the batman visualization server (used for Afrimesh).
1. Instructions for flashing a Nanostation 2 with the Supernode firmware are here. Note that the default IP is 192.168.10.20 (not 192.168.1.20). To access the Supernode from a Linux computer:
# ifconfig eth0:1 192.168.10.1 # telnet 192.168.10.20
2. Set the password (passwd), then telnet will be disabled and ssh enabled when you reboot.
3. Use vi to edit /etc/config/network so it looks like this:
root@OpenWrt:/etc/config# cat network
config 'interface' 'loopback'
option 'ifname' 'lo'
option 'proto' 'static'
option 'ipaddr' '127.0.0.1'
option 'netmask' '255.0.0.0'
config 'interface' 'lan'
option 'ifname' 'eth0'
option 'proto' 'static'
option 'ipaddr' '192.168.1.100'
option 'netmask' '255.255.255.0'
option 'gateway' '192.168.1.1'
option 'dns' '192.168.1.1'
config alias
option 'interface' 'lan'
option 'proto' 'static'
option 'ipaddr' '10.30.1.1'
option 'netmask' '255.255.255.0'
config 'interface' 'wifi0'
option 'ifname' 'ath0'
option 'proto' 'static'
option 'ipaddr' '10.130.1.1'
option 'netmask' '255.255.255.0'
Note in the file above we assume 192.168.1.1 is your Internet gateway, and DNS server, and that 192.168.1.100 with be the IP of the Supernode on the 192.168.1.0/24 network. Change as appropriate for your LAN.
4. Use vi to edit /etc/config/batmand to look like this:
config 'batmand' 'general'
option 'interface' 'ath0'
option 'hna' '10.30.1.0/24'
option 'originator_interval' ''
option 'preferred_gateway' ''
option 'policy_routing_script' ''
option 'disable_client_nat' ''
option 'disable_aggregation' ''
option 'gateway_class' '5000'
option 'routing_class' ''
option 'visualisation_srv' '10.130.1.1'
5. /etc/config/wireless should look like this:
config 'wifi-device' 'wifi0'
option 'type' 'atheros'
option 'channel' '1'
config 'wifi-iface'
option 'device' 'wifi0'
option 'encryption' 'none'
option 'ssid' 'potato'
option 'mode' 'ahdemo'
option 'bssid' '01:CA:FF:EE:BA:BE'
option swmerge 1
option bgscan 0
option 'network' 'wifi0'
6. Make sure dnsmasq (the DNS server) is started when the Supernode boots. If no link exists to /etc/init.d/dsnmasq create one:
root@OpenWrt:/etc/rc.d# ln -s ../init.d/dnsmasq S60dnsmasq
7. Reboot and test that you can ssh into the Supernode through the 192.168.1.0/24 IP:
# ssh root@192.168.1.100
8. ifconfig should look like:
root@OpenWrt:/etc/config# ifconfig
ath0 Link encap:Ethernet HWaddr 00:15:6D:A6:6E:FB
inet addr:10.130.1.1 Bcast:10.130.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54049864 errors:0 dropped:85 overruns:0 frame:0
TX packets:4177245 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3569047834 (3.3 GiB) TX bytes:574004051 (547.4 MiB)
eth0 Link encap:Ethernet HWaddr 00:15:6D:E0:6E:FB
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:580545 errors:0 dropped:0 overruns:0 frame:0
TX packets:272528 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:406619435 (387.7 MiB) TX bytes:374502504 (357.1 MiB)
Interrupt:4 Base address:0x1000
eth0:1 Link encap:Ethernet HWaddr 00:15:6D:E0:6E:FB
inet addr:10.30.1.1 Bcast:10.31.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:4 Base address:0x1000
gate0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:169.254.0.0 P-t-P:169.254.0.0 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1471 Metric:1
RX packets:206 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:60856 (59.4 KiB) TX bytes:420 (420.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:48316 errors:0 dropped:0 overruns:0 frame:0
TX packets:48316 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3672638 (3.5 MiB) TX bytes:3672638 (3.5 MiB)
wifi0 Link encap:UNSPEC HWaddr 00-15-6D-A6-6E-FB-00-00-00-00-00-00-00-00-00-00
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32718875 errors:0 dropped:0 overruns:0 frame:452913
TX packets:4177892 errors:3219 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:195
RX bytes:4042919661 (3.7 GiB) TX bytes:666069218 (635.2 MiB)
Interrupt:3 Memory:b0000000-b00ffffc
9. A ps command should show batmand, the visualization server vis, and the DNS server dnsmaq running:
root@OpenWrt:/etc/config# ps <snip> 675 nobody 1276 S /usr/sbin/dnsmasq -K -D -y -Z -b -E -s lan -S /lan/ - 694 root 1600 S batmand -a 10.30.1.0/24 -g 5000 -s 10.130.1.1 ath0 695 root 1600 S batmand -a 10.30.1.0/24 -g 5000 -s 10.130.1.1 ath0 697 root 1600 S batmand -a 10.30.1.0/24 -g 5000 -s 10.130.1.1 ath0 698 root 1600 S batmand -a 10.30.1.0/24 -g 5000 -s 10.130.1.1 ath0 708 root 1540 S vis -j ath0 709 root 1540 S vis -j ath0 711 root 1540 S vis -j ath0 712 root 1540 S vis -j ath0 <snip>
10. You should be able to ping devices on the LAN and mesh Wifi networks:
root@OpenWrt:~# ping 192.168.1.1 root@OpenWrt:~# ping 10.30.8.2 root@OpenWrt:~# ping 10.130.1.2
This example assumes there is a 10.30.8.2 computer on the LAN and a Mesh Potato with IP 10.130.1.2.
11. If other Mesh Potatoes are running you should be able to see them using batmand -d1 -c:
root@OpenWrt:~# batmand -d1 -c Originator (#/255) Nexthop [outgoingIF]: Potential nexthops ... 10.130.1.3 (255) 10.130.1.3 [ ath0]: 10.130.1.3 (255) 10.130.1.8 (254) 10.130.1.8 [ ath0]: 10.130.1.8 (254) 10.130.1.7 (252) 10.130.1.7 [ ath0]: 10.130.1.7 (252) 10.130.1.6 (245) 10.130.1.6 [ ath0]: 10.130.1.6 (245)
12. You should be able to ping IPs on the Internet and DNS should be working:
root@OpenWrt:~# ping 8.8.8.8 -c 5 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=241 time=222.475 ms 64 bytes from 8.8.8.8: seq=1 ttl=241 time=226.580 ms 64 bytes from 8.8.8.8: seq=2 ttl=241 time=225.057 ms 64 bytes from 8.8.8.8: seq=3 ttl=241 time=223.272 ms 64 bytes from 8.8.8.8: seq=4 ttl=241 time=224.434 ms root@OpenWrt:~# ping google.com -c 5 PING google.com (150.101.98.211): 56 data bytes 64 bytes from 150.101.98.211: seq=0 ttl=61 time=24.163 ms 64 bytes from 150.101.98.211: seq=1 ttl=61 time=24.952 ms 64 bytes from 150.101.98.211: seq=2 ttl=61 time=25.809 ms 64 bytes from 150.101.98.211: seq=3 ttl=61 time=27.851 ms 64 bytes from 150.101.98.211: seq=4 ttl=61 time=25.165 ms
Mesh Potato
The first step is to set the mesh Wifi IP. Each Mesh Potato must have a unique 10.130.1.x IP on the mesh network.
Using Web GUI
You need a Ubuntu Linux computer with an IP on the 192.168.1.0/24 network:
1. Connect your Ubuntu Linux computer to the Mesh Potato Ethernet port.
2. Start a terminal: Applications->Accessories-Terminal
3. In the terminal:
$ sudo su - # /etc/init.d/NetworkManager stop # ifconfig eth1 down # ifconfig eth0 192.168.1.4 # ping 192.168.1.20 -c 5
Make sure ping to to 192.168.1.20 works.
5. Point your web browser at http://192.168.1.20, login with user/password root/admin.
6. Change the IP Address from 10.130.1.x to 10.130.1.yourchoice. Note there is no save - just move your mouse to another GUI field and you are finished.
7. Reboot: Remove the power to your potato then plug the power back in.
8. To restore your Linux computer network settings:
# /etc/init.d/NetworkManager start
Using a Telephone
1. Switch on the Mesh Potato and wait for dial tone in the telephone (1 minute).
2. Dial 2663 (CONF using letters).
3. Wait for the prompt "Just what do you think you're doing Dave...."
4. Dial the IP on the telephone, for example 10*130*1*123
5. Wait a few seconds. The IP will be read back to you.
6. Count to 5. Hangup the phone.
7. Reboot: Remove the power to your potato then plug the power back in.
Testing the Mesh Potato
- You are now ready to make phone calls. Just dial the last digits of the IP of another Mesh Potato. For example to dial Mesh Potato 10.130.1.123 dial "123". You can also dial a full IP, for example 10*130*1*123.
- Check your mesh Potato Wifi IP on the GUI using the Web GUI procedure above.
- If other Mesh Potatoes are running you should be able to see them using batmand -d1 -c. You can run this on the Supernode (see above) or any Mesh Potato:
root@OpenWrt:~# batmand -d1 -c Originator (#/255) Nexthop [outgoingIF]: Potential nexthops ... 10.130.1.3 (255) 10.130.1.3 [ ath0]: 10.130.1.3 (255) 10.130.1.8 (254) 10.130.1.8 [ ath0]: 10.130.1.8 (254) 10.130.1.7 (252) 10.130.1.7 [ ath0]: 10.130.1.7 (252) 10.130.1.6 (245) 10.130.1.6 [ ath0]: 10.130.1.6 (245)
To get a console on a Mesh Potato use from your Linux computer use telnet 192.168.1.20. After the MP01 password has been set, use ssh root@192.168.1.20.
Mesh Potato with Wifi AP
In the Figure above, MP8 is connected to a Wifi Access Point (AP). This allows laptops a way to connect to the mesh network for:
- Internet access.
- Connecting to any other machine on the Mesh Network.
The laptop to AP link uses regular (non mesh) Wifi.
This section discusses the configuration of MP8, which can be repeated for other Mesh Potatoes that have Wifi APs connected.
Notes:
- DNS is provided by the Supernode at 10.130.1.1
- A DHCP server runs on MP8, which distributes configuration information to the laptops.
- The Wifi IP runs in bridge mode, any traffic between the Ethernet and Wifi interfaces is simply repeated. There is no NAT between the Ethernet and Wifi interfaces.
- The DIR-300 Wifi AP used for these tests runs standard firmware, as supplied from the store. It isn't running OpenWRT or batman. Any Wifi AP can be used, if it can run in bridge mode (called AP mode on the DIR-300 web GUI). The Wifi AP was configured to fetch it's configuration via DHCP.
- Batman handles routing of Internet traffic and traffic to other parts of the mesh.
- MP8 (mesh IP 10.130.1.8) uses the local LAN 10.30.8.0/24. For Mesh Potato 10.130.1.X, use local LAN 10.30.X.0/24.
1. /etc/config/network
config 'interface' 'loopback'
option 'ifname' 'lo'
option 'proto' 'static'
option 'ipaddr' '127.0.0.1'
option 'netmask' '255.0.0.0'
config 'interface' 'lan'
option 'ifname' 'eth0'
option 'proto' 'static'
option 'netmask' '255.255.255.0'
option 'ipaddr' '192.168.1.20'
option 'gateway' '192.168.1.20'
option 'dns' '10.130.1.1'
config alias
option 'interface' 'lan'
option 'proto' 'static'
option 'ipaddr' '10.30.8.1'
option 'netmask' '255.255.255.0'
config 'interface' 'wifi0'
option 'ifname' 'ath0'
option 'proto' 'static'
option 'netmask' '255.255.255.0'
option 'ipaddr' '10.130.1.8'
Note: The gateway entry above:
option 'gateway' '192.168.1.20'
Is needed, even though it does nothing (batman handles the Internet gateway function). Without it Asterisk can't make phone calls, and you see errors like:
-- extension exists, starting PBX 8
-- Executing [8@default:1] Dial("MP/1", "SIP/4...@10.130.1.8") in
new stack
[Dec 11 17:29:16] WARNING[597]: chan_sip.c:1775 __sip_xmit: sip_xmit of
0x585a88 (len 772) to 10.130.1.8:5060 returned -2: Bad file descriptor
-- Called 4...@10.130.1.8
2. /etc/config/wireless
config wifi-device wifi0 option type atheros option channel 1 config wifi-iface option device wifi0 option mode "ahdemo" option ssid "potato" option bssid 01:CA:FF:EE:BA:BE option swmerge 1 option bgscan 0 # Set distance for wireless long-shots in meters. # option distance 5500 # Broad- and multicast rate may be set here. Using default 1Mbit. # option mcast_rate 5500 # 17dBm transmit power is the maximum option txpower 17 # Don't use if you don't know what you are doing. option bursting 0 option turbo 0
3. /etc/config/batmand
config batmand general
option interface ath0
option announce 10.30.8.0/24
option gateway_class
option originator_interval
option preferred_gateway
option routing_class 1
option visualisation_srv 10.130.1.1
option policy_routing_script
4. The DHCP server is configured by /etc/udhcpd.conf
max_leases 100 start 10.30.8.100 end 10.30.8.199 interface eth0 lease_file /tmp/udhcpd.leases domain lan pidfile /tmp/udhcpd.pid option dns 10.130.1.1 option subnet 255.255.255.0 option router 10.30.8.1 lease 7200
The file /etc/init.d/dhcpd was added to start the DHCP server:
#!/bin/sh /etc/rc.common
# Copyright (C) 2008 OpenWrt.org
START=97
start() {
[ -f /etc/udhcpd.conf ] && udhcpd
}
And a symbolic link to start the DHCP server at boot time:
root@OpenWrt:# cd /etc/rc.d root@OpenWrt:/etc/rc.d# ln -s ../init.d/dhcpd S95dhcpd root@OpenWrt:/etc/rc.d# chmod 755 ../init.d/dhcpd
Now reboot.
5. After a reboot important entries of ps show
root@OpenWrt:/etc/rc.d# ps x <snip> 563 root 1584 S batmand -a 10.30.8.0/24 -r 1 -s 10.130.1.1 ath0 564 root 1584 S batmand -a 10.30.8.0/24 -r 1 -s 10.130.1.1 ath0 565 root 1584 S batmand -a 10.30.8.0/24 -r 1 -s 10.130.1.1 ath0 569 root 1968 S udhcpd <snip>
6. Useful tests
From MP8, try:
root@OpenWrt:# ping 10.130.1.1 (link to Supernode) root@OpenWrt:# ping 8.8.8.8 (an address on the Internet) root@OpenWrt:# ping google.com (tests DNS and Internet connectivity) root@OpenWrt:# ping 10.30.8.100 (Wifi AP, which should get it's address via DHCP)
From a laptop connected to the Wifi AP, check
- The laptop can connect and gets an address on the 10.30.8.x/24 network
- ping 10.30.8.100 (Wifi AP), and 10.30.8.1 (MP8)
- Then repeat ping tests above
IP04
1. You can set up an aliased eth0 interface and route by modifying /etc/inet.d/network-static to look like:
#!/bin/sh
# Start up file for network with static IP.
IF=eth0
IPADDRESS="192.168.1.216"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.1"
DNS="192.168.1.1"
case $1 in
start) ifconfig $IF $IPADDRESS netmask $NETMASK up;
route add default gw $GATEWAY;
ifconfig eth0:1 10.30.1.4 netmask 255.255.255.0;
route add -net 10.130.1.0/24 gw 10.30.1.1;
echo "nameserver $DNS" > /etc/resolv.conf;;
stop) ifconfig $IF down;;
enable) rm -f /etc/rc.d/S10network-static;
ln -s /etc/init.d/network-static /etc/rc.d/S10network-static;;
disable) rm -f /etc/rc.d/S10network-static;;
*) cat <<EOF;;
Syntax: /etc/init.d/network-static [command]
Available commands:
start Start the service
stop Stop the service
enable Enable service autostart
disable Disable service autostart
EOF
esac
Note the ifconfig eth0:1 10.30.1.4 netmask 255.255.255.0 and route add -net 10.130.1.0/24 gw 10.30.1.1 lines.
Be very careful to include the semi-colons in the file above. Any errors in this file will break the IP04 Ethernet interface. You will then need the serial cable to correct the problem.
3. Reboot the IP04, check ifconfig and route -n:
root:~> ifconfig
eth0 Link encap:Ethernet HWaddr 00:09:45:56:1E:DD
inet addr:192.168.1.216 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3953301 errors:0 dropped:0 overruns:0 frame:0
TX packets:2882733 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
Interrupt:48
eth0:1 Link encap:Ethernet HWaddr 00:09:45:56:1E:DD
inet addr:10.30.1.4 Bcast:10.30.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:48
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8176 errors:0 dropped:0 overruns:0 frame:0
TX packets:8176 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
root:~> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.30.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.130.1.0 10.30.1.1 255.255.255.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.1.22 0.0.0.0 UG 0 0 0 eth0
4. Check you can ping nodes on the mesh network:
root:~> ping 10.30.1.1 root:~> ping 10.130.1.1 root:~> ping 10.130.1.123
5. Telnet/ssh into 10.130.1.123, and check it can ping the IP04:
root@OpenWrt:~# ping 10.30.1.4
We assume that the Supernode is 10.130.1.1 and there is a Mesh Potato with IP 10.130.1.123.
6. Add some dial plan to /etc/asterisk/extension.conf on the IP04 to call an MP01 10.130.1.123:
[default] exten => 4000,1,Dial(SIP/10.130.1.123/4000)
At the IP04 command line, start and Asterisk CLI and reload the dialplan:
root:~> asterisk -r ip04*CLI> dialplan reload
Now try dialing 4000 from an IP04 extension. To debug monitor the Asterisk CLI on the IP04 and Mesh Potato (use set verbose 5 to gather lots of information).
Tests
The x86 sever is a convenient place to perform tests. Using ssh/telnet you can connect to any Mesh Potato from the x86 server. As the server has an IP on the 192.168.1.0/24 LAN you can ssh to the server from other machines on the local LAN.
1. nmap is really useful for testing which nodes are up:
david@dilivt1:~$ nmap -sP 10.130.1.0/24 Starting Nmap 5.00 ( http://nmap.org ) at 2010-04-12 07:47 CST Host supernode (10.130.1.1) is up (0.0018s latency). Host mp2 (10.130.1.2) is up (0.015s latency). Host mp3 (10.130.1.3) is up (0.016s latency). Host mp4 (10.130.1.4) is up (0.033s latency). Host mp5 (10.130.1.5) is up (0.015s latency). Host mp6 (10.130.1.6) is up (0.019s latency). Host mp7 (10.130.1.7) is up (0.034s latency). Host mp8 (10.130.1.8) is up (0.018s latency). Nmap done: 256 IP addresses (8 hosts up) scanned in 2.71 seconds
2. If you ssh to the Supernode, batmand -d1 -c gives you information on the mesh links, but the display can get very crowded:
root@OpenWrt:~# batmand -d1 -c Originator (#/255) Nexthop [outgoingIF]: Potential nexthops ... 10.130.1.3 (255) 10.130.1.3 [ ath0]: 10.130.1.3 (255) 10.130.1.8 (254) 10.130.1.8 [ ath0]: 10.130.1.8 (254) 10.130.1.7 (252) 10.130.1.7 [ ath0]: 10.130.1.7 (252) 10.130.1.6 (245) 10.130.1.6 [ ath0]: 10.130.1.6 (245)
3. From the Supernode ping the Internet, 10.130.1.x mesh Wifi nodes, 10.30.1.x hosts.
root@OpenWrt:# ping 8.8.8.8 (an address on the Internet) root@OpenWrt:# ping 10.130.1.123 (another mesh node) root@OpenWrt:# ping google.com (tests DNS and Internet connectivity) root@OpenWrt:# ping 10.30.1.4 (IP04, tests routing over mesh)
4. Some Wifi tests using ping are described on the Mesh Potato Howto page.
x86 Server
For the Dili Village Telco the x86 server is a Lenovo S10E netbook running Ubuntu 9.10 Karmic Koala. sshd and smokeping have been installed using the package manager. Some additional setup notes:
1. There is a nasty Ethernet bug on the S10Es. When large amounts of data at high speed flow over the Ethernet interface (for example a large web page or cat a large file over the LAN), the Ethernet port stops working! An ifconfig eth0 down && ifconfig eth0 up will bring the port up again. This doesn't occur with slower transfers, e.g. downloads from the Internet.
The fix is to add the file /etc/network/if-up.d/broadfix:
#!/bin/bash
if [[ "$IFACE" == eth[01] ]]; then
ethtool -K $IFACE rx off tx off
fi
Make this file executable and restart eth0:
root@dilivt1:/etc/network/if-up.d# chmod +x broadfix root@dilivt1:/etc/network/if-up.d# ifconfig eth0 down root@dilivt1:/etc/network/if-up.d# ifconfig eth0 up
It will be restarted every time the S10E boots.
2. Setting up an Ethernet interface on 10.30.1.0/24. This is a bit tricky as NetworkManager can interfere. An approach that works with NetworkManager is to add the file /etc/network/if-up.d/potato:
#!/bin/sh # Add interface and routes to connect to potato mesh network ifconfig eth0:0 10.30.1.3 netmask 255.255.255.0 route add -net 10.130.1.0/24 gw 10.30.1.1
Make it executable:
# chmod u+x /etc/network/if-up.d/potato
3. SmokePing is a really useful tool that graphs ping over time:
It can show when the mesh goes down for various reasons. It is useful to send short and long ping packets (Smokeping probes). Sometimes the mesh network will pass small packets well but long ones will be corrupted, for example due to interference. A mesh network that can reliably send large ping packets is a healthy mesh network. Here is some more information on debugging Mesh Networks.
Some Smokeping config files for the Dili Village Telco are:
/etc/smokeping/config.d/Probes
*** Probes *** + FPing binary = /usr/bin/fping ++ FPingNormal offset = 0% ++ FPingLarge offset = 50% packetsize = 1400
/etc/smokping/config.d/Targets
*** Targets *** probe = FPingNormal menu = Top title = Dili Village Telco remark = Dili Village Telco + Latency menu = DiliVT title = Dili Village Telco ++ localhost menu = localhostNormal title = localhost Normal probe = FPingNormal host = localhost ++ localhostLarge menu = localhostLarge title = locathost Large probe = FPingLarge host = localhost ++ SupernodeNormal menu = SupernodeNormal title = Supernode Normal probe = FPingNormal host = 10.130.1.1 ++ SupernodeLarge menu = SupernodeLarge title = Supernode Large probe = FPingLarge host = 10.130.1.1 ++ MP2 menu = MP2Normal title = MP2 Normal probe = FPingNormal host = 10.130.1.2 ++ MP2Large menu = MP2Large title = MP2 Large probe = FPingLarge host = 10.130.1.2
Just add more entries for MP3, MP4,......
Note: When smokeping is installed I needed to move smokeping.cgi:
# cd /var/www # mkdir smokeping # cp /usr/lib/cgi-bin/smokeping.cgi smokping
You can then view the smokeping web pages on http://server//cgi-bin/smokeping.cgi
4. Some files to make moving around the mesh easier, and save some typing:
/usr/bin/nmap-dilivt
#!/bin/sh nmap -sP 10.130.1.0/24
/etc/hosts
david@dilivt1:~$ cat /etc/hosts 127.0.0.1 dilivt1 localhost.localdomain localhost 127.0.1.1 dilivt-laptop1 10.130.1.1 supernode 10.130.1.2 mp2 10.130.1.3 mp3 10.130.1.4 mp4 10.130.1.5 mp5 10.130.1.6 mp6 10.130.1.7 mp7 10.130.1.8 mp8
Recovery
If you have serious problems with a Mesh Potato or Supernode, the best approach is to flash the firmware. This will reset it to a known state. Then configure the device as described above.
Mesh Potato Fall back IP
The Mesh Potato has a special fallback IP interface 172.31.255.254 that is always present if Linux is running. The 172.31.255.254 fallback IP may work even if the eth0 IP (192.168.1.20) has stopped working. You can telnet/ssh or try the web GUI using the 172.31.255.254 fallback IP.
Reflashing the Mesh Potato
1. The easiest way to flash a Mesh Potato is the ap51-flash command line tool (Linux version Windows version).
2. Connect your Mesh Potato into the Ethernet (eth0) port of your Ubuntu Linux PC. Make sure the Ethernet cable is OK and that you have located the correct Ethernet port. You can test using ping on the PC to send some data out of eth0, make sure the network light blinks on the Mesh Potato.
3. Stop the Ubuntu network manager running, as this can interfere:
[david@bunny mp_images]$ sudo /etc/init.d/network-manager stop
Or on some Ubuntu machines this step is:
[david@bunny mp_images]$ sudo /etc/init.d/NetworkManager stop
4. Unplug the power to the Mesh Potato for 2 seconds and plug the power back in. This will start the Mesh Potato boot loader.
5. When the "link" light comes on the Mesh Potato, start the ap51-flash software.
[david@bunny mp_images]$ sudo ./ap51-flash eth0 openwrt-atheros-root-rv233.squashfs openwrt-atheros-vmlinux-rv233.lzma
If it works, you will see:
Reading rootfs file openwrt-atheros-root-rv233.squashfs with 4325376 bytes... Reading kernel file openwrt-atheros-vmlinux-rv233.lzma with 720896 bytes... rootfs(0x006f0000) + kernel(0x000b0000) + nvram(0x00000000) sums up to 0x007a0000 bytes Peer MAC: 00:09:45:57:a7:f9 Peer IP : 192.168.1.114 Your MAC: 00:ba:be:ca:ff:ee Your IP : 192.168.1.0 Setting IP address... Loading rootfs... Sending rootfs, 8448 blocks... Initializing partitions... Flashing rootfs... Loading kernel... Sending kernel, 1408 blocks... Flashing kernel... Setting boot_script_data... Done. Restarting device... <snip>
6. The whole process will take a few minutes. The process will pause at the Flashing rootfs... and Flashing kernel... steps. Please be patient, the Mesh Potato is busy writing the images to flash, which is a slow process.
7. When you see Restarting device wait a few minutes for the Mesh Potato to reboot. When the Wifi light starts blinking, or there is dial tone in the telephone, you are ready to test.
With your laptop configured for a 192.168.1.0/24 network you can try to connect using telnet:
[david@bunny mp_images]$ sudo ifconfig eth0 192.168.1.2
[david@bunny mp_images]$ telnet 192.168.1.20
Trying 192.168.1.20...
Connected to 192.168.1.20.
Escape character is '^]'.
=== IMPORTANT ============================
Use 'passwd' to set your login password
this will disable telnet and enable SSH
------------------------------------------
BusyBox v1.14.4 (2010-01-13 18:19:41 CET) built-in shell (ash)
Enter 'help' for a list of built-in commands.
+++++++++++++++++++++++++++++++++++++++++++++++++++
Welcome to the
__ __ __ __ ___ __ ___ __
|\/| |_ |__ |_| __ |__| | | | |__| | | |
| | |__ __| | | | |__| | | | | |__|
O-----O
O / \ / \ 0
/ \ / \ / \ / \
O---O O----O O O-----O O---O
\ / \ / \ / \ /
O \ / \ / O
O-----O
Revision: 233 --- powered by OpenWrt Kamikaze
+++++++++++++++++++++++++++++++++++++++++++++++++++
root@OpenWrt:/#
Reflashing the Supernode
The easiest way to flash the Nanostation 2 Supernode is with the open-mesh-flash tool.
Notes:
- If you copy the supernode files to a USB key, use MD5sum to double check the files are OK. It is easy to pull the USB key out with the files partially written.
[david@bunny supernode]$ md5sum * 6902c581293a5aa9a02c082ac8c93f0e open-mesh-flash b021441eef6d6fd80cc10f6772eb9a0c openwrt-atheros-ubnt2-squashfs.bin eaa0ad13e984e45035a1174ba96859ac openwrt-atheros-vmlinux.lzma
- Timing can be important during this procedure. If you see Redboot telnet not supported, quickly restart open-mesh-flash using the up arrow.
- It is a good idea to turn off network manager first, see the Mesh Potato reflash procedure above.
- Do not touch the Nanostation while the colourful LEDs are blinking. This will take a few minutes. When they stop blinking remove the power for 2 seconds then plug the power back in. This will reboot the Nanostation.
- Use ping 192.168.10.20 to test:
# sudo ifconfig eth0 192.168.10.4 # ping 192.168.10.20
You need to configure your Ubuntu PC to be on the 192.168.10.0/24 network to ping and telnet the Supernode. You can change the IP of the Supernode later.
- If ping works then try telnet
# telnet 192.168.10.20
