MultiFiles

A System for File (Firmware) Distribution over UDP Multicast

Download multifiles v0.7

License: GPLv2, GPLv3.

MultiFiles Operation Principles

The file to be distributed is cyclically sent to the network in a UDP multicast stream to a specified multicast address (by default 224.2.2.4 port 2222). To receive the firmware the client needs to join the multicast group by sending an IGMP request and to start downloading the file. As soon as the download is complete, the client disconnects from the group and verifies integrity of the file. If the verification is successful, the file is unpacked (the format is .tar.gz) and customup.sh script is started that is contained in the archive just downloaded. The script updates required files and performs other actions needed for the update.

Using multicast allows to update any number of IP set-top boxes on-the-fly using only one stream.

The client doesn't perform the update if the current firmware version is similar to or newer than the one distributed in the stream. However, if the "force update" option is specified on the server, the update is always done regardless of the version of the software.

Multifiles scheme
Fig. 1. A scheme of the network with multifiles

Stream Specification

The stream consists of UDP packets with the header of the following format:

struct mulfiles_header{
uint32_t message_type;
uint32_t flags;
uint32_t data_size;
uint32_t sequence_no;
uint32_t file_offset;};

message_type can have one of the two values:

#define MF_MTYPE_INFO 0x04030201
#define MF_MTYPE_DATA 0x04030202

Packets of the MF_MTYPE_INFO type are informational packets and contain information about the distributed file, including the following data:

struct mulfiles_message_info{
uint32_t file_size;
uint32_t fw_version;
unsigned char md5sum[16];
char filename[1024];
};

file size, firmware version, file name and MD5 sum for integrity check. Informational packets are sent every 2 seconds by default.

Packets of MF_MTYPE_DATA type contain the content of the distributed file. The header includes information on the offset of the data contained in the packet with respect to the beginning of the file. The client can start downloading the file at any moment without waiting for the next cycle to start. This logic allows downloading files without any extra delays caused by searching for the beginning or end of file.

The speed of data transmission over the network can be adjusted by the command line options of the server application. By default the speed is 100 KBytes per second.

The Server Application Specs

Command line options of the server:

multifiles # ./mfsrv -h
NetUP's multicast file upload utility (server).
Copyright (c) 2001-2008 NetUP Inc.www.netup.tv.
Compile date: Nov 3 2008 13:22:49
Use this utility at your own risk.
usage:
options
[-i IP] Multicast IP address to use. Default: 224.2.2.4
[-p port] UDP port to use. Default: 2222
[-d] Enable debug messages. Default: disabled
[-f file] Name of the file to send. Default: testfile.bin
[-s speed] Upload speed Kbytes/sec. Default: 100 Kbytes/sec
[-t delay] Prophylactic info messages delay, sec. Default: 2 sec
[-u] Setting force update flag. Default: force update flag not set
[-a version] Firmware version. Default: 0
[-h] This help

An example of the server start:

multifiles # ./mfsrv  -i 224.5.6.7 -p 1111 -f ./netup.tgz -s 100 -a 1104 -t 6
Using file:./netup.tgz file name:netup.tgz prophylactic_delay:6
Speed:102400 bytes/sec psleep: 13513
+ Sending file:netup.tgz sequence:1
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o Prophylactic info: 1068 bytes from 1068 send to socket:3
o File:netup.tgz successfully sent
+ Sending file:netup.tgz sequence:2

As it can be seen from the example, the netup.tgz file is being sent to the multicast group 224.5.6.7:1111 at a speed of 100 Kbytes/s. Firmware version is 1104.

The Client Application Specs

Command line options of the client:

multifiles # ./mfcln -h
NetUP's multicast file upload utility (client).
Copyright (c) 2001-2008 NetUP Inc.www.netup.tv.
Compile date: Nov 3 2008 13:22:48
Use this utility at your own risk.
usage:
options
[-i IP] Multicast IP address to use. Default: 224.2.2.4
[-p port] UDP port to use. Default: 2222
[-t socket wait timeout] UDP socket timeout, sec. Default: 10 sec.
[-d] Enable debug messages. Default: disabled
[-f file] Name of the file to save.
Default using filename coming from server
[-s stat_file] Name of the file to save stats in.
Default: /tmp/mfstat.log
[-a fw_version] Current fw version. Download is started only
if the received version is newer than current.
Default: 0 [-h] This help

An example of the client start:

multifiles # ./mfcln -i 224.5.6.7 -p 1111 -a 1100
Waiting for incoming data ...

Info message received.
sequence:9240
filename:netup.tgz
outfilename:netup.tgz.9240
size:5741931 bytes

Receiving data ...
Filling file to required 5741931 bytes, blocks size is:1048576
File filled. Size is:5741931 requested size is:5741931.
Starting write into
First offset:1379
Prophylactic info message received.
filename:netup.tgz
offset:208380
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:615480
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:1022580
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:1431060
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:1838160
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:2245260
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:2653740
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:3060840
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:3467940
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:3875040
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:4283520
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:4690620
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:5097720
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:5506200
size:5741931 bytes
force update:0
fw_version:1104
Prophylactic info message received.
filename:netup.tgz
offset:69000
size:5741931 bytes
force update:0
fw_version:1104
File 'netup.tgz.9240' received. Checking ...
e7a646426d66b5df3318ba09f1ae33d2 - our file md5. size:5741931
e7a646426d66b5df3318ba09f1ae33d2 - received md5
MD5 check success
File 'netup.tgz.9240' checked. Success.

As it can be seen from the example, the netup.tgz file is being received from the multicast group 224.5.6.7:1111. Current file version is 1100, but version provided by the server is 1104.

Creation of the Firmware

Content of the scripts directory:

multifiles # ls -al scripts/
total 16
drwxr-xr-x 4 root root 84 Oct 27 13:51 .
drwxr-xr-x 5 root root 4096 Oct 27 15:36 ..
-rwxr-xr-x 1 root root 115 Oct 27 13:51 createfw.sh
-rwxr-xr-x 1 root root 187 Oct 27 13:42 customup.sh
drwxr-xr-x 3 root root 35 Oct 27 13:51 firmware
-rwxr-xr-x 1 root root 889 Oct 27 13:39 update.sh

createfw.sh script is used for firmware creation. As an example, the script is:

multifiles # cat scripts/createfw.sh
FWPATH="firmware"
FWNAME="mf_fw.tgz"

(cd $FWPATH; tar cvfz ../$FWNAME ./*)
echo "Firmware created. File: $FWNAME"

As it can be seen, the script archives the content of the firmware folder into the mf_fw.tgz file. This file will be provided over multicast as described above.

Directory scripts/firmware/ contains files required for the update and the script customup.sh that performs required actions on the client device after download. A simple example of such script is given below:

multifiles # cat scripts/firmware/customup.sh
echo "Running custom upgrade shell script ... Please, wait"
echo "Copying new files ... "
cp stb_client/bin/stb_client /netup/stb_client/bin/stb_client

echo "Updating config files ... "

As it can be seen, the script copies new files and, if required, updates configuration files.

Working with the Firmware on the Client

An example of the script that organizes receiving and processing the firmware on the client device (the example is provided for an IP STB):

multifiles # cat scripts/update.sh
MFPATH="/mfupdater"
NEPATH="/netup"
DSTPATH=$MFPATH/dst
STATFILE="/tmp/mfstat.log"#

Show update splash screen
cat $MFPATH/updatesp.bmp > /dev/fb0 2>/dev/null

# get current ver
if [ -f $NEPATH/version ]; then
export CURVER=`cat $NEPATH/version`
else
export CURVER=0
fi

$MFPATH/mfcln -s $STATFILE -t 3 -i 224.5.6.7 -p 1111
-f $MFPATH/download.bin -a $CURVER
if [ $? -eq 0 ]; then
echo "Updating software ... "
# Create dst path
mkdir $DSTPATH 2>/dev/null
# unbzip received file into /netup
cd $DSTPATH
cat $MFPATH/download.bin | tar -xz
# run custom script in received file
cd $DSTPATH
sh customup.sh
# save new version
if [ -f $STATFILE ]; then
cat $STATFILE > $NEPATH/version
fi
# cleanup dst dir if required. You can do this in customup.sh
echo "Update done. New fw version:"
cat $NEPATH/version
fi

As it can be seen from the code above, the script performs the following steps:

  • Fetches the current firmware version. Version info is stored in /netup/version
  • Starts downloading the firmware from the multicast group
  • If the file has been successfully downloaded, unpacks it and runs the customup.sh script contained in the archive. The script preforms required actions (copying, configuration update, etc.)
  • The new version data is saved to the file /netup/version. The next update will be performed if a newer firmware version comes from the server or if the "force update" option is set on the server.

Binary Builds

The archive includes ready binary builds for the following platforms:

  • SH-4
  • Mipsel uclibc
  • ARM9
  • PPC
  • x86

For IP STB Manufacturers

It is suggested to include the client application (mfcln) in the basic IP STB supply. On each IP STB start up it is required to run the application in order to check the presence of the multicast stream with the firmware. The check won't take more than 2 seconds, and if no firmware multicast stream found or an older firmware is provided, an ordinary start-up without any changes proceeds. If a new firmware version was obtained, it is installed and next start-ups may follow a different scheme provided by the installed firmware. This allows for just installed IP STB's to be automatically brought into operation by getting the firmware specific for the current network.