MultiFiles

A System for File (Firmware) Distribution over UDP Multicast

Download multifiles v0.7

License: GPLv2, GPLv3.

MultiFiles Operation Principles

The file to be distributed is cyclically sent to the network in a UDP multicast stream to a specified multicast address (by default 224.2.2.4 port 2222). To receive the firmware the client needs to join the multicast group by sending an IGMP request and to start downloading the file. As soon as the download is complete, the client disconnects from the group and verifies integrity of the file. If the verification is successful, the file is unpacked (the format is .tar.gz) and customup.sh script is started that is contained in the archive just downloaded. The script updates required files and performs other actions needed for the update.

Using multicast allows to update any number of IP set-top boxes on-the-fly using only one stream.

The client doesn't perform the update if the current firmware version is similar to or newer than the one distributed in the stream. However, if the "force update" option is specified on the server, the update is always done regardless of the version of the software.

Fig. 1. A scheme of the network with multifiles

Stream Specification

The stream consists of UDP packets with the header of the following format:

struct mulfiles_header{    uint32_t message_type;    uint32_t flags;    uint32_t data_size;    uint32_t sequence_no;    uint32_t file_offset;};

message_type can have one of the two values:

#define MF_MTYPE_INFO 0x04030201#define MF_MTYPE_DATA 0x04030202

Packets of the MF_MTYPE_INFO type are informational packets and contain information about the distributed file, including the following data:

struct mulfiles_message_info{    uint32_t file_size;    uint32_t fw_version;    unsigned char md5sum[16];    char filename[1024];};

file size, firmware version, file name and MD5 sum for integrity check. Informational packets are sent every 2 seconds by default.

Packets of MF_MTYPE_DATA type contain the content of the distributed file. The header includes information on the offset of the data contained in the packet with respect to the beginning of the file. The client can start downloading the file at any moment without waiting for the next cycle to start. This logic allows downloading files without any extra delays caused by searching for the beginning or end of file.

The speed of data transmission over the network can be adjusted by the command line options of the server application. By default the speed is 100 KBytes per second.

The Server Application Specs

Command line options of the server:

multifiles # ./mfsrv -hNetUP's multicast file upload utility (server). Copyright (c) 2001-2008 NetUP Inc.www.netup.tv. Compile date: Nov  3 2008 13:22:49 Use this utility at your own risk.usage:options [-i IP]          Multicast IP address to use. Default: 224.2.2.4 [-p port]                UDP port to use. Default: 2222 [-d]             Enable debug messages. Default: disabled [-f file]                Name of the file to send. Default: testfile.bin [-s speed]               Upload speed Kbytes/sec. Default: 100 Kbytes/sec [-t delay]               Prophylactic info messages delay, sec. Default: 2 sec [-u]             Setting force update flag. Default: force update flag not set [-a version]             Firmware version. Default: 0 [-h]             This help

An example of the server start:

multifiles # ./mfsrv  -i 224.5.6.7 -p 1111 -f ./netup.tgz -s 100 -a 1104 -t 6Using file:./netup.tgz file name:netup.tgz prophylactic_delay:6Speed:102400 bytes/sec psleep: 13513+ Sending file:netup.tgz sequence:1o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    Prophylactic info: 1068 bytes from 1068 send to socket:3o    File:netup.tgz successfully sent+ Sending file:netup.tgz sequence:2

As it can be seen from the example, the netup.tgz file is being sent to the multicast group 224.5.6.7:1111 at a speed of 100 Kbytes/s. Firmware version is 1104.

The Client Application Specs

Command line options of the client:

multifiles # ./mfcln -hNetUP's multicast file upload utility (client). Copyright (c) 2001-2008 NetUP Inc.www.netup.tv. Compile date: Nov  3 2008 13:22:48 Use this utility at your own risk.usage:options [-i IP]          Multicast IP address to use. Default: 224.2.2.4 [-p port]        UDP port to use. Default: 2222 [-t socket wait timeout]       UDP socket timeout, sec. Default: 10 sec. [-d]             Enable debug messages. Default: disabled [-f file]        Name of the file to save. Default using filename coming from server [-s stat_file]   Name of the file to save stats in. Default: /tmp/mfstat.log [-a fw_version]  Current fw version. Download is started only if                   the received version is newer than current. Default: 0 [-h]             This help

An example of the client start:

multifiles # ./mfcln -i 224.5.6.7 -p 1111 -a 1100Waiting for incoming data ...Info message received. sequence:9240 filename:netup.tgz outfilename:netup.tgz.9240 size:5741931 bytesReceiving data ...Filling file to required 5741931 bytes, blocks size is:1048576File filled. Size is:5741931 requested size is:5741931. Starting write intoFirst offset:1379Prophylactic info message received. filename:netup.tgz offset:208380 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:615480 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:1022580 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:1431060 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:1838160 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:2245260 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:2653740 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:3060840 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:3467940 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:3875040 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:4283520 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:4690620 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:5097720 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:5506200 size:5741931 bytes force update:0 fw_version:1104Prophylactic info message received. filename:netup.tgz offset:69000 size:5741931 bytes force update:0 fw_version:1104File 'netup.tgz.9240' received. Checking ...e7a646426d66b5df3318ba09f1ae33d2 - our file md5. size:5741931e7a646426d66b5df3318ba09f1ae33d2 - received md5MD5 check successFile 'netup.tgz.9240' checked. Success.

As it can be seen from the example, the netup.tgz file is being received from the multicast group 224.5.6.7:1111. Current file version is 1100, but version provided by the server is 1104.

Creation of the Firmware

Content of the scripts directory:

multifiles # ls -al scripts/total 16drwxr-xr-x 4 root root   84 Oct 27 13:51 .drwxr-xr-x 5 root root 4096 Oct 27 15:36 ..-rwxr-xr-x 1 root root  115 Oct 27 13:51 createfw.sh-rwxr-xr-x 1 root root  187 Oct 27 13:42 customup.shdrwxr-xr-x 3 root root   35 Oct 27 13:51 firmware-rwxr-xr-x 1 root root  889 Oct 27 13:39 update.sh

createfw.sh script is used for firmware creation. As an example, the script is:

multifiles # cat scripts/createfw.shFWPATH="firmware"FWNAME="mf_fw.tgz"(cd $FWPATH; tar cvfz ../$FWNAME ./*)echo "Firmware created. File: $FWNAME"

As it can be seen, the script archives the content of the firmware folder into the mf_fw.tgz file. This file will be provided over multicast as described above.

Directory scripts/firmware/ contains files required for the update and the script customup.sh that performs required actions on the client device after download. A simple example of such script is given below:

multifiles # cat scripts/firmware/customup.shecho "Running custom upgrade shell script ... Please, wait"echo "Copying new files ... "cp stb_client/bin/stb_client /netup/stb_client/bin/stb_clientecho "Updating config files ... "

As it can be seen, the script copies new files and, if required, updates configuration files.

Working with the Firmware on the Client

An example of the script that organizes receiving and processing the firmware on the client device (the example is provided for an IP STB):

multifiles # cat scripts/update.shMFPATH="/mfupdater"NEPATH="/netup"DSTPATH=$MFPATH/dstSTATFILE="/tmp/mfstat.log"# Show update splash screencat $MFPATH/updatesp.bmp > /dev/fb0 2>/dev/null# get current verif [ -f $NEPATH/version ]; then    export CURVER=`cat $NEPATH/version`else    export CURVER=0fi$MFPATH/mfcln -s $STATFILE -t 3 -i 224.5.6.7 -p 1111 -f $MFPATH/download.bin -a $CURVERif [ $? -eq 0 ]; then    echo "Updating software ... "    # Create dst path    mkdir $DSTPATH 2>/dev/null    # unbzip received file into /netup    cd $DSTPATH    cat $MFPATH/download.bin | tar -xz    # run custom script in received file    cd $DSTPATH    sh customup.sh    # save new version    if [ -f $STATFILE ]; then        cat $STATFILE > $NEPATH/version    fi    # cleanup dst dir if required. You can do this in customup.sh    echo "Update done. New fw version:"    cat $NEPATH/versionfi

As it can be seen from the code above, the script performs the following steps:

• Fetches the current firmware version. Version info is stored in /netup/version
• Starts downloading the firmware from the multicast group
• If the file has been successfully downloaded, unpacks it and runs the customup.sh script contained in the archive. The script preforms required actions (copying, configuration update, etc.)
• The new version data is saved to the file /netup/version. The next update will be performed if a newer firmware version comes from the server or if the "force update" option is set on the server.

Binary Builds

The archive includes ready binary builds for the following platforms:

• SH-4
• Mipsel uclibc
• ARM9
• PPC
• x86

For IP STB Manufacturers

It is suggested to include the client application (mfcln) in the basic IP STB supply. On each IP STB start up it is required to run the application in order to check the presence of the multicast stream with the firmware. The check won't take more than 2 seconds, and if no firmware multicast stream found or an older firmware is provided, an ordinary start-up without any changes proceeds. If a new firmware version was obtained, it is installed and next start-ups may follow a different scheme provided by the installed firmware. This allows for just installed IP STB's to be automatically brought into operation by getting the firmware specific for the current network.