asynchronous poster production

The rendering of the lgm posters is split into two asyncronous parts, because the actual poster generator involves some stuff that no administrator wants to have running on a stable web server. Part 1 is a simple input interface, that does nothing more than writing plain text files entered via a html textarea. This runs on the lgm server. Part 2 synchronizes with the online part, generates posters and uploads new files. This runs on a machine in the our studio.

The script starts by defining some general variables and storing information about the current/local voice files.

  TMPDIR=tmp
  PDFDIR=o/__
  VOICEDIR=o/free/voice
 
  REMOTELIST=http://libregraphicsmeeting.org/2013/r+w/list.php

The current, local number of voice files is stored (VNUMLOCALE=`ls $VOICEDIR/*.txt | wc -l`) and I start a list with 40 random voice files, that will be used together with new files downloaded from the server.

  VNUMLOCALE=`ls $VOICEDIR/*.txt | wc -l`
 
  ls -t $VOICEDIR/*.txt | \
  rev | cut -d "/" -f 1 | rev | \
  shuf -n 40                                         >  q+a.list

To be able to differentiate new from old voice files I do the following: ls $VOICEDIR/*.txt | rev | cut -d "/" -f 1 | rev | sed ':a;N;$!ba;s/\n/|/g' lists all current voice files separated by newlines (:a;N;$!ba;s/\n) and replaces the newlines with a |. This string that is used later on by egrep as a filter (e.g. egrep -v "foo.txt|bar.txt|montana.txt).

  XGREP=`ls $VOICEDIR/*.txt | \
         rev | cut -d "/" -f 1 | rev | \
         sed ':a;N;$!ba;s/\n/|/g'`

For downloading new voice files from the server I use wget. Because directory listing is disabled on the lgm server we use a php file that lists all txt files in the voice directory. I download this list (wget -O $LIST $REMOTELIST) to use it in the next step as input file for wget (wget -r -np -nH --no-directories -N -S -i $LIST).

# -------------------------------------------------------------------------------------- #
# UPDATE QUESTIONS & ANSWERS
# -------------------------------------------------------------------------------------- #
 
  REMOTEDIR=http://www.forkable.eu/generators/r+w/i/free/voice/
  LOCALEDIR=$VOICEDIR
 
  cd $LOCALEDIR
 
  LIST=voices.list
  wget -O $LIST  $REMOTELIST
 
  wget -r -np -nH --no-directories -N -S -i $LIST
 
  cd -
 
  rm $LOCALEDIR/$LIST

After downloading I can calculate the number of remote voice files (which are now local): VNUMREMOTE=`ls $VOICEDIR/*.txt | wc -l`. By filtering out all voice files which have been there before the synchronisation (ls -t o/free/voice/*.txt | rev | cut -d "/" -f 1 | rev | egrep -v "$XGREP") I get a list of all new voice files, which are added to the previously started list. I use this combination of old and new voice files, because I decided to generate also posters from already existing voice files on each synchronisation run (NUM2GENERATE=`expr $VNUMREMOTE - $VNUMLOCALE + 1`).

# -------------------------------------------------------------------------------------- #
 
  VNUMREMOTE=`ls $VOICEDIR/*.txt | wc -l`
 
  ls -t o/free/voice/*.txt |
  rev | cut -d "/" -f 1 | rev | \
  egrep -v "$XGREP"                                 >> q+a.list
 
# -------------------------------------------------------------------------------------- #
 
 NUM2GENERATE=`expr $VNUMREMOTE - $VNUMLOCALE + 1`
 
 if [ $VNUMREMOTE -gt $VNUMLOCALE ]
  then
# -------------------------------------------------------------------------------------- #
 
# -------------------------------------------------------------------------------------- #
# CREATE NEW POSTERS                                                                         # 
# -------------------------------------------------------------------------------------- #
 
  CNT=1 ; while [ $CNT -le $NUM2GENERATE ]; do
 
       SELECTID=`tac q+a.list | \
                 rev | cut -d "_" -f 2 | cut -d "/" -f 1 | rev | \
                 awk ' !x[$0]++' | head -n $CNT | tail -1`
        SELECTQ=`ls -t i/free/voice/*_Q.txt | grep $SELECTID | head -n 1`
 
       ./do.sh $SELECTQ
       CNT=`expr $CNT + 1`
 
    done

Finally I upload the generated stuff on the webserver using ftp.

# -------------------------------------------------------------------------------------- #
# UPLOAD NEW FILES
# -------------------------------------------------------------------------------------- #
 
# GENERATE FTP COMMANDS ---------------------------------------------------------------- #
 
  ACCESS=`cat ~/.forkable/ftp.input`
 
  echo $ACCESS                                                >  ftp.tmp

 
  for PDF in `ls -t ${PDFDIR}/*.pdf | head -n $NUM2UPLOAD`
   do
      echo "put $PDF "`basename $PDF`                         >> ftp.tmp
  done
 
  for GIF in `ls -t ${PDFDIR}/*.gif | head -n $NUM2UPLOAD`
   do
      echo "put $GIF "`basename $GIF`                         >> ftp.tmp
  done
 
  echo "put $ALLHTML "`basename $ALLHTML`                     >> ftp.tmp
  echo "bye"                                                  >> ftp.tmp
 
  mv ftp.tmp ftp.input
 
# UPLOAD VIA FTP ----------------------------------------------------------------------- #
 
  ftp -n forkable.eu < ftp.input
  rm ftp.input
 
# -------------------------------------------------------------------------------------- #

 
 else
  echo "-> NOTHING NEW"
 fi
  rm q+a.list
 
exit 0;

The voice files are saved and versionized in a git repository, because some spelling correction and spam deletion had to be done. Thanks to Ale Rimoldi for taking care for the setup on the lgm server and the github synchronization.