[Labs-l] Bots OGE "qcronsub" submission wrapper

Thu Mar 14 17:04:52 UTC 2013

The script on the toolserver is copied below.
The main thing it does is adding '-hard -l unique' to the arguments
and passing everything to qsub.
Then it checks the return status.

#! /bin/bash
. /sge/GE/default/common/settings.sh
PATH="/sge/GE/bin/sol-amd64:/sge/GE/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/sbin:/usr/bin:/usr/sbin:/bin:/sbin"
MYTEMPFILEO=`mktemp -t qcronsub.oXXXX`
MYTEMPFILEE=`mktemp -t qcronsub.eXXXX`
ARGS="-hard -l unique"
VERBOSE="0"
for ARG in "$@"
 do
  if [ "$ARG" = "-verbose" ]
  then
    VERBOSE="1"
  else
    ARGS="$ARGS $ARG"
  fi
done
/sge/GE/bin/sol-amd64/qsub $ARGS >${MYTEMPFILEO} 2>${MYTEMPFILEE}
RC="$?"
if [ ${RC} -eq 1 ]
 then
  grep -q -P "Unable to run job: Job .* already queued." ${MYTEMPFILEE}
  if [ $? -eq 0 ]
   then
    RC=0
  fi
fi
if [ -s ${MYTEMPFILEE} ]
 then
  if [ ${RC} -ne 0 -o ${VERBOSE} -ne 0 ]
   then
    cat ${MYTEMPFILEE} >&2
   else
    grep -v -P "(Unable to run job: Job .* already queued.|Exiting.)"
${MYTEMPFILEE} >&2
  fi
fi
rm -f ${MYTEMPFILEE}
if [ -s ${MYTEMPFILEO} ]
 then
  if [ ${RC} -ne 0 -o ${VERBOSE} -ne 0 ]
   then
    cat ${MYTEMPFILEO}
   else
    grep -v -P "Your job(-array)? .* (".*") has been submitted" ${MYTEMPFILEO}
  fi
fi
rm -f ${MYTEMPFILEO}
exit "${RC}"

On Thu, Mar 14, 2013 at 10:57 AM, Petr Bena <benapetr at gmail.com> wrote:
> yes we can! Just describe it a bit more and I will make one
>
> On Thu, Mar 14, 2013 at 5:47 PM, Daniel Schwen <lists at schwen.de> wrote:
>> Hi Bots guys,
>> on the toolserver we have a wrapper to qsub (named qcronsub) which
>> ensures that only at most one instance of a task is running at any
>> given time.
>> This is useful for bots on a schedule, where sporadically a run may
>> take longer than the time between scheduled runs (and where running
>> two copies of the same bot could screw things up big time).
>> Do we have such a wrapper on the bots project? If not could we get it
>> please? I don't want to start fiddling with lockfiles on my own in
>> every submission script.
>> Daniel
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l