[Labs-l] Bots OGE "qcronsub" submission wrapper

Thu Mar 14 17:09:18 UTC 2013

ok I will try to adapt it to bots

On Thu, Mar 14, 2013 at 6:04 PM, Daniel Schwen <lists at schwen.de> wrote:
> The script on the toolserver is copied below.
> The main thing it does is adding '-hard -l unique' to the arguments
> and passing everything to qsub.
> Then it checks the return status.
>
> #! /bin/bash
> . /sge/GE/default/common/settings.sh
> PATH="/sge/GE/bin/sol-amd64:/sge/GE/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/sbin:/usr/bin:/usr/sbin:/bin:/sbin"
> MYTEMPFILEO=`mktemp -t qcronsub.oXXXX`
> MYTEMPFILEE=`mktemp -t qcronsub.eXXXX`
> ARGS="-hard -l unique"
> VERBOSE="0"
> for ARG in "$@"
>  do
>   if [ "$ARG" = "-verbose" ]
>   then
>     VERBOSE="1"
>   else
>     ARGS="$ARGS $ARG"
>   fi
> done
> /sge/GE/bin/sol-amd64/qsub $ARGS >${MYTEMPFILEO} 2>${MYTEMPFILEE}
> RC="$?"
> if [ ${RC} -eq 1 ]
>  then
>   grep -q -P "Unable to run job: Job .* already queued." ${MYTEMPFILEE}
>   if [ $? -eq 0 ]
>    then
>     RC=0
>   fi
> fi
> if [ -s ${MYTEMPFILEE} ]
>  then
>   if [ ${RC} -ne 0 -o ${VERBOSE} -ne 0 ]
>    then
>     cat ${MYTEMPFILEE} >&2
>    else
>     grep -v -P "(Unable to run job: Job .* already queued.|Exiting.)"
> ${MYTEMPFILEE} >&2
>   fi
> fi
> rm -f ${MYTEMPFILEE}
> if [ -s ${MYTEMPFILEO} ]
>  then
>   if [ ${RC} -ne 0 -o ${VERBOSE} -ne 0 ]
>    then
>     cat ${MYTEMPFILEO}
>    else
>     grep -v -P "Your job(-array)? .* (".*") has been submitted" ${MYTEMPFILEO}
>   fi
> fi
> rm -f ${MYTEMPFILEO}
> exit "${RC}"
>
> On Thu, Mar 14, 2013 at 10:57 AM, Petr Bena <benapetr at gmail.com> wrote:
>> yes we can! Just describe it a bit more and I will make one
>>
>> On Thu, Mar 14, 2013 at 5:47 PM, Daniel Schwen <lists at schwen.de> wrote:
>>> Hi Bots guys,
>>> on the toolserver we have a wrapper to qsub (named qcronsub) which
>>> ensures that only at most one instance of a task is running at any
>>> given time.
>>> This is useful for bots on a schedule, where sporadically a run may
>>> take longer than the time between scheduled runs (and where running
>>> two copies of the same bot could screw things up big time).
>>> Do we have such a wrapper on the bots project? If not could we get it
>>> please? I don't want to start fiddling with lockfiles on my own in
>>> every submission script.
>>> Daniel
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l