On 11/23/06, James Hare <messedrocker(a)gmail.com> wrote: There are a lot of mirrors out there that do not credit us. DB had a lot of problems with his bot. -- geni

Yes, please work on a anti-copyvio bot. I've seen several bad things on the increase (you don't want to see newpages right now) Any tool that automates part of the process is useful. Like someone said earlier: it's good if you make it semi-automatic, to filter out false positives. Ask help of other bot-builders if you need it. Mgm On 11/23/06, Chris Picone <ccool2ax(a)gmail.com> wrote:

On 23/11/06, Chris Picone <ccool2ax(a)gmail.com> wrote: Sure. Try this: http://downlode.org/Code/Perl/Wikipedia/copyviofinder.txt It's a little rough and ready, but seems to work. A couple of random examples: [[USS Dale (DD-290)]] Search phrase: Dale operated with Destroyer Squadrons, Scouting Fleet, on the Atlantic Found: http://www.historycentral.com/navy/destroyer/DaleIIIdd290.html http://www.globalsecurity.org/military/agency/navy/dd-290.htm http://www.hazegray.org/danfs/destroy/dd290txt.htm [[Cincinnati Bengals]] Search phrase: The Cincinnati Bengals are a professional American football team based Found: http://www.jazzsportsnews.com/nfl/teams/64.html http://www.goupstate.com/apps/pbcs.dll/section?category=NEWS&template=w… ati_Bengals http://www.acheapseat.com/cincinnati_bengals_tickets.html http://www.blinkbits.com/bits/viewforum/cincinnati_bengals_bio?f=2552 http://www.frontrowking.com/football/cincinnati_bengals_tickets.html http://www.ticketstogo.com/sport/nfl/Cincinnati-Bengals.html http://www.free-picks.org/nfl/articles/bengals.html http://www.foxboroticketking.com/vscincinnatibengals.html [lots more URLs snipped] Whether the copyvio is an inward or outward bound one in each case is sadly beyond the scope of my programming skills, so I leave that to you. -- Earle Martin http://downlode.org/ http://purl.org/net/earlemartin/

James Hare stated for the record: Absolutely true, but what is also distressing is the number of people (*cough* Trekphiler *cough*) who cannot get their minds around the concept that material explicitly released into the public domain (*cough* DANFS *cough*) is not copyright-reserved and its credited use is not plagiarism. -- Sean Barrett | The only thing we have to fear is fear itself-- sean(a)epoptic.com | nameless, unreasoning, unjustified terror | which paralyzes needed efforts to convert | retreat into advance. --Franklin D. Roosevelt

On 27/11/06, Michiel Sikma <michiel(a)thingmajig.org> wrote: As it stands I've coded in an ignore list to skip certain domains - generally speaking, WP mirrors - one of which is answers.com. -- Earle Martin http://downlode.org/ http://purl.org/net/earlemartin/

On 28/11/06, Steve Bennett <stevagewp(a)gmail.com> wrote: Aha, it looks like you're talking about this: http://meta.wikimedia.org/wiki/Mirror_filter Thanks very much, I'll work it in. -- Earle Martin http://downlode.org/ http://purl.org/net/earlemartin/

Yeah. Since all edits are manual anyways (and mirrors are usually low on Google), the "bot"'s confusion can't hurt. On 11/28/06, Alphax (Wikipedia email) <alphasigmax(a)gmail.com> wrote:

On 12/21/06, Chris Picone <ccool2ax(a)gmail.com> wrote: Cool :) Found any instances of other sites plagiarising us without giving credit? I found one once where a woman was even using photos I'd taken on her site...I wasn't sure whether to be flattered, offended, or just...confused. Steve

On Thu, 21 Dec 2006 15:59:21 +0100, Ryan Wetherell <renardius(a)gmail.com> wrote: Heh, the Register used one of my photos in theyr "Wikipedia Chicken attacked" article (mostly because it happens to be the first image in the article I guess). That's cool though, I released it to the public domain so I can hardly complain about people using it, and in that context it would have been a fair use either way I guess. -- [[:en:User:Sherool]]

I don't know anything about Perl, but I get this foreign error trying to run the script in my ocmmand line: Can't locate HTML/FormatText.pm in @INC (@INC contains: /sw/lib/perl5 /sw/lib/perl5/darwin /System/Library/Perl/5.8.6/darwin-thread-multi-2level /System/Library/Perl/5.8.6 /Library/Perl/5.8.6/darwin-thread-multi-2level /Library/Perl/5.8.6 /Library/Perl /Network/Library/Perl/5.8.6/darwin-thread-multi-2level /Network/Library/Perl/5.8.6 /Network/Library/Perl /System/Library/Perl/Extras/5.8.6/darwin-thread-multi-2level /System/Library/Perl/Extras/5.8.6 /Library/Perl/5.8.1/darwin-thread-multi-2level /Library/Perl/5.8.1 .) at copyviofinder.pl line 37. BEGIN failed--compilation aborted at copyviofinder.pl line 37. hope it's of any use and thanks a lot! Chris On 11/26/06, Chris Picone <ccool2ax(a)gmail.com> wrote:

On 11/24/06, Earle Martin <wikipedia(a)downlode.org> wrote: I don't think this is a programming program -- its a conceptual problem. A good copyvio bot -- one which doesn't waste one's time with false positives or outward copyvios -- would be one which monitors NEW additions and did not try to parse previously existing material. If someone says, "This is new, original text" but it gets Google hits, it is almost certainly copy-and-pasted (whether that makes it officially a copyvio still needs to be decided, but it is a vastly simpler problem than the previous one). Trying to go through the entire database by finding random pages and taking random lines seems extremely hit-and-miss to me, and if you have to worry about mirrors and false positives then I can't see how that would possibly be productive. The odds of finding a copyvio are going to be quite low, and the amount of time needed to sort through them is going to be quite high. Monitoring RC for copyvio seems much simpler by comparison -- if finding previously-existing copyvios is going to be an impossible effort to automate successfully (which I think it is), preventing new copyvios would be comparatively easier. (I started, ages ago, to work on a program which could do things like this, but got bogged down and lacked time. Sigh...) Just my two cents... FF

On 12/21/06, geni <geniice(a)gmail.com> wrote: Daniel Brandt is an obsessive with way too much time on his hands, it seems. Not a practical solution to the overall problem. -Matt

On 12/21/06, James Hare <messedrocker(a)gmail.com> wrote: I fear that level of obsession does not scale. Or at least, I hope it does not. -Matt

geni wrote: [snip] And other databases of copyrighted texts, such as InfoTrac (http://www.gale.com/onefile/) or similar, and things like Google Book Search. -- Neil

On 12/21/06, Neil Harris <usenet(a)tonal.clara.co.uk> wrote: Go ahead: Write the software, make it good, make it scale, make it robust so that you don't have to constantly twiddle with it to keep it working. I have no doubt that Google's ratelimit can be worked out. I promise you that good work done towards these ends will not be work wasted. Make sure that it's sufficently modular that we'll be able to use it to generate queries against other texts sources. The logic for software to do this well is not trivial but certainly not impossible. Working out the right access with Google is also not impossible. Someone just needs to step up an do it.

Do a search in Google; invariably the first reference you are given is Wikipedia. I don't have a clue what you are talking about technically, but I believe any agreement between Wikipedia and Google to improve service would be mutually beneficial. Marc

Most of the copyvios aren't from other encyclopedias. Also, I get a copyvio 1 in 10 times... that is way, way too many. I'd have hundreds of edits if I ran this for hours a day. People don't realize how serious a problem this is... I've been noticing this for months, and I want to give Earle Martin a million barnstars for writing this (it inspired me to start learning Perl so I can write an automated, newpages based one... hint hint) --Chris On 12/23/06, Andrew Gray <shimgray(a)gmail.com> wrote:

On 12/21/06, geni <geniice(a)gmail.com> wrote: Did he do it by using random pages? It strikes me that it would be something most easily done if you downloaded a copy of the database and then ran it off of that systematically (you could filter out short articles while you are at it). FF

On 21/12/06, Fastfission <fastfission(a)gmail.com> wrote: Did you try actually running the program? I guess not. Let's have another demonstration. I just ran the program a few times. On the fifth try it got me this: http://en.wikipedia.org/wiki/Hilal-i-Jurat which is copied wholesale from this copyrighted text: http://faculty.winthrop.edu/haynese/medals/Pakistan/pakistan.htm On the very next try, I got: http://en.wikipedia.org/wiki/A._J._Seymour which was either copied from, or to (either way, without attribution) this website: http://www.triste-le-roi.blogspot.com/ajs_main.html The revision history indicates the person responsible was the maintainer of the website. Six or seven more program runs later, I got: http://en.wikipedia.org/wiki/Abd%C3%BClhak_H%C3%A2mid which contains material copied from: http://www.osmanli700.gen.tr/english/individuals/a19.html My conclusion? First, the basic idea of the program is a sound one. Second, there are thousands more copyvios on the English Wikipedia than most people would ever imagine. By the way, my program has a second use - you can also use it on a specific page if you're suspicious that it contains copyrighted text. -- Earle Martin http://downlode.org/ http://purl.org/net/earlemartin/

Yeah... first I have a dynamic IP so Google's not an issue (I use the https: server to connect to Wikipedia). Also, er... I've encountered one small developmental rpoblem: There aren't any browsers I know of that are very compatible with Applescript (except maybe Safari). I'll keep trying. On 11/23/06, geni <geniice(a)gmail.com> wrote: