On Mon, 12 Jan 2004 11:39:59 -0800 Brion Vibber <brion(a)pobox.com> wrote: On my IBM 80 Go, 7200 trs/mn Before : IO_support = 0 (default 16-bit) After : IO_support = 1 (32-bit) No changes seen.... Shaihulud

On Mon, 12 Jan 2004, Nick Reinking wrote: "Like" is the key word there. 6k$ for a [censored] Apple logo is insane. I've bought similar hardware for less than 1/10th that price. If all you need is a drive shelf, start searching eBay. (I can recommendations if anyone cares.) If you grab a fibre channel shelf (or more than one), I have plenty of drives for the cause (14x18G and 10x9.1G drives gathering dust.) I'd offer an entire Eurologic shelf, but you can find those local in CA easier and faster than shipping across the country. (6 of mine came from Canada :-)) --Ricky

On Mon, 12 Jan 2004, Nick Reinking wrote: Wait a sec, those two hundreds cards total 800GB. Are you sure that wikipedia needs that kind of storage? I thought that current articles are a few gigs, and history a few tens of gigs. I agree, Flash cards are not OK for our needs. A server with lots of RAM is much better. Geoffrin would be really optimal, if it worked :( Someone mentioned $6K for Apple raid storage, but that price is not so high: here at work we have some kind of raid NAS (half a terabyte I think) that went for about $4K. So yeah, NAS can be expensive, a simple internal SCSI raid like our current setup is enough, and with regular backups it can be transferred to another machine in case of need (heck, that's what Brion did multiple times in the last weeks if I am not mistaken, and that's enough proof that it works). Alfio

I think I remember reading about projects fairly similar to this many years ago. However, I don't think they went anywhere. I imagine people realized that if it costs $10k to build a generic machine that can handle these disks, and a huge number of hours to configure it and get it running, then you were probably better off buying a more reliable $15k external disk system. -- Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN

On Mon, 12 Jan 2004, Alfio Puglisi wrote: And it's certainly IDE based. You might have it connected to a SCSI card but the drives are the same unreliable IDE drives everyone throws away every year. SCSI hardware RAID is, strictly speaking, unnecessary. The two opterons in the box are far more powerful than any RAID card. Sure, hardware is nice, but not really necessary with good drives from the get-go (i.e. they're all SCSI and we don't need some magic from a hardware RAID card to make IDE performance acceptable.) I currently have a $250 eurologic fibre channel shelf (FC7 or FC9) with dual power supplies and 7 146G seagate (300$ each) heating my apartment :-) The shelf is easy to find. Wiki would want new drives vs. the OEM pulls I'm using which doubles the cost of the drives. (they aren't easy to find used anyway.) That's just under 1TB. Each drive can stream ~60MB/s. Yes, EACH. (all the way to bus saturation: 2Gb/s) --Ricky PS: I have 6 more of those shelves loaded with 18G drives... old NetApp filer disks.

On Mon, 12 Jan 2004, Nick Reinking wrote: Uhhh, we're already doing that for a brand new server. It'd cost exactly what it's currently costing. Everything in that server is new and thus supported both by the people who sold it to us and the people who made it. A pair of drives from Maxtor purchased 1.5 years ago were replaced in 48hours from the time requested the RMA. (it usually takes a week.) This isn't "penny pinching". The cost savings is enormous. And you're using hardware someone else has burned-in. Of course, you have to has some trust in the source of the gear. If you want spankin' new, go price a shelf from IBM, DEC/Compaq, or Dell. A quick look at Dell shows a SCSI shelf with 14 36G 10k drives, dual 600W power supplies, rails, cables, etc. for ~6k. (ETA 8days.) --Ricky

On Mon, 12 Jan 2004, Ricky Beam wrote: Yes, if one needs throughput, IDE or SCSI will hardly matter, and the former will be cheaper. But it seems that our disk subsystem is hampered by latency - lots of small reads looking for DB indexes, articles, things that got pushed out from the cache, all of them from multiple concurrent threads. Short of a pure ramdisk, SCSI raid is still the leader in this deparments :) Alfio

Nick Reinking wrote: I am not saying running two copies per se will be faster. Just that running a copy from a database file in ramdisk will be much faster. One instance of Mysql whose data file is on the mechanical disk, one instance being a replication slave whose data is on the ram disk. Data served from the ram disk, not the hard disk. I am almost certain multiple instances of MySQL will co-exist on the same machine. If not directly, then through a chrooted environment. Sockets can span chrooted environments using 'mount --bind'.

Jimmy Wales wrote: I understood the server was I/O bound from a recent posting. Server maxed out with low CPU utilisation. The information you and I have differs. You are in a position to have better information on this than I. I certainly don't want to waste time solving a non-existent problem. Do we have a bottleneck? If so, where is it and how do we know?

On Tue, 13 Jan 2004 13:29:51 +0000, Nick Hill wrote: The temporary server doesn't have enough ram, that's all. Geoffrin has 4Gb and worked fine, adding more ram to Ursula would do the same. -- Gabriel Wicke

On Tue, 13 Jan 2004 05:46:15 -0800, Jimmy Wales wrote: Jimbo- how much ram are you going to order for the Squids? I've read that it's possible to address more than 4Gigs though highmem on x86, but only 3 Gigs per process. Squid is a single-thread asynchronous app, so this is a limit. Is there a major price penalty in a single-processor 64bit CPU? Having enough ram for all content on the Squids is a good thing for both speed and reliability- not an immediate problem for now, but it might get one as wikipedia grows. On the other hand x86 Squids could be converted to Apaches later if they are the same kind of machine. The ram issue propably also applies to the DB server, but i guess you'll order 64bit anyway for that. -- Gabriel Wicke

== Access patterns == I did some stats ages ago, showing an approximately Zipfian distribution for page accesses. A bit of calculation shows that for small numbers of articles, this means a small amount of cache will give a large performace boost. However, as the number of articles increases, the vast majority of seldom-accessed articles will start to dominate the behavior of the system for article fetches. Thus, RAM caching will decrease in usefulness over time as the project progresses, unless the RAM cache is close to the size of the entire working set. Nick's suggestion of tuning the filing system page size to the article size is a good idea; it will tend to make the RAM cache which is currently available more effective. I'm rather dubious about some of his other suggestions. Where RAM caching is really important is in the "hot" data such as article timestamps and link tables. These have already been partially addressed by the use of memcached, I believe. These commonly accessed pieces of data should be small enough to keep in RAM all the time, giving a large speedup to the system. == Seek bound performance == Since disk I/O requests are effectively random, the load will be dominated by seek and rotational latency. It will cost very nearly the same to pick 64kbytes off the disk for an article as to get 4 bytes for a timestamp. Using high-performance disks and spreading the database across many RAID spindles should greatly increase performance. I agree with the posters who are arguing for software RAID: it has higher performance than hardware RAID in many cases, and again, we can fine-tune stripe sizes etc. to our application. (Big stripe sizes are a bad idea for random-seek loads, but give better performance for streaming loads). We should also consider kernel 2.6: there are major gains in disk I/O performance in this kernel, and most of the teething troubles are not related to server issues. == Not all disks are equal == Consider buying the disks specifically by access time statistics. In particular, high-performance SCSI disks should greatly out-perform IDE for random seek access patterns, even though their performance may be roughly the same for data streaming. SCSI command tagging will further increase performance, where there is concurrency on a single spindle. See http://www.storagereview.com/php/benchmark/bench_sort.php for some interesting stats: * a Fujitsu MAS3735 has an average read access time of 5.6ms, for a price of $700 for 73 GB. * a Hitachi Deskstar 7K250 has an average read access time of 12.1ms, for a price of $250 for 250 GB * a seagate U6 has an average read access time of 20.0ms, for a price of ??? for 80 GB According to this, if performance is dominated by read access time, the most expensive drive should have almost four times the random-read performance of the cheapest, all else being equal. Using price and performance figures such as those above, we should be able to calculate the best price/performance/storage compromise for this application. == The Google strategy for article caching == Google seem to use a large number of RAM-based cache servers, based on the observation that network access latency on a small network is tiny, but disk latency is large. This does not make any sense for us now: we don't have the resources, unless Google open-source their Google filesystem. For future expansion, it might be cheaper to buy 10 4Gbyte RAM commodity machines than one 40 Gbyte enterprise-class machine, and spread the load across them. Although this would still be costly, the performance of serving data directly from RAM would be very high. -- Neil */ /*

On Tue, 13 Jan 2004, Nick Reinking wrote: Indeed. Those drives are "bastard stepchildren"... it's a SCSI servo with IDE controller electronics tied to a SATA bus via a built-in bridge. It's a f'ing IDE drive. Stay away from it. If you want SATA (read: something cheaper than SCSI), buy real, native SATA drives which only Seagate are currently making. (which exact models, I don't know.) Haven't we already had the "cheap is not the goal" mini-flame? If you want fast and reliable, it's gonna take some green. If you want cheap, it'll be less reliable and slow. I would personally recommend a FC array deployed under LVM2 with XFS riding on top. The array is physically expandable to 127 drives. And the drives are really fast, esp. when you have 100 of them in parallel :-) --Ricky

This is my understanding of how it works: There are two RAID devices in each box. Each device supports up to 7 drives, and can use RAID 0, 1, 3, 5, 0+1, 10, 30, and 50 with these seven disks. Now, if you want to use the entire 14 in one gigantic RAID setup, you can do that with software RAID (you can do RAID 10, 30, or 50 across the two RAID systems). This is not actually terribly optical. It is best to use the two RAID systems separately. If you want a gigantic file system, you can use LVM and pull both RAID systems in (probably running RAID 5). However, how I really think it would be most useful would be if we had two database servers. One would run en, and one would run the international wikis. Since the RAID system has two fibre channel ports, you can give each system its own external RAID hardware subsystem in the future. (so, both could have up to 1.75TB of storage at their disposal). In the near term, just using half the machine (7 disks, just one RAID controller) would still make a huge improvement, I think. It's not the cheapest solution, but it is a lot more reliable than hoping the disks in the servers don't croak, and it gives us the ability to move the database around fairly easily if a server should die (just plug it into a different machine). -- Nick Reinking -- eschewing obfuscation since 1981 -- Minneapolis, MN On Jan 12, 2004, at 2:35 PM, audin(a)okb-1.org wrote:

how about the cost? --- Nick Hill <nick(a)nickhill.co.uk> wrote: __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus

On Mon, 12 Jan 2004, Gabriel Wicke wrote: It seems a good idea. It would eliminate the DB timestamp lookup for each page request. But the purging code must be implemented, and that would take time. Alfio

Gabriel Wicke wrote: Perhaps I will be burned at the stake as a heretic for this, but I am not convinced squid proxies are the answer. The delays in the wikiserver system are caused by waiting for I/O- the time taken for mechanical devices to seek a particular block of data. If the data is being served from a squid cache rather than from a cache on the wiki server, how will this reduce the overall I/O blocking problem? The busiest page data won't substantially add to I/O blocking on the wiki server as it will likely be in memory all the time. The squid proxy is ideal to solve the problem of network load from commonly accessed pages or pages which demand a lot of CPU power to generate but this is not a problem on wikipedia. If Squid proxies are being implemented to increase performance, then they are the right solution to the wrong problem. If they are to increase reliability by adding redundancy - multiple data sources-, they do this to a degree but are far from ideal. The most commonly used pages are going to be in the memory of the database server so these are not costly to serve. The costly pages to serve are those which need disk seeks to serve. The more I/O seek operations a page requires, the more costly it is to serve. The proxy server will need to make a database lookup (for the URL) and, unless the page is in memory rather than on-disk storage, use I/O to reach the fine grained data. The data for each unique URL will be bigger than that held in cache on the database server as it will contain html formatting and other page data. The likelihood of the data being in the memory of a proxy server is lower than the data being in memory of a similarly equipped database server as the data size of the final HTML page will be ~7.5k bigger than that of the database data. If performance is the criteria, I suggest a proxy isn't a good idea. Instead, the memory otherwise used in a proxy would be better utilised caching database data directly. Either as a ramdisk or perhaps a network attached database storage with plenty of solid statememory. From what I have gathered, the cost (limiting factor to performance) is that of delays seeking fine grained data. Either this seek load will need to be spread across many mechanical devices such that the work is not unduly duplicated, or store the fine grained data in solid state storage so that it can be seeked quickly.

On Mon, 12 Jan 2004 16:23:30 -0800, Brion Vibber wrote: Ahem. Doesn't matter, that's what a replacement algo is for. There are much more sophisticated ones than LRU to pick from- both for ram and for disk. If these two servers have some Ram they'll be fine. We could even disallow disk caching all together, only using their Ram. -- Gabriel Wicke

On Mon, 12 Jan 2004 20:16:27 -0800, Brion Vibber wrote: Are there unused cache slot on Ursula? If we could use some of this ram we should be set i guess- Geoffrin also has 4 Gigs and didn't seem to swap badly. Might be imcompatible ram though, don't know. Hm- could we stick some of Carol's ram into this one? -- Gabriel Wicke

On Jan 12, 2004, at 17:01, Gabriel Wicke wrote: ...which would only work as such if we make changes to the wiki which are roughly identical to the changes we'd have to make to the present caching system to avoid hitting the database to confirm timestamps. (ie, explicit purging of cached pages that are no longer valid) Squid may be a great caching system, but for purposes of this discussion it's not really different from what we've got. -- brion vibber (brion @ pobox.com)

On Jan 12, 2004, at 17:48, Gabriel Wicke wrote: This thread is about disk access on the temporary database server. No matter how insanely fast the caching is, it doesn't happen on the database server. Superduperfast squid caching that doesn't touch the database server and kinda speedy php caching that doesn't touch the database server are exactly equal in the eyes of the database server. Lemme put it this way: you're talking about tuning up the engine on a car, when we're trying to discuss how the tires all are flat and the parking brake is stuck on. The car's not going to go very fast in this condition no matter how much the engine is improved. -- brion vibber (brion @ pobox.com)

Gabriel Wicke wrote: Given that popular articles will be in the database memory cache, requests for popular articles should not lead to database HDD seeking. I would expect a Squid proxy be best at serving popular pages and poor at serving less popular pages. So I can't imagine how squid is very helpful at saving HDD seeks. I meant the squid server will need to look up it's own database (in whatever form that may be- filesystem or indexed DBMS) to check if it has a copy of the required data- using the URL as a key. If it has a copy of the data, it will need to either pull it out of memory or from the disk. If not, forward the request then add the page to it's own database. If the squid server needs to pull an article page off the disk, then disk I/O will be required in the same way as it would be required if an uncached piece of data is read from the database by the web server. As I/O is the bottleneck, the squid server is likely to suffer the same problems as the underlying database server. These problems are likely to be bigger as the chunks of fine grained data handled by squid will be larger (compressed fully formed HTML pages) than the fine grained chunks of data (article text) handled by the database server. I am not contending that squid is not a very high performance server. I believe it is a high performance server. I believe it can substantially reduce the bandwidth needed by ISPs to serve web surfers. The issue for wikipedia is how many disk accesses, in total, are needed for each article hit. Wikipedia has millions of discrete pieces of data. Most pieces referenced singularly by a unique URL. Squid will not be able to hold a substantial proportion of these in memory. Squid will be able to hold fewer of these data chunks (articles/ HTML pages) in a given amount of memory than a database server could as the articles stored on the database consist of less data than the article rendered in HTML. For the larger articles, the comressed html page will be smaller, but for most articles, the compressed HTML page will be bigger. (the relative weight of the page HTML is much greater for short articles than for long articles. Compression reduces page size by about half.) I assume viewing an article history page requires several pieces of information leading to multiple seeks per request: If squid were able to serve article histories, then a single I/O on the squid box could save several database seeks on the database server, providing a substantial economy. However, individual page histories are each fairly rare and forcing a squid cache reload on an article history page when an article is updated may be a poor use of resources. I suggest four avenues for investigation: 1) Store articles in the MySQL table in compressed (gzip) format. This will reduce the size of the articles, making them fit more easily into the available cache memory, increasing the chances of a cache hit almost by a factor of two. Perhaps this can be made as a patch to MySQL. 2) Investigate ways of prioritising data cached in memory such that smaller chunks of data have a higher value than larger chunks so that smaller chunks are not flushed according to the basic least recently used algorithm. To reflect the relative cost of reading a small chunk of data from the HDD. 3) If the SQL code underlying wikipedia relies on temporary tables as part of the SQL queries, investigate whether the I/O of writing temporary tables tends to flush data from the disk cache. If so, write temporary tables to ramdisk or other storage which does not cause flushing. More recent versions of MySQL support sub-queries. This may obviate the need for temporary tables. 4) Judicious use of solid state storage. Could dramatically reduce seek times and I/O bottleneck. Some issues to resolve regarding flash memory durability and possible MySQL hotspots. Also cost of mass solid state storage. Might be worthwhile for some wiki data.

At 00:07 13/01/2004 +0000, you wrote: You should not be burnt Agreed One solution is to lower the options available to the user to make the pages more static (unpopular) Probably the only solution apart from splitting the various wikis to more servers Or data replication over more/many db servers. We really do need to spread the load (high power processors are less important than high speed disk. As things are at the moment to cache the db to get full performance we need a server with 50gig ram (we cannot get 4gig working yet). So for the next best thing I believe the suggested hot swap disk array is needed for both performance and reliability (2 off) to make two fast db servers. plus squids apaches and dns round robin etc Dave Caroline aka archivist