|
Official Web Development Group
|
|
|
|
|
|
A place to discuss development of the RT website. Please be sure to visit the Site Bugs forum thread for bug reports.This is the official web dev group where you can discuss suggestions for the site, post bugs, or ask questions about why things work the way they do. We'll do our best to answer your questions and hopefully build a better understand of how things work. |
|
Podcast serving clarification
I appreciate all the suggestions you guys are offering, however I need to clarify something a lot of you have said. Many people have posted "just put the podcast on iTunes and Zune and have people download it from there and you don't have to use your server". That would not help us at all. When a podcast is posted to iTunes or Zune Marketplace, those services still redirect you back to our servers to download the podcast. Those services only act as a gateway and funnel all download traffic back to our server. Even the image you see for our podcast in iTunes is served off of our server.
|
|
|
| |
Recent Site Slowness
You may have noticed the site was super slow last night. Well, it turns out the Drunk Tank has gotten way more popular than we ever anticipated. When we first started hosting it, I threw it on our image server figuring that server could easily handle all the downloads....well, I'll admit, I was wrong. When the podcast comes out that server struggles to serve all user images and the podcast (so sometimes you get error 500 or user avatars won't load Wednesday evenings). Additionally, the podcast is now maxing out our bandwidth when we release it (it was maxed out for 8 hours straight last night starting the second the podcast went up).
I'm working on a solution but I'm not sure what I'm going to do yet. Most likely we will have to begin serving the podcast from a server outside the cabinet that serves the website (so its not using the same bandwidth). Unfortunately I'm not sure where that will be, but I'll figure something out.
|
|
|
| |
Attempted DoS
Someone was prodding us with a DoS this morning. I'm currently setting up countermeasures to make sure this type of DoS is ineffective against us.
|
|
|
| |
This morning
Site was down briefly this morning due to something stupid I did. When we run backups the database typically dumps a copy of itself locally then copies that dump to another server where it is compressed and then copied to an external USB drive. After this process is complete the database deletes its local uncompressed backup of the database. Well, when we were having backup trouble the other day I disabled the step where the database deletes its local uncompressed database dump, so after dumping a few uncompressed databases and not deleting them the hard drive on the database filled up, causing the site to become unresponsive this morning. Took me a while to troubleshoot this one because it was an unusual problem.
|
|
|
| |
Yesterday's Maintenance
I don't know if you remember this or not, but on Friday morning the site was down for a little while. The database server had locked up and had to be rebooted. After the server came back up I noticed that one of the tables was marked as "crashed". That just means a repair has to be run on the table in order to repair it. I started to run the repair on Friday and normally repairs just take two or three minutes but this table kept erroring out so I had to start running a super slow/intensive repair on it. After twenty minutes I gave up and just brought the site back online. The table affected was the forum search table so we just disabled forum search till I had time to run the repair. The forum search table is essentially every post from the forums stripped of html in order to make searching easier.
Well, on Sunday evening I was double checking our backups and realized the site hadn't been backed up successfully since Friday. After some more digging I realized that the backups would not run as long as that table was marked as crashed. So, we decided to offline the site yesterday afternoon to run the repair so that we could get the backups working again. Unfortunately at the time I didn't realize there were about ten million rows in that table, so the repair took WAY longer than anticipated (hence all the downtime). Eventually the repair completed and the backups successfully ran, so now everything is good. Forum search should be coming back online some time this morning.
|
|
|
| |
Emergency Maintenance
I'm about to take the site offline for an hour or two for emergency maintenance before our episode release tonight.
|
|
|
| |
Outage
Last night during its backup the database server decided it needed a break...it does this every couple of months. Had to drive down to the colo and reboot it and everything was fine (well, almost everything, the journal comments table was corrupted so I had to repair it, no biggie).
This makes me think of a story I want to tell. There are only two access keys to the colo, Burnie has one and I have the other. When I flew out to Comic Con last week I realized that both Burnie and I were in San Diego and nobody in Austin had access to the colo. I had to overnight my keys back to the office to Nathan just in case something like this happened while we were out of town. If this had happened with both keys in San Diego we would've experienced downtime of several days. How dumb, right?
|
|
|
| |
Maintenance Complete
Long version: Too long to type.
Short version: One of our servers that hosts images and some shared code for the site had to be rebooted tonight. Several processes had become zombified and I couldn't kill them. The zombie processes were eating up a lot of CPU cycles (aka brains) so I had to reboot the server or else the site would end up going offline.
Maintenance ended up taking a little longer than expected because the server wouldn't power down so I had to unplug it (funny story, I did this once and we lost a file system and that resulted in two days of downtime a couple of years ago). After plugging it back in, the server wouldn't boot up. The server would only sit at a black screen with a blinking cursor in the lower left corner of the screen. I powered down, unplugged a USB drive and reordered the boot sequence and then the system came back online.
tl;dr version:
everything is fine now.
|
|
|
| | | |
|
|
|
|
|
|
|
|
|
|
|
|
|
Formed:
|
1 year ago |
|
Profile views:
|
15,546 |
|
News views:
|
9,176 |
|
|
|
|
|
|
|
|
|
|
|
