Adam Director of Technology
Adam Director of Technology
1 week ago
Lesson learned with the Feedback thread;
Forums are a terrible place for Feedback. Swapping to a general group post instead.
To submit feedback; Comment on this post. If possible; look through the other comments first and if your bug exists; mod +1 ditto that issue instead, which will allow us to easily find the worst offending issues. By using comments instead of forums; it will allow ongoing conversation (replies) to actually nest instead of continuing our current situation of having 50 pages of posts that no one is reading, and then for every 1 actual bug report, 5 people reply, which was never the intended purpose.
- Please create ONE comment per feedback / bug report. This makes it a lot easier for us to delete things when they've been fixed.
- If a bug report; explain as closely as possible what you did. Extra points for attaching browser / os information if it could be related to that.
- We might not be able to address everything, but this gives us a single place to look for feedback and suggestions.
- Make sure to use the mod system to indicate what is the most & least important feedback, so I can set priorities.
- Also, shitty bug reports will be deleted with irreverence. ♥
Top Issues Right Now;
- Episodes not playing with a cross domain issue.
- Forums not loading properly with a "whoops" error.
- Notifications firing to strange destinations (ie /index.php/index.php/index.php/forums/ect)
- Notifications not going to paginated content.
- Notification counting is off.
- Global Login seems to have broken once we enabled SSL requirement.
- Intermittent Image Uploading Issues.
Adam Director of Technology
1 week ago
Hey Everyone -- I posted a HUGE 1 Month website update thread HERE.
It's pretty enormous so i'm not going to promote it to the home page, but if you're interested in checking out the latest info on the site, what we're working on, what we've been focused on in the last 4 weeks, check it out!
Adam Director of Technology
1 week ago
TLDR; We fixed some stuff, and we've got more stuff to fix.
Tonight at midnight will officially mark 1 Month since the new website, and with RTX mixed in there, the fact that we are where we are today is pretty monumental. Here's a brief overview of the last month;
I can't speak highly enough of my team; @tim @Curryshoe @Dave @natalya @CiQuat -- through the fire and the flames we've put out a LOT of fires this month, especially when you consider RTX was smack dab in the middle of everything.
A Look Back;
Day 1; Melting Database Servers, Scale & Optimization
When we flipped the switch, we immediately had hundreds of thousands of queries running hitting our databases and absolutely murdering them. Immediately redlining all systems, causing cascading issues across all platforms. After about 6 hours of "database on fire" load, we saw the first glimmer of hope; 95% CPU utilization. Optimization continued every day for the last 30 days. I can proudly say that at time of writing this entry; we're sitting at 15% CPU utilization, after countless optimizations to queries, replacing the database hardware twice, and even replacing our entire previous RDBMS.
Day 3; Then The Spam Came
Twas the night before christma--- wait wrong story. Either way, some big fat guy decided to climb into our house and threw shit all over the place. Suddenly the site was borderline unusable, first because the spam botnet sat over our site for several weeks just hammering the database with "FREE MOVIE DOWNLOAD" bullshit (again, ramping up our CPU back to 100%, and second because the "FREE MOVIE DOWNLOAD" overflowed the feeds, groups, games, profiles, posts, forum topics, forum posts... Shit had hit the proverbial fan, and had been thrown all over the fucking place. Over about two weeks, we wrote an entire anti-spam stack for RoosterTeeth. All incoming content is now inspected via our inspection engine, which checks for common spam practices and keywords. If it looks like spam, our Staff & Site Admins are immediately alerted to it's presence so they can deal with it. Also during this process; We got Site Admins back off the ground. They're now able to banhammer as they should. We had to delete over a MILLION articles of spam on the site. Fuck.
The Great Wall of Notification & Sign on issues
So, our original spec for notifications worked. Problem is, they worked a little too well, and in general people don't want a notification for _everything_; just some stuff. Day 5 notification grouping went in, which brought notifications from 650 notifications down to 15, a much more manageable number. Then we found that often times notifications would take you to a different brand, and by thus log you out. I'm proud to say that we now have Global Login; so when you login to one brand, and move to one of the other brands, it will auto-detect and ask you to refresh the page.
Notifications still have quite a ways to go before I would say they are "complete" -- we'll be spinning them out into their own microservice here shortly, which should increase page load time by about 30-40%, while also allowing them to be FAR more fine tuned in terms of what, when, and how, you get notified of things here on RoosterTeeth.
"No Playable Sources Found" and Buffer Hell.
Good lord. So. As we all know; we used to be hosted on Blip. Blip managed all our videos and our entire mezzanine for RoosterTeeth. They were our catalog of videos, and now that they're officially gone, RoosterTeeth had to take over that entire job. As it turns out, migrating 11000+ videos from one system to another completely different system left a LOT of lingering edge cases that for the first few weeks, caused a lot of videos not to play properly. "No Playable Sources Found" As of this morning, I believe our library to be roughly 97-98% operational, and this will continue to grow until all videos are playable. There's a lot of the really old videos that don't have a proper encode and we'll have to pull from our backups and upload a new source file. Soon we'll be rolling out a "flag" feature, which will allow the community to flag videos that do not play properly, so myself or one of our video team can review, and decide the appropriate course of action.
On top of fixing the linking issues to our videos -- The Player. DAT BUFFER We've moved to a more modern version of the player, and despite it still not being where I want it, it works noticeably better. Lots and LOTS of improvements will be coming to this over the next few weeks, as it's kind of the bread and butter of RT. There's still some lingering issues on Safari (not allowing us to control quality) and a few mobile hiccups that are currently among our top priorities.
As of this week; RT is now 100% SSL, with the exception of our CDN (coming soon), and third-party content (I'm not able to control if someone embeds a http image instead of the https. This is something that isn't likely to effect much, however feel safe in that the communication between your computer, and the RoosterTeeth network is now secure to sniffing.
Whoops; Something Went Wrong.
This one is a two-part and an ongoing issue. A large percentage of these issues currently exist because of the spam attack. Without getting too involved with the actual problem, When we banhammer a user, you would think that the "delete" listener events would get fired for child items, however, they are not. Because of that, lots of orphaned data currently exists around the site, and much to our amazement, that's "generally accepted behavior" for the framework we are using. This became a pretty monumental problem going forward, and while we're chewing through edge cases as quickly as possible, we should be able to return to normal viewing soon.
A LOOK FORWARD;
Top priorities for Development right now.
- Whoops; Something Went Wrong error is far too pervasive. Most of it is orphaned records continuing to be cleaned up.
- Notifications still aren't quite where they need to be.
- Migrations; This is actually proving to be quite the bitch. Every time we run the message migration, our systems crash because you guys have posted SO MANY private messages to each other. We're continuing to look for ways to optimize this process to get you your private messages from years past.
- Chat Updates. Yeah, Chat. We have a rough version in right now, but we need some better settings for chat as well as giving it a bit more presence on the site.
- Continuing to resolve Player & Playback issues.
- Fix sponsorship branding. Right now it's hard to understand what you "get" with sponsorship. This obviously needs to be fixed.
- Further fleshing out the Sponsorship control panel, allowing for changing billing information, viewing receipts, better links to the latest enjoyable sponsor-only content, ect.
- Going through the Feedback thread, Cleaning up fixed issues, and finding the biggest issues currently effecting the community and starting the process of knocking those out.
- User Experience Updates. Holy dicks there's so many little tweaks that need to get done. Just a lot of polish as the dust settles.
The Full Change Log;
I can't quite remember where in this list my last Change log post stopped, so for sake of being thorough, I'm just going to cover the change log since Launch. This isn't quite an exhaustive list, but rather the direct issues closed in our project management system. There were plenty more bugs fixed that didn't go through "official channels"
- Feature: Payment Listener. We can now track when a user changes settings on their cards.
- Feature: Notification display is now grouped.
- Feature: Fixed careers pages (still needs some UX love, but they atleast work)
- Feature: PayPal IPN listener for existing sponsors to continue being a sponsor.
- Feature: Added a "Contact Us" page. lol
- Feature: RT Anti-spam Stack.
- Feature: Global Login
- Feature: JWPlayer 7 Implementation.
- Improvement: JWPlayer Google Analytics tracking updates.
- Improvement: Added a redirect to /my/profile and /members for those with bookmarks.
- Improvement: Community Dashboard icons and buttons were completely reworked.
- Improvement: Updated admin panel for show management to include better image uploading.
- Improvement: "Live" Live-streams red bar was not obvious enough. It's now very obvious.
- Improvement: Search implemented, created as a component so it can be dropped anywhere.
- Improvement: Resumes & Cover letters can now be submitted to a job position.
- Improvement: Improved a mirad of query performance, most going from 3-5 second queries down to 0.04 seconds.
- Improvement: Live Streams will now display the sponsor lock if a non sponsor tries to visit them, instead of just redirecting the user to sponsorship.
- Improvement: Introduced a core controller library for uploading items, allowing us to drop it in nearly any location.
- Improvement: Fixed an issue that caused images to go far beyond the width borders on a post.
- Improvement: Games & Groups are now searchable.
- Improvement: Launched an entirely new Admin interface for our video uploading team.
- Improvement: Watched Statuses needed styling.
- Improvement: Slugs for episodes were using an episode number instead of title, making some things get REALLY strange with ordering.
- Improvement: Moved admin panel for chat.
- Improvement: Uploading an episode needed some kind of progress bar.
- Improvement: Dismissed notifications now link to the content that was dismissed.
- Improvement: Applied a series of improvements to the reset password flow.
- Improvement: Old Journal links now 301 redirect.
- Improvement: Old episode links now 301 redirect.
- Improvement: Old podcast links now 301 redirect.
- Improvement: Slugs were reviewed for duplicate episodes overriding each other.
- Improvement: Friends whom have been banned or removed no longer break relating profiles.
- Improvement: Journals on the front page looked really silly with their images and text not blocking
- Improvement: Default timezone is now CST, unless a user has specifically changed their timezone.
- Improvement: Private groups now show up to members, but with a lock symbol.
- Improvement: Accounts now require verification to submit content to RoosterTeeth's network.
- Improvement: Banning an IP no longer shits the bed if the IP has already been banned.
- Improvement: Sponsorship now has a "disable auto renew" button right in your sponsor panel.
- Improvement: Akismet antispam tactics are now used on RT as well.
- Improvement: clicking See all live streams in the mega menu no longer directs a user to the "livestream" user.
- Improvement: You can now see Author name & Link on post pages in mobile browsers.
- Improvement: We now auto-censor content that akismet says is spammy.
- Improvement: Sponsorship Backend for Sponsor Staff.
- Improvement: Image Uploader now sets a default cache lifetime on images to help improve load times on profiles.
- Improvement: Launched the Know's new logo.
- Improvement: Added an admin panel to allow admins to see censored content.
- Improvement: Performed UAC audit, ensuring roles had proper permissions globally.
- Improvement: Added manifest upload time to admin panel so we can replace videos as necessary if ever needed.
- Improvement: Changed out a bunch of our ad tags with better ones.
- Improvement: URL Helper functions now have an SSL override.
- Bugfix: API is now properly pulling videos based on time.
- Bugfix: Moved a forum query from 8 seconds load time down to 0.05seconds. Performance abounds.
- Bugfix: MAJOR bugfix for videos, fixed about 2000-2500 videos on RoosterTeeth.
- Bugfix: Permissible wasn't working properly.
- Bugfix: If sponsor deletes their account from roosterTeeth, it now updates billing information and stops payment as well.
- Bugfix: Users on a Trial, couldn't cancel their trial.
- Bugfix: New Episodes for each brand are now promoted to frontpage by default.
- Bugfix: Fixed broken image on 404 page.
- Bugfix: Fixed a lot of lingering issues due to banhammer.
- Bugfix: We now flush related caches when a user deletes content or is banhammered.
- Bugfix: Careers page no longer requires an RT account.
- Bugfix: Careers page admin panel can now see closed positions.
- Bugfix: Careers page doesn't require a user to insert a RT account, even if they dont have one.
- Bugfix: Careers page admin panel got some increased view size to help cope with the sheer number of resumes.
- Bugfix: Careers page posted jobs no longer paginate, all positions are available on that careers page.
- Bugfix: Moved a lot of links to protocol independant links, so they float between http and https. (not that this matters now that we enforce SSL.)
- Bugfix: Lots of bugfixes relating to rogue whoops! error pages.
- Bugfix: A Mirad of career page related bugfixes, including making it work in the first place.
- Bugfix: Fixed a security bug where private posts where less than private.
- Bugfix: Password resetting problems globally.
- Bugfix: Fixed a security bug where private forums were less than private.
- Bugfix: Found a bug that caused forums to simply not exist.
- Bugfix: Subscription/sponsorship page was completely useless. It's now slightly less useless.
- Bugfix: the www domains were not doing proper domain matching, causing RT stuff to show up on www.ah
- Bugfix: When an admin banhammered a user; the entire forum whoopsed.
- Bugfix: Store Wishlists were broken. They're now not broken.
- Bugfix: fixed a bug that caused our deployment scripts to break the entire god damn site.
- Bugfix: User Messaging was FUBAR.
- Bugfix: Subscriptions where creating an unusally high amount of Whoops errors.
- Bugfix: Notifications are created per-domain, these should be relative.
- Bugfix: Forum posts now return you to where you were, instead of moving you back to the front page.
- Bugfix: Fixed a few issues relating to unpublished videos going out before intended.
- Bugfix: Fixed a bug where the friend accept page was limited to 10 requests.
- Bugfix: We now strip extra information from playable URL's for admins.
- Bugfix: We can no longer submit an empty youtube URL to a video for admins.
- Bugfix: Spaces in HLS names broke mobile playback completely.
Adam Director of Technology
3 weeks ago
Adam Director of Technology
1 month ago
We had a site anti-spam system go haywire earlier today, and while it did a fantastic job cleaning up some of the spam -- it also had an alarmingly high rate of false positives, and a good amount of normal users were caught up in the banwave.
We've rolled back the ban wave and are looking at tuning the anti-spam systems over the next few days.
On the upside; Our new backup system for the new site is fully tested and works very well, considering... In the future I plan on doing system-wide snapshots every 6 hours instead of every 24. In the case we DO have to roll back ever again, it should sting less so than today.
Edit: All sponsors that went missing due to the roll back should now be restored.
13+ Year MMORPG Veteran. Twitch | Twitter | YouTube
Beta Participated: Wildstar, ESO, SW:TOR, RIFT,EQ2:SF, EQ2:TSO, AoC, STO, GW, RO
Veteran of: SWTOR, AOC (again), STO, EQ2, LOTRO, WAR, AoC, Van, DAoC, CoH, WOW, GW, Lin2, SWG, Lin1, AO, RO, EQ, UO, Medievia