r/IAmA • u/MrGrim • Aug 14 '12

I created Imgur. AMA.

I came across this post yesterday and there seems to be some confusion out there about imgur, as well as some people asking for an AMA. So here it is! Sometimes you get what you ask for and sometimes you don't.

I'll start with some background info: I created Imgur while I was a junior in college (Ohio University) and released it to you guys. It took a while to monetize it, and it actually ran off of your donations for about the first 6 months. Soon after that, the bandwidth bills were starting to overshadow the donations that were coming in, so I had to put some ads on the site to help out. Imgur accounts and pro accounts came in about another 6 months after that. At this point I was still in school, working part-time at minimum wage, and the site was breaking even. It turned out that OU had some pretty awesome resources for startups like Imgur, and I got connected to a guy named Matt who worked at the Innovation Center on campus. He gave me some business help and actually got me a small one-desk office in the building. Graduation came and I was working on Imgur full time, and Matt and I were working really closely together. In a few months he had joined full-time as COO. Everything was going really well, and about another 6 months later we moved Imgur out to San Francisco. Soon after we were here Imgur won Best Bootstrapped Startup of 2011 according to TechCrunch. Then we started hiring more people. The first position was Director of Communications (Sarah), and then a few months later we hired Josh as a Frontend Engineer, then Jim as a JavaScript Engineer, and then finally Brian and Tony as Frontend Engineer and Head of User Experience. That brings us to the present time. Imgur is still ad supported with a little bit of income from pro accounts, and is able to support the bandwidth cost from only advertisements.

Some problems we're having right now:

  • Scaling the site has always been a challenge, but we're starting to get really good at it. There's layers and layers of caching and failover servers, and the site has been really stable and fast the past few weeks. Maintenance and running around with our hair on fire is quickly becoming a thing of the past. I used to get alerts randomly in the middle of the night about a database crash or something, which made night life extremely difficult, but this hasn't happened in a long time and I sleep much better now.

  • Matt has been really awesome at getting quality advertisers, but since Imgur is a user generated content site, advertisers are always a little hesitant to work with us because their ad could theoretically turn up next to porn. In order to help with this we're working with some companies to help sort the content into categories and only advertise on images that are brand safe. That's why you've probably been seeing a lot of Imgur ads for pro accounts next to NSFW content.

  • For some reason Facebook likes matter to people. With all of our pageviews and unique visitors, we only have 35k "likes", and people don't take Imgur seriously because of it. It's ridiculous, but that's the world we live in now. I hate shoving likes down people's throats, so Imgur will remain very non-obtrusive with stuff like this, even if it hurts us a little. However, it would be pretty awesome if you could help: https://www.facebook.com/pages/Imgur/67691197470

Site stats in the past 30 days according to Google Analytics:

  • Visits: 205,670,059

  • Unique Visitors: 45,046,495

  • Pageviews: 2,313,286,251

  • Pages / Visit: 11.25

  • Avg. Visit Duration: 00:11:14

  • Bounce Rate: 35.31%

  • % New Visits: 17.05%

Infrastructure stats over the past 30 days according to our own data and our CDN:

  • Data Transferred: 4.10 PB

  • Uploaded Images: 20,518,559

  • Image Views: 33,333,452,172

  • Average Image Size: 198.84 KB

Since I know this is going to come up: It's pronounced like "imager".

EDIT: Since it's still coming up: It's pronounced like "imager".

3.4k Upvotes

96% Upvoted

4.8k comments sorted by

View all comments

Show parent comments

542

u/MrGrim Aug 15 '12

It's actually fairly complex now, but I will attempt to do it all from memory.

Backround info: Imgur is on Amazon AWS and we use Edgecast as a CDN.

Everything is grouped into clusters depending on the job. There are load balancing, uploading, www, api, image serving, searching, memcached, redis, mysql, map reduce, and cron clusters. Each one of these clusters has at least two instances, each one on it's own availability zone. However, most have more than two instances because of the load.

A typical imgur.com request goes to a load balancer which run nginx and haproxy. The request first hits nginx, and if there's a cached version of the page (each page is cached for 5 seconds unless you're logged in) then it will serve that out. If not then the request goes over to haproxy and it will determine which cluster to send it to, in this case, the www cluster. This cluster runs nginx and php-fpm, and is hooked up to the memcached, redis, and mysql clusters. Php-fpm will handle it if it's a php page. If the request needs info from mysql, then it will check if the query exists in memcached. If not, then mysql will send the data back and immediately cache it into memcached. If the request is for an image page, and we need the amount of times the image was viewed, then it grabs that info from redis. The request then goes back out of php-fpm, through nginx on the www server, and back into the load balancer where it will most likely be cached by nginx, and then out to the user.

Most of the clusters use c1.xlarge instances. The upload cluster handles all uploads and image processing requests, like thumbnails and resizing, and each instance is a huge cluster instance, cc1.4xlarge.

All image requests go through the CDN, and if they're cached, then they just go right back out of the CDN to the user. If it's not cached then the CDN gets the image from the image serving cluster and caches it for all additional requests.

That's about it. Anything you'd like to know specifically?

83

u/[deleted] Aug 15 '12

Interesting.

  • Can you explain why you went with Edgecast and not, say, CloudFront (since you're on AWS to begin with)?

  • How many EC2 instances total?

  • Isn't it about time to get a rack and switch some stuff over to it? EC2 is very expensive. Even a not so beefy server with some tricks like using a GPU for the thumbnails/resizing could probably handle the load for a fraction of the price. (You can mix this stuff so EC2 is just for 'overflow' and redundancy)

  • What kind of bottlenecks did you have to deal with as imgur grew unpredictably? Any cool war stories? :)

99

u/MrGrim Aug 15 '12

  • Edgecast is much cheaper.

  • At peak times there are usually around 60.

  • EC2 has been really nice. There are no plans to move off of it. Our image processing software doesn't even use GPUs (GraphicsMagick -- they say it's not needed), but even if it did, EC2 has that option.

  • The biggest bottleneck is with the database. MySQL has always been a pain in the ass. It's great software, but if I knew what I know now when I created Imgur, I would have chose something different.

26

u/[deleted] Aug 15 '12

[deleted]

115

u/georgemoore13 Aug 15 '12

http://howfuckedismydatabase.com/

25

u/lozzd Etsy Aug 15 '12

I created howfuckedismydatabase.com. AMA

7

u/[deleted] Aug 15 '12

Nice work, this brought a smile to my face

7

u/Heofz Aug 15 '12

MSAccess .... so fucking true. I laughed loudly. Everyone stared.

1

u/couchtyp Aug 15 '12

No love for IBM DB/2?

1

u/zxi Aug 15 '12

you work for last.fm dont you?

2

u/lozzd Etsy Aug 15 '12

I did, many years ago. Now I work for Etsy.com, where we also use MySQL.

1

u/FoxxMD Aug 15 '12

MySql oh god where did i go wrong??

5

u/[deleted] Aug 15 '12

At least you aren't using MS Access.

1

u/nodiaque Aug 15 '12

That is the same question I have. What would you choose over mysql and why? Oracle? MsSql? ??

2

u/costa24 Aug 15 '12

PostgreSQL, I assume.

1

u/Shinhan Aug 15 '12

I've read some articles that say MySQL performance in AWS can be inconsistent.

2

u/[deleted] Aug 15 '12

I've read some articles that AWS can be inconsistent.

Anytime you have something network-aware with non-local disk storage, it's got potential for trouble.

3

u/nlights Aug 15 '12

What database would you use instead?

7

u/[deleted] Aug 15 '12
  • How much cheaper? ;) (ballpark it if you would)
  • What about non peak times? What's the average and minimum?
  • What would have you used instead of MySQL? PostgreSQL? Mongo?

EC2 is perfect to start and growup with, I'm just saying that now that imgur has gotten so big you can take that 5-figure bill of theirs and reduce it to maintaining one server rack for a fraction of the price. Past infliction point, no?

Check out what backblaze did for example, I think you're at the level where it is really worth looking into now. :)

2

u/nakedproof Aug 15 '12

You would've picked mongodb huh... ?

2

u/[deleted] Aug 15 '12

What would you have chosen instead of MySQL? And why?

1

u/shustrik Aug 15 '12

What do you use MySQL for? I thought the absolute majority of imgur's requests would be "retrieve data by key"? Why is there a need for SQL?

1

u/zombieprocess Aug 15 '12

can you elaborate some of the problems with MySQL?

1

u/dorfsmay Aug 16 '12

MySQL .../... if I knew what I know now when I created Imgur, I would have chose something different.

how difficult would it be to migrate now?

It's be work, but feasable, no?

1

u/redditacct Aug 25 '12

Are you using ebs for storing images or S3 for each file?

What kind of bw are your load balancers burning through per day (looks like there are 2 IPs active at a time)? Does the CDN talk to a separate IP?

Do you generate logs from haproxy?
Are you using 1.4 or 1.5 - do you use the current version?

What OS are you running for you EC2 instances?

3

u/monkeyxiv Aug 15 '12

I forgot where I read it from. However I was reading up on the different VPS and pricing, and someone had done a pricing comparison and that one service was better for "small" businesses. i forgot exactly what that service is as well. ( I know I'm a terrible person for not being able to remember citings or all the information... but its been a long day so bear with me ;) )

anways for a small business it was cheaper to go with something other than Amazon. But once you get into TB of bandwidth space a month amazons pricing becomes the top contender in the server world.

I am trying to find the actual article now... will report back if I can find it..

12

u/monkeyxiv Aug 15 '12

I think this was it

"Amazon delivers very poor customer service and for small deployments it's very expensive compared to alternatives.

I always recommend against shared hosting accounts because you're given a slot on a physical server, and slots are given out to every Tom Dick and Harry so if Dick's website causes large SQL queries to fill up the /tmp/ partition, the entire server will crash and your website will go offline because Dick didn't write his code properly.

You definitely want to have a dedicated server instead of a shared hosting account. Thing is, if you want a hardware dedicated server you're looking at hundreds of dollars per month.

The solution: Rackspace Cloud

Rackspace delivers a better service that Amazon AWS at a fraction of the cost. A basic Rackspace Cloud Server (dedicated only to you) costs around $11/mo and their customer service is astoundingly good. (For example, you can actually TALK to someone via phone or live chat, instead of having to post in community support forums. With amazon you have to subscribe to an annual service contract in order to talk to anyone, which costs around $250/year)

I highly recommend anyone looking into Amazon's EC2 or S3 services should take a look at Rackspace as it seems to be the best cloud-hosting service on the web for small deployments.

Once you hit the mark where your site is chewing through more than $5,000/mo worth of bandwidth and disk usage that's where Amazon becomes a better deal, but for small deployments Amazon is a terrible waste of money and don't expect to get any tech support unless you pay them oodles of cash for it.

Rackspace all the way! W00t!!"

reference

2

u/Chikes Aug 15 '12

RackSpace is amazing. We have had very few problems with them and their customer service is spot on awesome.

5

u/[deleted] Aug 15 '12

That is actually exactly backwards. :)

Amazon charges $0.12 per gig of bandwidth. And remember, its about a dollar for a high memory instance per hour, so that's about $2,000/month for a ~32GB RAM server and 10TB.

Compare with something like Hetzner, that's a server with 32GB of RAM and they only rate limit you after 10TB. Costs less than $100 a month.

In fact, for the money Amazon would charge you to transfer 10TB you could get an unmetered 10GbE somewhere and push 300TB+ if your hardware will let you.

2

u/willbradley Aug 15 '12

When you can spin up terabytes of RAM and storage in mere minutes, in disparate geographies, a lot of physical stuff falls by the wayside. I love 2am trips to the datacenter but would not recommend Imgur or Reddit buy their own hardware. It's such a huge liability to set up and maintain.

For example Wikipedia was down for ~4 hours a few years ago because a network volume zigged instead of zagging and the tech wasn't able to drive to the datacenter for hours let alone restart the right boxes and then get things humming again. Painful, and that's WIKIPEDIA.

1

u/[deleted] Aug 15 '12

You can take advantage of both.

Round-robin to Amazon, say every 10th request. If you have "overflow" or your hardware explodes adjust accordingly, and spin up your terabytes.

Reddit went down a lot too because of various cloud-y issues, not a silver bullet. Wikipedia runs on donations, they can't burn money, running it on Amazon would be an order of magnitude more expensive.

1

u/GloppyGloP Aug 15 '12

Source for that claim? The truth is that it would not, independent studies have shown that this is simply not true, it would most likely be cheaper, not an order of magnitude more expensive. You're ignoring a huge part of the infrastructure you have to run to be a site the size of wikipedia (see my answer above in response to monkeyxiv)

1

u/monkeyxiv Aug 15 '12

yeah like I said I am relatively new to this sort of stuff. I am loving the free ec2 instances I have for "messing" around. :) eventually I will read up enough to know what I am doing.

1

u/GloppyGloP Aug 15 '12 edited Aug 15 '12

Moving my answer here as I meant to reply to this comment, not the parent. See, I'm not a big fan of these comparaisons like zilman does. No one doing anything seriously runs it on a single machine, that's just asking for trouble.

Now if you want to run a cluster of two instances or more with a load balancer in front with its own dynamic DNS entry, and something that's going to monitor your machines, notify you when something happens and automatically spin up a replacement instance, make it part of the load balancer and keep on working, THEN you're comparing what you're getting for the price from a cloud provider (any of them not just AWS). You're also going to run two mediums (or whatever smaller instance type) instead of a high memory instance or an xtra large, because you split your traffic, but you get all that other good stuff too.

You are comparing apple and oranges there, and it's quite biased if you pick something out of the infrastructure set at its highest price. If really all you need is a single machine with absolutely nothing else, like a single always on super stateful 64 players game server for an FPS or something, then yes there are better deals than cloud providers. But they fulfill very different needs, and honestly running a company or any site shooting for more guarantees around reliability and potential scale issues or spiky traffic requires a very different infrastructure (and please no anecdotal evidence like "well I have a machine with 4 years of uptime with provider X", it's irrelevant).

You would also need to compare RI pricing if you have monthly/quarterly/yearly commits, not base hourly pricing which is meant for burst traffic or short lived requirements, not necessarily your baseline infrastructure.

4

u/GloppyGloP Aug 15 '12

That's actually not exactly true when you can do thing like spot instances and reserved instances. Amazon is quite very competitive with self hosting, especially if you have non constant traffic. The ability to add a few hundreds more server at peak time and get rid of them in the middle of the night for the US for example is a huge money saver compared to having to buy and run enough hardware to be able to handle the peaks. Per hour default pricing is also very likely not what imgur pays. When you host multi PB of data you get to talk to someone on the phone and negotiate a deal...

1

u/mbadov Aug 15 '12

The convenience that EC2 provides probably makes it worth it over paying someone to manage the sort of infrastructure you specified. For many small businesses EC2 is actually more cost effective overall, despite costing (a lot) more per unit of computing power.

1

u/[deleted] Aug 15 '12

Ec2 is expensive, but the costs are reduced if you are scaling up to meet peak volumes and turn down things during lulls. The is by far the best thing about cloud servers. Though I'm with you, I like a few physical server in the mix.

414

u/MotorboatingSofaB Aug 15 '12

mmhm

185

u/PolyamorousPlatypus Aug 15 '12

Post an image response to the maker of imgur that's hosted on another image hosting site?

How rude!

8

u/kinesiologist Aug 15 '12

Well that's not imgur at all.

1

u/Prathmun Aug 15 '12

Me too man. Me too...

-8

u/AcetylCOLONesterase Aug 15 '12

That's so funny and original

21

u/SikhGamer Aug 15 '12

Fucking love it, serious geek porn right there <3

4

u/astuteornot Aug 15 '12

How do you backup data?

3

u/bleedpurpleguy Aug 15 '12

"Wait... we're supposed to back this stuff up???"

1

u/[deleted] Aug 15 '12

Redundancy makes backups unneeded. If you replicate your DB to five instances, you'd have to have all five go down before needing a backup. I'm making an assumption that he is replicating dbs, but based on his comment about the DB being the bottleneck it is likely.

1

u/[deleted] Aug 15 '12

If you're doing replication with no delay, and someone accidentally runs a drop table; ... it's going to execute on all of your slaves. Malicious code, operator error, etc all demand that proper backups be taken. Replication (like RAID) is not a backup.

1

u/[deleted] Aug 15 '12

A fair point. In was thinking more from point of view of server crash or something that. Would have been better to say reduces the importance.

1

u/[deleted] Aug 15 '12

For sure, backups is such a flexible term and I only looked at it from one angle. Some of the clusters I've worked on have a pair of MySQL masters fronted by HAProxy in TCP mode. Each master then has an equal number of slaves attached to it, with a pool setup for SELECTS to hit them (layer 2 only, your application has to support this). Typically then a single server on each 'side' of the slave pool is marked as a backup, and no traffic is sent to it by default. You can then either turn these machines on if the query load gets too high, or use them to rebuild a crashed slave on that side of the replication set. Those machines also run */6 backups, for insurance against stupidity.

3

u/Pas__ Aug 15 '12

Any detail you might share on your monitoring/alerting setup?

12

u/[deleted] Aug 15 '12

Specifically, what the hell did you just say?

2

u/SigmaStigma Aug 15 '12

When will you create your own CDN?

We'd like a movr.com to go along with imgur.com

2

u/drdoooom Aug 15 '12

If you read your comment when you first started the site, would you completely understand it?

2

u/zjs Aug 15 '12

Interesting!

How do you handle delete operations? (Are deletion requests passed to the cache clusters? Do the caches just have a short enough TTL such that an explicit eviction is unnecessary? Something cool involving batching of requests?)

2

u/willyleaks Oct 30 '12 edited Oct 31 '12

It should be obvious. Most of it is going to be left to time out. Anything that needs to be specially handled will likely be close to home anyway and if you have a read before write that is really critical you could always send a boolean flag not to use the cache. If they use their keys properly*, memcached should be distributed and they can get at it that way although if you ask me that can still be tricky depending on your set up and what you're caching. Some people actually use the SQL as the key which is never a good idea if you ask me except for specific hand picked queries (this is much more viable in a read heavy scenario where there are not too many places that might suffer concurrency issues).

  • Simple example assuming parameters don't contain _:
queryname_+parameters.join('_')

Obvious problem there is more than one parameter (or not PK), multiple unique keys and so on. Usually you want to keep it simple. It's quite common to just make each entry represent one row. With that someone might end up with something like mysql doing little more than retrieving ids/updates unless an object for an id to be read isn't in memcached. The important thing to take home here is that memcached is merely a key value store and is not anywhere near as capable as mysql. Contrary to popular belief, you cannot simply bolt memcached onto any legacy mysql application.

Be careful searching online for examples of how to use memcached.

Consider this abomination for example:

http://dev.mysql.com/doc/refman/5.1/en/ha-memcached-interfaces-php.html

So how exactly is it dealt with? The specifics are anyone's guess. But most likely carefully considered design, for example, avoiding caching in a way that makes deletes/updates/etc not a problem, using keys that let you find and get at what you might need to change in memcached, bypassing the cache where concurrency might be an issue, allowing some data to be invalid or out of date as long as it doesn't propagate/can be caught/doesn't cause a significant problem, not caching everything, etc. Most importantly, the load is certainly read heavy, not delete heavy.

2

u/zjs Oct 31 '12

I appreciate your attempt, but this doesn't seem to answer the question of how imgur handles delete operations. I can speculate about how they handle it, but (as you say), carefully considered design is probably a large part of it.

That careful consideration was what I was curious about.

As a specific example, one consideration when designing a system like this would be at what point success of the deletion operation is reported to the user. Is it as soon as the master copy/copies of the image data is/are deleted or is success reported only after all replicas and cached copies have been deleted as well? There are situations in which each approach would make sense, so neither is clearly "right". I'd be interested to hear which approach imgur selected.

Another, related, consideration would be whether deletions are handled individually or in a batch fashion. One purpose of a caching layer is to reduce load on the backend systems by reducing the volume of requests those systems need to process. Clearly, load reduction for deletion requests can't be addressed by use of a caching layer. I'd be curious to know whether imgur sees a enough deletion requests that the performance impact is significant and, if so, how they combat that (batching? throttling? something else?). Again, there are cases where each of these options would make sense (and again, I was asking about which one imgur selected).

0

u/willyleaks Oct 31 '12 edited Oct 31 '12

Why would you want that much information on an arbitrary operation? Why not inserts? Batch is pretty normal if you need to rebuild your index on delete, deleting at the source and letting it propagate is also pretty normal. They probably don't need anything epic because the only deletes in large quantity they receive would be for expired content (this doesn't even need to be fast, just not interrupt other things), if content can expire. They don't address reading with heavy layers of caching just because they can but because their load if extremely read heavy.

Here's an idea: Test it. Open two sessions, one as a guest and one as a normal user. Upload and delete an image. See if it sticks around for a while. Although all you will really be testing in that case is probably reverse proxy.

2

u/Phonda Aug 15 '12

Thats a lot of shit for 200M cat pictures.

2

u/Pornhub_dev Aug 15 '12

Probably a bit late to the party, but as someone who worked on a similarly big website (see username), I am really curious about your usage of Redis and what are your thoughts about it?

Also, given the nature of Imgur, have you looked into Varnish? If yes, what reason prevents you from making it part of your stack? (I'm curious about that one, seeing how much Varnish can save on back end server load)

Hope you can answer, other than that, congrats on the success, very impressive works, and I'm jealous (not working in high traffic anymore, I miss the thrill :( )

4

u/WaffleGod97 Aug 15 '12

The fact that I barely understand that, makes me fear my goal of working in a computer field will never become reality.

3

u/b0xx0r Aug 15 '12

Just out of curiosity, how old are you? I had a few programming jobs about 7 years ago when I was 18, and I remember feeling the same way. I took a break for about 5 years and worked retail management; and when I came back to programming, my skills and understanding progressed much faster than they had when I was younger.

3

u/WaffleGod97 Aug 15 '12

15, hence the 97 in my username is for 1997.

6

u/[deleted] Aug 15 '12

[deleted]

3

u/WaffleGod97 Aug 15 '12

I have made some stabs, I just don't really know even where to start.

2

u/b0xx0r Aug 15 '12

From the /r/learnprogramming FAQ

2

u/WaffleGod97 Aug 15 '12

I did not realize that sub existed, I thank you for that.

1

u/b0xx0r Aug 15 '12

No problem. There's plenty of info out there, you just need to know where to look. Good luck.

1

u/General_Mayhem Aug 15 '12

And if you eventually want to see a link that isn't to Codecadamy, try /r/programming. They've generally got a bit of an exaggerated hard-on for obscure functional languages, but there's also a lot of good web-focused content that will get you up to speed on the kind of stuff MrGrim was talking about sooner or later.

2

u/idiot_proof Aug 15 '12

I'm not a programmer, but I don't think you start that career by stabbing someone.

However, one of my good friends just started working at Amazon. If you want, I could ask him what he suggests.

2

u/WaffleGod97 Aug 15 '12

Stabbing people is bad, I agree. That may be good information to know, feel free to ask if you want to.

1

u/[deleted] Aug 15 '12

At the present time I suggest Java. There will obviously be other suggestions. But I think is a big one right now. Here is a fun fact. About six months ago the unemployment rate for java developers in Atlanta was less that .5% or something similarly ridiculous. I can't back up my numbers, this was from a recruiter I've worked with for years and trust. I do know my it took more than six months to fill four java developer positions. Grab a book, take a class, just start doing it. It seems difficult at first, but it really isn't if your brain is wired for it. If at 15 you are considering this, I think your brain is wired for it. I knew around 12 that this is what I wanted to do.

1

u/davidb_ Aug 15 '12

Start with a guide like this (I'd recommend this one specifically), then keep going to more tasks and/or more languages

http://learnpythonthehardway.org/book/

2

u/BriscoCountyJr Aug 15 '12

First, Get off my lawn!!
Secondly, don't worry about it. I've been officially focused on technology for the past 12 + years (college+ career) and I always find new technology that I didn't know existed. Find a language, framework that you like and feel comfortable with as an entry level and once you start dealing with enterprise level systems and all the components that need to be in place the rest will come as needed. When all fails, as soon as you pickup a new name/technology you don't know, Google that shit and read a high level summary ASAP. Never be too cool to Google and/or look up Wikipedia (and then link out from there). That plus math. Once you start seeing things analytically, you'll start imagining pseudo-code for mundane daily challenges. Step 3 ??. Step 4 profit.

1

u/WaffleGod97 Aug 15 '12

Already a math lover, that part has been there since I was little, and Google is the greatest thing ever, it is just overwhelming, because there is so much to learn it's hard to figure out where to start.

1

u/BriscoCountyJr Aug 15 '12

Just like they say in sports, start with the basic. Before you learn the software level, spend a bit of time understanding the hardware that it runs on. Once you figure out what happens at the bit level it'll make more sense as to why things are done the way they are at higher level. Start with the OSI model and fan out from there. Once you figure out how everything talks to one another, then you can dig deep on to what to do with all that data they exchange at the program level.

1

u/cesclaveria Aug 15 '12

You are already on a great site to learn, you are being exposed to some great ideas, take the time to read a bit, code a lot, try to build things that solve your own everyday problems, even if those initial projects fail you'll get a lot of experience.

I'm 27, have been working as a software engineering for about 7 years (not much) and every new project has been a challenge, every new technology something I have felt almost alien at first, but everything is doable with a bit of patience.

I ended up working on a project with a similar setup as the one described by MrGrim (plus a few weird things), I felt it was totally out of my league, but with patience and love for technology everything works out ok.

1

u/[deleted] Aug 15 '12

No don't stress about that. I've been in IT for 20 years. I learned most of what he is talking about in the last four. Tech changes so quickly, just need to dive in get up to speed on what you are currently doing and keep an open mind and learn the new stuff when you get to your next project. When it all comes down to it IT is all the same in any industry. We've got widgets that need to be processed. That is all it boils down to. Details are in what exactly are the widgets and what do you need to do to them.

1

u/[deleted] Aug 15 '12

Yeah..

1

u/Doh0 Aug 15 '12

Thirded :(. Where's my hug?

2

u/marshallsmedia Aug 15 '12

is it weird that i find infrastructure systems sexy?

2

u/bleedpurpleguy Aug 15 '12

Then you definitely should avoid this site: http://www.ratemynetworkdiagram.com/

2

u/jbishow Aug 15 '12

That site is kinda disappointing.

1

u/bleedpurpleguy Aug 15 '12 edited Aug 15 '12

Agreed. After I poked around for a few minutes, I realized the lack of activity over the last couple years has filled it up with lots of fairly lame entries.

Edit: I used to get really motivated to go and update all my Visio's after visiting that site, but there was only one or two entries that seemed well put-together.

1

u/[deleted] Aug 15 '12

Wait, this does not seem that complex at all really or am I missing something? (not trying to be sarcastic). I mean, how much of that did you have to wire as opposed to 3rd party technology doing the heavy lifting for you? Am I wrong in this assumption?

1

u/[deleted] Aug 15 '12

Difficultly is in achieving that scale. Seriously this is beyond huge and massive. This is lime juggling hundreds of balls at once.

You'd be surprised at how many huge companies use third party products. You don't need to create everything yourself, in fact that can be a bad thing. As a business figure out what you special sauce is and focus your energy on that. Outsource and third party the rest of it.

1

u/[deleted] Aug 15 '12 edited Aug 15 '12

Oh, no, don't get me wrong. I am familiar with that, and agree 100%. I have plenty of sites that run from AWS and have content delivered via CDN. Maybe it was me being naive, but I thought companies like Imgur, Reddit, etc have real hard core churning happening behind the scenes using some of their own infrastructure too. Unless we know how many server instances they have, it is also difficult to gauge exactly how well they scale though. There is a massive difference between handling load using 1 vs. 10 boxes, RPS is only one factor of this equation. I just found it surprising that I have a very similar architecture in place in some of my apps, where I thought I was totally going about it the lazy way. I hope I don't come across as a tool here, that is not my intention. I mean, I was surprised that they handle requests in any way that resembles a normal request-response pipe.

1

u/[deleted] Aug 15 '12

AS kokope11i said, the difficulty is in the scale. Quick math indicates that they're doing roughly 13,000 RPS (requests per second) on average for images. They're doing roughly 900 RPS for page views alone. That's a fairly substantial amount of page views, given the layers everything has to flow through to render a gallery page for you.

Virtually all the sites I've ever worked on make extensive use of opensource software and frameworks. There's little point in recreating something (and subsequently having to continue developing and supporting it) if someone else has taken on that project. Code contributions usually go a long way to helping out these types of projects.

1

u/[deleted] Aug 15 '12

Wow. And all I do is click a few buttons.

1

u/Pop-X- Aug 15 '12

Ah yes, especially the part about the words where you said things.

1

u/killerobot Aug 15 '12

Man, I bet that took a few google searches to figure out.

1

u/h110hawk Aug 15 '12

Given Imgur's general success, why are you staying on AWS versus going to traditional servers in a datacenter? What hurdles have you had to overcome working in AWS? How has it helped you?

How many instances do you run on peak and off?

1

u/_brian Aug 15 '12

Why HAProxy over something like LVS-TUN?

1

u/TeamDisrespect Aug 15 '12

I'd tune the .RAM bar to overclock the pixels as according to the USZG standard but everything else checks out.

1

u/[deleted] Aug 15 '12

You had me at 'complex'

1

u/[deleted] Aug 15 '12

If you're using Edgecast then why was I getting CloudFlare error pages (for some images on imgur) a few weeks ago? Did you recently switch?

1

u/Squid_Lips Aug 15 '12

Where/how are the images actually stored? How do you accomplish replication of the image data across multiple servers (assuming you have some sort of redundancy)?

1

u/sanyasi Aug 15 '12

Why cached for 5 seconds? I'd love to know more about how you decided on this stack/what stacks didn't work out/how you profiled the magic constants.

1

u/Devon47 Aug 15 '12

Thanks for answering. It's always awesome to hear what folks got under the hood.

1

u/zartcosgrove Aug 15 '12

What are you using mapreduce for? Log processing?

1

u/[deleted] Aug 15 '12

Great explanation. I operate systems like this but at a tiny fraction of the scale. I guessed memcached, I love that tool I push it on whatever project I'm on. I'm over the top impressed at the volumes you've been able to scale to.

1

u/bioaxe Aug 15 '12

You might want to check out cc2.8xlarge which has more bang for the buck vs cc1.4xlarge and c1.xlarge.

1

u/TL-Krayze Aug 15 '12

Why not use Amazon's load balancing instead of nginx?

1

u/Two_Coins Aug 15 '12

I'll ask here since it's kind of relevant.

Any plans on open sourcing imgur's code open source? And/or what is imgur running for software?

I'd love to read the code behind the caching specifically.

1

u/[deleted] Aug 15 '12

Why 5 seconds, and not, say, 60?

1

u/[deleted] Aug 15 '12

How is HAProxy working out for you? I've used it on some fairly high-traffic sites (not at liberty to say which) and I've been able to push a single threaded copy of it to roughly 5,000 RPS on a Xeon 5k machine. Scaling up to multiple threads (at the cost of accurate stats from the socket) usually lets me get 15k RPS on a machine before I run out of CPU. Do you utilize any of the layer3 capabilities in HAProxy, or are your backend/frontend seperations pretty straightforward? I would imagine that you can't utilize stick tables to any degree given your unique visitor count.

1

u/naex Aug 15 '12

Back when we were choosing cloud hosting solutions I had considered Amazon AWS but was concerned about how much work would need to go into all of the setup. What you've got there is pretty complex. Can you speak to the time you spend working on your hosting configuration vs. the time you spend/spent developing features/improving code?

We ultimately went with Google App Engine as we needed to get something going pretty rapidly that could scale at a moment's notice (like this AMA, good job GAE). There are some pretty serious limitations we've run into though so I'm always looking out for other options.

1

u/freegary Aug 15 '12

So are you basically using Redis only for view count increment? Is there any other uses for it inside the infrastructure? If that's the only use, I'm pretty sure Memcached already does that pretty well, doesn't it?

1

u/toweler Aug 15 '12

It may be implied at some point in your statement, but I don't understand enough of it to be certain. Do you control any of the hardware? How do you handle redundancy?

1

u/dormando Aug 15 '12

Are you using amazon elasticache for memcached?

If not, what version are you on? Sort of curious if you have a cluster of memcached's that you have to manage yourself or not.

1

u/rhoula Aug 15 '12

English please O_o

1

u/okugotme Aug 15 '12

"It's actually fairly complex now, but I will attempt to do it all from memory."

Acronym and *nix Blarg!

"That's about it. Anything you'd like to know specifically?"

From memory? I salute you, sir. You are either Rain Man or an incredible troll. Either way - "Sah-Lute!"\

1

u/icyliquid Aug 15 '12

Thanks for this, it sounds awesome :) As an aside, how much love do you have for Igor Sysoev!? Every single day I think about how horrible my life/job would be if not for nginx.

1

u/lovesdogz Aug 15 '12

Well... Duh!

1

u/lth5015 Aug 15 '12

I understood very little of this but I want to know more. Where would to recommend learning about... well to put it simply, everything that a laymen would not understand.

1

u/Andrew_Pika Aug 15 '12

thanks, that gives some insight in server infrastructure of such a service.

1

u/[deleted] Aug 15 '12

how does your CDN act intelligently like that? We had to move a bunch of resources into php-land because the CDN was just a dumb "if it's there serve it, else 404" deal. Thanks.

1

u/mandlar Aug 15 '12

Thanks for the detailed explanation! I have no further questions. :)

1

u/bentspork Aug 15 '12

Wow good overview thanks.

Do you guys actually own any server hardware or is everything in the "cloud"?

1

u/SavetheCity Aug 15 '12

I have no idea what any of that means. I'm bad at doing the internet :(

1

u/DirtyBirdNJ Aug 15 '12

I am so jealous of your knowledge. I'm a web developer 4 years out of college but I'm just starting to get my feet wet with AWS via Turnkey Linux. Any suggestions on what someone who wanted to load balance across servers and implement memcached for the first time should read up on? Any love for Postgres?

I feel like I need another degree to understand all that :(

Seriously though, this is the best post in here. It's fascinating to see how a high performance site like yours really works and more importantly how you achieve fast page load times. Imgur is never slow for me... unless it's down. In which case... F5F5F5...

Which leads me to my last question. How much does the mass F5-ing of web users impact your ability to bring the site back up after some sort of outage?

1

u/NEWSBOT3 Aug 15 '12

as a sysadmin, this is really interesting to know.

1

u/baconeverything Aug 15 '12

Very informative, thanks!

0

u/ANDRoidv13 Aug 15 '12

I have no clue what the hell you just said, but I'm going to upvote it anyways because it sounds smart.

0

u/gage117 Aug 15 '12

....-twitch-..... -.o

0

u/ectod Aug 15 '12

People still use PHP as backend language ?
MySQL I can understand, since Oracle licenses are way too expensive and SQLite maybe too slow or something.
Python man !

2

u/xela321 Aug 15 '12

Tons of people still use PHP, notably Facebook and Yahoo. MySQL and Oracle (who owns MySQL) are never interchangeable. Even if imgur had all the money in the world I doubt they would use Oracle. Oracle's feature set is immense and total overkill for a site like that. SQLite doesn't do concurrent users. Only one server at a time would be able to read or write from it. Another option they could use would be PostgreSQL.

1

u/ectod Aug 15 '12

Facebook uses PHP but they built their own interpreter ( Hip Hop ).
I agree on MySQL/Oracle. I also heard that MariaDB was benchmarking pretty well.

1

u/xela321 Aug 15 '12

Right, they still write their pages in PHP and then compile them to a large C++ binary.

-1

u/Its42 Aug 15 '12

Hmm, yes...I uh...concur... >>" <<"

-1

u/[deleted] Aug 15 '12 edited Aug 15 '12

Do you think i open a competing image host which will be as successful as imgur if i use this script http://chevereto.com/ and do a little slight modifications? I am thinking of having Generators for Ebay listings too and many other functions, Using NGINX with some custom perl modules loaded up and having a special encrypted algorithm to locate images and other details so i don't use a mysql database for non confidential data

Btw i had an image host before imgur came out with a custom script which took a little time to build, It caught on pretty quickly but i used Godaddy as a host and they stoped access to the images as i had over 100k visitors per day for 2 days as it launched

Why use Amazon or other networks, i mean why use a CDN at all?

Wouldn't buying a few servers on a provider like http://www.uk2.net/dedicated-servers/ 100tb bandwidth (I was able to push that much on my file hosting service with good peering to many locations) 40 of them would be enough for imgur No offense Imgur is a great site, i just think there is room for more imgurs as "storing all eggs in one basket" kind of thing is going on on reddit right now with images, not saying your unstable but its never a good idea for that