I want to start by saying that this post is not a post about griping about the recent headaches we all experienced here at ActiveRain. Nor is this post about how wonderful I think ActiveRain is. What this post is about is simply one thing, the people of ActiveRain. While we have all experienced deep frustrations over the past week and many have already voiced opinions about ActiveRain on Twitter or Facebook. This is not my post ranting about things, it is a post about the people who run the network.
I get the point that some of us are paying for a service that was not working 100% for the past few days. I get that by the servers being down may have cost some of us some potential money. Heck, I even get the complaint that you couldn't get online to leave your 10 comments/day and 10 posts/week for points. But, the thing that many do not realize is that the down time was not something ActiveRain wanted either.
Let's take a moment to actually look at the daunting task that ActiveRain is accomplishing at this time. At the time of writing this blog post, there were 1,628,526 other posts on ActiveRain. In addition there are 180,822 profiles. And approximately 5,000 outside blogs per a conversation with Bob last week. So lets say that each of those outside blogs have 100 posts, that means another 500,000 pieces of content. In addition, we have localism with another 180,822 profiles, and another 800,000+ posts. The other big piece of data that would need to be moved would be all the pages for each state, county, city, channel, & group. Easily that can account for other 1.5 million pieces of content. So, if we were to add this all up, there would easily be over 4 MILLION pieces of content to be moved. To put this into perspective, it would be like trying to move the entire population of the state of South Carolina in a week's time.
Definitely not an easy task. But, that also is not the point behind writing this post. The point of writing this post is to shed some light on the team at ActiveRain. Since I used to work here at ActiveRain, I was given the pleasure of getting to know the team fairly well and thought it was time to share a little bit about them and what your team is really like. I know sometimes members feel that there are better things that ActiveRain could do to handle situations, and that is probably true. Are they perfect? No, but if you can find someone who is, please send them my way to meet me, because as soon as they do they will become imperfect. Anyways, the next four people I am going to talk about are solely responsible for keeping this network where it is today.
First, there is the newest community builder here at ActiveRain, Kerrie Greenhalgh. Kerrie has probably one of the worst jobs here at ActiveRain. She is in charge of handling answer every help ticket that comes into the network. After doing this job for a year, I know how much of a task this job is.
On a normal day, there are anywhere from 200-500 tickets that can come in. During this last week's downtime, I can only imagine what the numbers were like, so if you submitted one with a problem, please be patient with her as she tries to dig out from this. Hopefully by now most things are back working properly, so you will not need to wait on the response. But, if you are needing help, I promise if you give it time, her smiling face will either be emailing you a response, or possibly even a phone call!
The second piece of the team that keeps this ship running and who has probably put in the most work over the past ten days is the head developer here at ActiveRain, Jorgen. I doubt that this man has slept more than three hours a day during the past week as he has taken on this major project of moving things. Normally when something this major is going on, there is a full team to make this type of move. A team of several people, not just two or three other people. Personally, I cannot believe they were able to accomplish things as fast and as quickly as they did. This guy just astonishes me with his skills and ability to get things done. In addition to handling the major tasks like this one going on, Jorgen is the one who single handily keeps the entire network running by himself behind the scenes. He probably would not like the fact I said all this because he prefers being unseen, but please go send him some appreciation on his page.
Lastly, the two guys that have probably come under the most fire over the past week have been Bob Stewart & Brad Andersohn. The reason that Bob & Brad have taken so much heat is due to the fact that they are the faces of ActiveRain. They are the people that you turn to when you have a question or need help with something on the network. From working along side these two for almost a year, I learned very quickly their work ethics and dedication to members. Many mornings when I would awake and start work, I would already find them online at 5:30 AM their time! When I had started here at ActiveRain, I remember Bob saying to me this: "Jeremy, to work this job, you have to realize you do not get time off." Never before were such true words spoken. I used to receive phone calls at 7AM from my good friend Lenn Harley and also at as late as 11:30PM from people like Lisa Hill. We have all become so used to their prompt responses and instant troubleshooting of our issues that when there comes a time that we aren't getting the immediate fix we get a little frustrated.
There is nobody on this network more passionate about ActiveRain than these two guys I promise you. Unfortunately though, when you are the mouthpieces for an organization, you are also the persons who take the abuse when something goes wrong. The past week I have seen all sorts of comments directed at both Bob Stewart & Brad Andersohn for the things that happened here. The part that most do not realize, is that neither had control over the network going down, or control over bringing it back up.
Both Brad & Bob have zero connections to the technical back end of what makes things run here. Their jobs are to build our community and make this the most successful real estate network in the planet. Really and truly, just as T Boardman said today in her post, "There is no such thing as a never fail blog or web site or a site that is never down." She is 100% correct. It does not matter what the site is, there are times that things happen.
I remember a few months back when Google & gmail completely crashed. There have also been those infamous times where Twitter & Facebook have had their issues and gone completely down. In fact, today we were all ready for our weekly radio show, Social Media Edge on Blog Talk Radio and right as we began the show, the entire thing crashed. We had special guests lined up to speak but could not host our show. Unfortunately these things happen.
So, take a few moments go visit Bob, Brad, Jorgen & Kerrie to let them know how much you appreciate their work here.
Photo Credits: http://www.flickr.com/photos/a2gemma/1448178195/sizes/l/ & http://www.flickr.com/photos/wwworks/1384952210/sizes/o/
_______________________________________________________________________________________________________
I am the Co-Founder of 210 Consulting- Social Media Advisors. I have traveled the country as a social media speaker. If you would like more information on any of the topics discussed in this blog, please contact me directly. We specialize in social media training, online consulting, and the creation of your own custom WordPress site.
Feel free to subscribe to my blog to keep up to date with my latest posts of tips & tricks to enhance your experience on ActiveRain. If you would like, you can sign up to receive updates via email directly in your inbox.
Also, let's connect on these great social networks:
Useful Blog Posts:
Twitter Applications- 125 Twitter Tools To Make Twitter Easier To Use
Here Here! Great post.
Yup! Amazing! Go Bob!
Jeremy - Bravo! Your post puts into perspective on how daunting the task at hand was with the server changeover. I have no doubt about the dedication that Bob, Brad, Kerrie and Jon have to the membership and how stressful this past week or so has been for all of them and the rest of the team at Active Rain.
Very well said Jeremy! I think the majority though frustrated understands!
Recently, I saw lots of discussions on whether people would leave AR, due to the recent problems. Talk about cutting your nose, to spite your face. Sure, it was frustrating not being able to log on or to add comments, but there's still no other site like it. Glad to have it up and running, again.
I agree and understand your points made. I do. But I also understand the statement, "don't tell me about the labor, just show me the baby". I don't even pretend to have a clue about what was going on behind the scenes or what was wrong with what. I have no idea. None. Although I am interested and would like to understand more, I am fine with the fact that I probably won't ever know. I can understand both sides of the down time. But I'm good, it's raining again.
Thank you
Well said Jeremy. Certainly we all knew the general scope of the transition but your numbers bring it home big time ! Thanks for the update.
I may have mumbled a few things under my breathe when I tried to log in but I know the task they were facing. This isn't/wasn't like putting a backup drive on your PC and saving all your data. Hopefully your post with the numbers will help others understand how tough this was. Thanks to everyone on the AR team for making it happen.
Jeremy - I am glad that the basic Active Rain is back and apparrently functional. I am really waiting for outside blogs and localism to be restored.
You guys did a wonderful job! Best to you in the future! Keep the communication flowing!
Everything seems to work fine now. I am sure the move was for the best. I admit it was frustrating, but then that shows the importance of this network!
Thanks everyone for the comments!
I have been here about 156 weeks so far, so being down for a little while gave me the oportunity to work on that novel i have always talked about. All kidding aside, that is less than 1% of the time I have participated in Active rain. Since the migration things are going so well that I have renamed you Speedy Rain.
I appreciate the numbers; it puts it all into perspective. I wasn't really upset because my franchise goes down from time to time for upgrades and they have a whole department. When I read about the few hours a night, I didn't see how they were going to accomplish it. I'm here for the long haul, but I sure needed my active rain fix!
Things are now in perspective. Thanks!
I think we all forget that not everything is just fixed with a flick of a switch. We have become so reliant on the internet but I for one had nothing to complain about, I found other things to do-I sold 3 houses!
Hello Jeremy, you did a fantastic job putting the facts into perspective, and I thank all of those in Active Rain who put a lot of dedication and effort into keeping this great AR networking site going.
Jeremy - Great post giving all the numbers. It gives new perspective on what is happening behind the scene. I had no problem with the down time. I found something else to do. You guys at AR are doing a wonderful job! We do appreciate it.
Jeremy, thanks for the in depth explanation but just so you know, some of us understand that stuff like this is bigger than all of us put togehter and we just appreciate all the effort and thought you guys put into achieving the transition! THANK YOU!
Jeremy Arthur and I support the team to the fullest - even expressed it on the AR Facebook page last week
Hey Jeremy I missed you at REtech South. I saw you figured I would catch up to you and then you were gone. Anyway great post. But hopefully future planning will prevent the great sucess here from causing another bout of problems like we went thru
Charlie- Planning was put into this. I had talked w/Bob probably 3-4 weeks before the actual transition took place. I am sure their team spent countless hours in planning & prepping for the migration. However what they could not plan for was some of the glitches they faced. One of the big ones was the fact that the first place they were moving to wasn't large enough & needed another place.
Let me use a football analogy: When you hit a snag & have to call an audible, it doesn't always go off seamlessly. Which is what happened here. But, I think they did an excellent job recovering the fumble and at least picked up a first down. Have we made the touchdown yet? Not exactly, but we are definitely within scoring position!
Jeremy- I had an idea of what was going on, until you made it "clear" with your football analogy ;-)
How do I become a social media advisor? I like the title---
Just came up for air and wanted to say thanks to Jeremy and to all of you for being so understanding and so supportive. We are on the road to complete recovery and to an even better ActiveRain than we've ever known!!
Special thanks and kudos goes out to Jorgen today as well!! :-))))))
Jeremy we take a lot for granted and don't like to be inconvenienced, but the facts are, that inconvenience is part of making things better. Everyone did a great job and the system seems to be working fasssssssssster than every :)
Jorgen is my hero. Seriously. I'm glad we're back. I've missed all of you. :-)
Way to give a shout out and give credit where credit is due.
I believe this time period gave members time to fix/repair other social networks, ie facebook, twitter and linkedIn....Thanks to the team for a job well done!
Jeremy. I certainly wouldn't want any of their jobs thid past week. Frustrating to say the least. All anyone can do is theur best.
Thanks AR gang for all your work. It IS appreciated!!!
Thank you SO MUCH for ALL YOU DO! You are all amazing! Jeremy you made a really good point....their task was an amazing one. THANKS!!!! Active Rain ... let's rock and roll!
Thanks for the enlightened wisdom, Jeremy. Many people have become so accustomed to AR performing well that when problems crop up, it's hard for them to be patient and wait them out. AR is still, for all intents and purposes, an infant, and like all infants, there are growing pains and hiccups that we "parents" have to weather and learn from. Just as you wouldn't throw your infant out on the trash pile the first time it misbehaves, neither should you abandon AR just because of some system downtime. Roll with the punches, be patient, and eventually the child will emerge stronger, wiser and better prepared to handle similar events in the future. And so will we.
Welcome back, AR!
Thank you Jeremy..... Good things come to those who wait..... The AR Network is great!
Four million is a very, very tiny number in this context. I just hope they've resolved the errors and we can quickly get back to normal.
Hi Jeremy, ~ For one thing, I surely did not know that you did not work for activerain anymore. Secondly, life happens and no one is expected to be perfect, that is why we are called humans! I love activerain unconditionally!
Jeremy, if you ask me, the pain was worth it. AR is running smooth as "butter!" I love the responsiveness now. Thanks for the update!
Had such a busy week -- I have to say I didn't miss it! But it is exciting to think about the improvements made and as Bob said, "an even better Active Rain than we have ever known!"
Jeremy I am glad they got it up and running finally. Maybe they can cap the oil well in the gulf?
Hi Jeremy...Knowing everyone was doing every single thing they could to make things better made every PROXY ERROR and BAD GATEWAY easier to take.
I was and am a very happy camper.
Kate
Jeremy thanks for writing a post acknowledging the people behind the scenes. They have shaped this community into what is today.
Jeremy - Glad things are working. Glad for the effort it took. I can say most of us work hard to achieve results. I appreciate results. Here's to a better Active Rain
Active Rain is a very young prodigy - who knew in the beginning how big and powerful it would become?
So the young giant was asleep for a week -- much needed - awake, and stronger than ever.
The brilliance and dedication behind the scenes - the brilliance and dedication of the customer service and community managers - what a team!
We win.
Jeremy, This was well written much needed applause for those behind the scenes. By having AR down, real estate was my total focus..just listed 3 more properties. Wonder how many there would have been without AR these past few years..a lot fewer, for sure!
Jeremy - it is a huge task, and I cannot imagine transferring that much info. It is obvious that there are many here utilize AR on a regular basis.
Jeremy--thanks for the reminder of just how big a project this is. Good luck to the new girl on the help desk! Ha!
Jeremy, seeing an estimate of the numbers does help put things into perspective. Thanks for taking the time to put this together. I love Active Rain and had confidence that they would resolve the issues.
This is right on and needed to be said in just this way. It must have been a tremendously huge job and all I've seen and heard from staff has been reassurance and done with a smile.... Yet they've heard a lot of complaining...
I, like many members, knew the task was huge. I also knew that if AR could have don't it without the ensuing delays they would have done so. We missed it, but I think for the most part, we knew it was a huge undertaking and for the most part, we simply just understood and waited patiently. I must have missed all of the complaining except those expressed in jest and in the fondest way in an attempt to liven the wait a bit.
Alright now.....I was one of these Tweets and Bookers you make mention of! I think I was pretty positive, once I knew what was going on! Indeed, this was no small task....like I said on another post, I have this picture of a bunch of trucks loading up computers and such and hauling them across country. Shows what I know about "servers."
I'm still amazed most days when I hit "enter" and I get something back. Technology. Can't live with it and can't live without it.
Active Rain has done a great job with a monumental task, and we'll all be better for it.....
It would be so easy to jump on the blame train, but my blog has gone down... and I've moved it before. It isn't easy OR fun.
All the well meaning mission statements aside, AR has become a technology platform and the managers have a responsibility for a maintaining a predictable level of service. The volume of data to be moved was known and if the scope of the project was misjudged the employees and consultants who worked on the project must take accountability. Conversions have been done millions of times since the introduction of computers and the process is both known and predictable. It's called work for a reason.
What's a few glitches in the sheme of things in life? When you get right down to it. this was NOT the end of the world. AR rocks.
Now that is community support at its finest....and that's what built this network. HUGS to Jeremy....for a diplomatic way to put things in perspective. I LOVE IT! Worth the wait and FAST!
Very appreeciative of the staff. Glad to be back!
Tom
Thanks Bob, Brad, Jorgen, Kerrie and all...
ActiveRain may NOT be about people to some members but it is to me. I will be disappointed any time technology does not do what I want it to do for me.
I think we need to practice some Tough Love on ourselves as ActiveRain but remember that the people we have access to as ActiveRain Corporation are doing the best they can with what they have to work with. ActiveRain is the number one real estate network because of people. Jon and those above and Matt, Caleb, Lydia, Rich, Jeremy and others in the past.
Jeremy - You wrote an excellent background piece on the operations of ActiveRain.
The ActiveRain backend is coming back very nicely. As soon as it was back in decent operation a couple of days ago, everyone was commenting how nice it was to be back in operation. The loyal members will continue to use ActiveRain but will also have learned to use other sites as well which is a good thing. Even huge sites and companies can disappear. Cross-promotion on other sites is always a good idea.
Bob and Brad are the "face of Activerain", that is who we know and communicate with.
So they hear the most of the gripes, not sure how they do it.
I can't even imagine the tickets Kerry has to deal with.
Much thanks to all!
Jeremy, your insight to the issues and the poor people who received the brunt of the crticism is wonderful. Thank you. As Lenn wrote easlier this week, the only reason we were so upset is because we love the site so much!
Jeremy: Technology issues are a daily grind for most of us. The folks behind the scenes at AR...and the faces, too...have just been through what most of us relative non-techies can not even fathom surviving!
Kudos to all. I deeply appreciate the knowledge and expertise that has built this site and helped so many of us gain higher visibility in our respective markets. Kudos to the brainiacs and the brawny and brave ones at Active Rain....
Big shout out to them!!!!! thank you for sharing what all of their roles are!
Leslie- Even though I do not work for AR anymore, I can guarantee that they put many days/weeks of preparation into moving this stuff. Since it sounds like you are an expert on the subject, next time they need assistance, I will have them contact you.
Thanks Maureen- I really did love being a part of this family. I never before worked with such dedicated hard working team of people. It is amazing that a staff of 12-15 can handle the volume of 180,000+ members!
Missy- I am sure being the future Mother in law you have heard a few of the fun stories...I remember a few very colorful phone calls we would have :)
Very well said Jeremy. Thanks for defining everyone's roles and introducing the newbies. And thank you to the AR team for all they do to make out lives better.
Jeremy,
Your post is right on for me. I've been raining for about 7 months now. Love. Love. Love. Though I am absolutely addicted this is not like being without power, water, or any other of life's necessities. Well ... maybe it is, but I have found everyone at AR going beyond any of my expectations.
Love all y'all.
Yes, am fully impressed with the Rain gods here! Thank you for writing the post that I ahve wanted to write!
I'm glad we are back!!
Patricia
Jeremy, I agree it was not an easy task. We were actually not inconvenienced that much. To not have the errors now was worth it.
Jeremy........ I am just so glad the network responds FASTER than it did before.....
I was in the middle of moving when this all happened..... so it really did not affect me at all.
Brad is an awesome person and I applaud all of ARs employees!!!
I am sure Brad and company are glad you posted this!!
=-)
Jeremy: I have been a bit annoyed at all the angry banter going on over the past week. I agree with you and the quote that nothing is ever up 100%. I am just glad that the problem (proxy errors etc) was being addressed and handled. I realized it may take more than a few hours and that the site might be down for a while. In the grand scheme of things, it wasn't a big deal and hopefully those that were ticked off have calmed down by now. I'm just super-happy I don't have to deal with dang proxy errors and bad gateways any more so I say "Thank You, Active Rain!"
I am very appreciative of AR's efforts in the past week. I have not had the first issue since Tuesday!
Jeremy,
Solid content right there. Kerrie and Jorgen are due a hefty raise. Bob and Brad should get a few weeks off to chill, I say.
Jeremy- Thanks for a great post and setting the record straight. I must say I never knew how much I really depended on AR until it wasn't there. So glad it is back and up and running!
Jeremy - Nice job of puttings things in perspective here. That is a LOT of content! I've always known that AR had people in charge who genuinely care about us - glad to see that things are back on track, even if some minor stuff still needs fixing.
Jeremy -- In their usual style, the ActiveRain team's dedication and determination has been very evident and is much appreciated. We all knew they weren't sitting on their hands. Moving 4M pieces of content is indeed a daunting task...especially without taking the system down for a period of time. Kudos to everyone who helped to work it out.
Jeremy,
I'm surely glad that I wasn't any of them this last week. Big thanks to Jorgen.
It's almost impossible for a site to be up 100% of the time, and no one will guarantee that.
Moving and adding servers to a different state has to be a difficult task, which was made necessary due to the rapid growth of AR.
Most understand that. Many don't and that's unfortunte.