Ian Kirk - SQL DBA

No - I haven’t gotten into Policy Based Management (PBM) as much as I really want to, especially after seeing so much of it at PASS this last November.  And no, I’m not going dedicate this blog post to lots of first-hand information that will help you get further in using it.  This is, for all intents and purposes, a really big tweet to @SQLBatman.

First, the MSDN PBM blog that isn’t updated too often, but the (current) top post for “complex” policies seems pretty helpful.

Next, Lara Rubbelke, who had a presentation at PASS that I missed.  She’s been working with Policies, PowerShell, and built an entire framework to help you get it done.

Third, Buck Woody, who seems to be a lot more of PowerShell kind of guy, but knowing PowerShell, I think, is going to be key to automating policy evaluation across your enterprise.  (Of course Lara may have taken care of that for us…).  Buck offers an awesome PowerShell framework that he presented at PASS on his blog as well.

As for my first hand experience, it came before PASS and it was a great experience.  I used a policy stored on my sandbox SQL 2008 server and evaluated 8 instances that were a mix of SQL 2000 and SQL 2005.  I had that policy check specific server configuration items (such as Max DOP) and correct the instances that didn’t meet the stated policy.  After creating the policy, it was just a click of the mouse to evaluate.  Like I said earlier, though, to automate, you’re going to need to know a scripting language like PowerShell.

As a final link, I’ll send you to the PASS Presentation Decks, where you can search through for the presentations and find a number of slide decks regarding PBM.  Specifically:

• DBA-320-M
• DBA-412
• AD-301-M

Enjoy - and be sure to let me know if you figure it all out because learning PBM is high on my personal list… it’s just that I haven’t had any time to devote to that particular list!

Chris Shaw posted a SQL Quiz, setting it up as a game of tag, and I got my tag from SQL Fool (@SQLFool on Twitter) when she answered the question -“What are the largest challenges that you have faced in your career and how did you overcome those?”

I got tagged while I was standing in the middle of the Magic Kingdom in Florida with my family.  I briefly considered answering the call, but realized I’d better focus on the Mouse first.  I’ve been back for a week and I’ve still not gotten my head back 100%, so the reply just hasn’t come up until now.

Database Administration - isn’t that a little twisted?

I think the first challenge simply has to be my choice to take this career path, and the process of getting to where I am today.

From the age of 13, I knew for absolute certain that I would be a police officer.  I was consumed by the culture and the idea.  I bought a radio scanner and listened to their communication.  I played every one of the Police Quest games on my computer.  I even got a four year Criminal Justice degree because my parents told me I couldn’t skip college to take my “dream job.”

So many seeds of my current path had been sewn during that time without my quite realizing it.  First, I was on or around a computer all the time.  My grandfather had my parents drive to his office and pick up the Apple IIe each weekend for us to have around the house.  I was five years old and from this point forward I was surrounded by computer technology.

My father ran an IT based small business from home when he was laid off from his engineering job.  My grandfather and uncle amassed a collection of both Apple and IBM machines.  I went to BASIC programming camp at least one summer.  Yet the entire time I was growing up, I actively rebuffed people’s ideas that I would take a job in IT.  I was going to be a cop and that was that.

Fast forwarding many years, I find myself in the middle of the 6 month long police academy in my hometown - my life’s goal achieved - and I realize that it isn’t anything like what I had dreamed of.  While I had thoroughly enjoyed the people I worked with as a civilian there (doing, what else, Access database work), the daily grind of an officer was looking less appealing.  The people were different here, and I looked inside myself to find that this was not my dream job.

I pulled out of the academy, an immensely difficult thing for me to admit, and I went back to my job at the mall - selling computer software and games - that I had done for more than 8 years prior to now.  I dove into the certification process, grabbing up every book I could find for an alphabet soup of certifications.  I self studied my way to an A+, I-Net+, Server+, MCSE, MCSA, and finally, the last one, a MCDBA.  This was over a six month period of time where I balanced moving in with my younger sister, doing the mall gig and building a solution in MS Access for a family friend’s business.

I knew going in that despite a litany of certifications, I wasn’t going to get in anyplace with some experience.   Luck, though, was on my side.  I got a call one day at work about a 3 week contract with a local manufacturer - their DBA was going on vacation and they needed someone to cover for a week.  To this day, I have no idea how I got the call, but I took it.  I worked with their guy for a week, learning the environment as much as possible, and then it was mine for a week.  I had backup from non-SQL DBAs, but it was basically mine.  The third week, he came back and we reviewed what had gone on.  They called me two weeks later and asked me to do another three weeks so they could speed up a project - I’d made an impression.

I can’t say that I was suddenly on board with this place, but I finally had my foot in doors - I finally had some experience.  From there I landed a gig an old friends from my mall job… which leads me to the next challenge.

Repl-what?

Several months into this gig with a small business where we did bench work for a few other computers, they decided to take on developing software for the retail stores of our major customer.  It was going to be backed out of SQL Server, and I was the DBA.  Long story short - we decided that we’d drop a SQL Server into each of the 80 stores and replicate, over the public network, the data back to the home office.

Yeah - the public network.  Cost savings.  And the “cost savings” continued into the fact that we didn’t get static IP addresses on the end points in the retail stores.  I fought for weeks to sort out merge replication, passing out ID seeds, and working with FTP replication in SQL 2000.  We tested in the “lab” and when the day came, we all drove out to our first stores to set everything up.

There were problems - the app wasn’t quite right, our store setup routine was brand new and clunky, and then the replication just wasn’t working.  At this point, remember, I’m the guy.  We’re 10 hours from home, sitting in a retail store for 14 hours, a few days in a row, while the app code is tweaked and I hammer my head against the desk over replication.

Eventually we pulled the plug on the replication.  It just wasn’t going to happen in the time frame we had, and we had to start moving forward.  The first three stores were allotted more time than the others, and that time was running out.  We sorted out a routine of seeding all the tables’ ID columns with the store ID and decided to figure out how to get the data centralized later.

All I know is that I was in the hot seat, pushing the very edge of my abilities and I had to make a nasty decision to say, I’m sorry - I can’t do what we wanted to do, but we can do [X] instead.  I was lucky and my boss was understanding.  She was technically oriented, understood the situation, and we moved on.

Once we got out of those first stores, my buddy Paul and I ended up working (or driving) for nearly 40 hours straight.  I will never forget that whole process, and the lessons learned.

Into the shark tank

Eventually the retail business worked on my nerves, and I found a new job out of town.  This was a relative start up and they had big contracts in the pipe.  I was part of a two man team preparing to handle a good bit of health care data for some big customers.  My partner was an experienced DBA and I felt very comfortable.  I learned a tremendous amount from him about the day-to-day maintenance of servers.

Being a start-up with big ideas, things eventually unwound and I was in the market.  Luck shone upon me once again and I had stumbled into a contact through playing ice hockey.  This guy, a substitute player on my team, was a DBA manager in town and he was looking for somebody.  I’d turned him down once, with the promise of the start-up still gleaming in my eye, but when I came back a couple of months later, he was still in the market.

I joined up, despite hearing stories of long hours, and I quickly realized I was going to get run over by six runaway trains if I didn’t get moving fast.  I dove in, tearing through the ticket queue, learning about the hundreds of SQL instances in the environment, and generally flying low, but steady while I learned.

Eventually the team shrank, we changed directors and even managers, but by then, I realized I’d positioned myself pretty well.  I suddenly found myself full of institutional knowledge - people on the team would ask about X job on Y server and I could tell them who requested, why they wanted it, what it did and when it ran.  I knew the outage windows for dozens of servers by heart.

I worked really hard to be outgoing, friendly and approachable - something I’d wanted to do from the very start of working in IT.  I never wanted to be the guy to push the user away from the mouse and do the work.  This paid off - I got noticed, and I got added to critical support teams with huge visibility.

At this point, I’m a Senior DBA at this job and I’m working hard to be the mentor that the former teammates were to me.  We pull some long weeks, but I’m working with technology and processes that I just don’t see being used around me.  I’ll trade a few extra hours at work for the experience I’m gaining.

While we were discussing his interview process, I told the former manger, who is now an architect with the company, that I realized I never could have passed his interview.  If I had walked in and faced him, I wouldn’t have seen past the interview room.  His reply?  “I have a greater faith in potential than I do experience.  I knew from talking with you that you had a great potential and I wanted that on my team.”

OK… I was a little late to the party, and I was a lot long winded.  Thanks to all that, I’m not sure that there is anybody left to tag… so - if you’ve read this far and haven’t been tagged, then I tag YOU.

Like most people returning to the office from the PASS conference, the first hours of my day were spent deciding which emails just went straight to the trash can, and which emails required direct attention. Luckily, the second bucket was considerably smaller than the first. Once I got through the emails, I took a short side trip down Twitter-lane, and installed TweetDeck.

I think I chose TweetDeck over the others for its simple black theme. I also think it’s notification system is becoming a giant distraction to me. I’ve seen countless times that I should turn off my Outlook notifications, but now that I have two message notification systems, I’m really beginning to notice the productivity drain.

Once I got through that, I dove into a stored procedure that Michelle Ufford (@SQLFool) had written and posted recently. Like I had said on her blog, it was almost exactly what I had envisioned while I was sitting in the replication troubleshooting session in Seattle. I worked with the code and exchanged a few emails with Michelle through the day.

This exchange may never have happened if it hadn’t before the PASS conference and the use of Twitter. Michelle followed me on Twitter, I found her blog, and then I found a solution to a problem. Soon I’ll have fantastic replication latency information that will help me understand my environment, and do it through an automated manner.

Beyond the replication latency procedure, I wrote a list of ideas for things I want to see me and the team complete. Things like creating policies and automating their evaluation via PowerShell. (Check that out - three big items all in one sentence, huh?) Getting in-person knowledge directly from people like Lara Rubbelke and Buck Woody make this an even more attainable task. The list didn’t end there, though - transactional replication base lining with the PerfMon counters, digging further into DMVs, and Centralized Management Server, just to name a few, ended up somewhere in the list, not to mention just trying to communicate everything we can to the rest of our team.

Today the team ran into a database mirroring problem, and I thought almost immediately to throw it out to the folks I’ve met on Twitter. No one has jumped on it, so we’ve headed to Microsoft. However, now I’m in a position for us to help the community - when we dig out of this particular mess, we’ll have people to tell and an avenue to do it.

We’re three days outside of PASS and yet I cannot wait to be at next year’s.

So I had been hanging out in the Twitter world for half a year, but I was certainly more of a lurker than a participant.  This changed when I ran head-long into a few of the Twitter monsters at the SQL PASS Conference this past week.  I realized what my friend Paul had been, well, twittering about for the past months - it’s a useful process of delivering basic information in near real-time.

It’s already been written, and will continue to be in ways I can’t expect to match, about how fantastic of a contribution Twitter had on the PASS event.  However, some of the inefficiencies were wildly apparent to me - I couldn’t quickly or easily see the hashtag conversations from my mobile phone.  Now maybe I was overlooking something, but it just wasn’t happening.  I would switch between my twitter update software and my Opera mini browser, one to update, one to be updated by the community I don’t already follow.  This quickly lead me back to Paul.

Paul (@pwnicholson) and my brother-in-law Garrett (@phragmunkee) set out together to resolve this very issue a few months ago.  Out of their hard work came a re-tweet bot.  Here, like-minded individuals will @reply or direct message to a central user account, and that account, under the control of the bot, will re-tweet it back to all followers.  This way, you don’t have to follow everyone around in your community / event / etc, nor do you need to bounce over to the Twitter search page to monitor the hashtag.

So - apply this retweeter to the PASS conference.  All of those crazy Twitterers that were keeping the hashtag warm would instead be direct messaging (for a cleaner look overall) or @reply to the @SQLPASS account.  Then, all of the people that are lurkers can simply follow @SQLPASS to get updates on all the latest gossip.

But why stop in Seattle?  Why shouldn’t there be a re-tweet account that lives beyond the PASS gathering and keeps all of these new faces connected as they scatter about the country?  And should it just be one - why not several?  @SQLQuestions anyone?  @SQLGossip?  The possibilities are endless.

Paul’s post on the re-tweet bot gives better examples and probably explains it all better than I have.  They haven’t really unlesashed the bot for the open market, but I’ve got connections, so I can get started as soon as I can think of the appropriate first account - maybe it’s @SQLPASS?  The bot is white-listed by twitter and in action for the Nashville Predators fan group, running as @PredFans.  It’s a beautiful sight - reading only the tweets on topics you want from a large group.

Comments, questions, suggestions?  I’ll let everyone know when I get an account up and running…