Random Bits of Randomness

Removing Outliers in a SQL Server Query

Wed, 17 Feb 2010 12:10:00 -0800

At the risk of getting too geeky, I figured I’d share a cool query that I put together to remove outliers when acting on a dataset. It’s nothing too special and certainly not doing anything new mathematically, but I’ve never had to do it within a single query before. It came together nicely and figured I’d share if anyone else needs to do something similar.

The query below will get the average weight from a fictional Scale table holding a single weigh-in for each person while not permitting those who are overly fat or thin to throw off the more realistic average:

  select w.Gender, Avg(w.Weight) as AvgWeight
    from ScaleData w
    join ( select d.Gender, Avg(d.Weight) as AvgWeight, 
                  2*STDDEVP(d.Weight) StdDeviation
             from ScaleData d
            group by d.Gender
         ) d
      on w.Gender = d.Gender
     and w.Weight between d.AvgWeight-d.StdDeviation 
                      and d.AvgWeight+d.StdDeviation
   group by w.Gender

There may be a better way to go about this, but it works and works well. If you have come across another more efficient solution, I’d love to hear about it.

NOTE: the above removes the top and bottom 5% of outliers out of the picture for purpose of the Average. You can adjust the number of outliers removed by adjusting the 2* in the 2*STDDEVP as per: http://en.wikipedia.org/wiki/Standard_deviation

Special thanks to buddies Komeh and Victor for the help in figuring this out.

This was by far the best Super Bowl commercial for this years...

Mon, 08 Feb 2010 11:49:38 -0800

This was by far the best Super Bowl commercial for this years game. Unexpected, relevant and funny.

A bit of awesome news

Tue, 19 Jan 2010 23:20:58 -0800

I just wanted to share with everyone that I’m actually going to be a dad.

It’s a little hard for me to believe that it’s true, but the doctor claims it to be the case… and who am I to argue with the person waving the magic ultrasound wand.

Provided everything progresses normally, we should be celebrating a birthday around the first of August.

Chargers sign Norv Turner to a 3 year contract extension

Mon, 18 Jan 2010 21:35:48 -0800

Chargers sign Norv Turner to a 3 year contract extension:

On Sunday the Chargers dissapoint everyone except a few Jets fans by making bad offensive decisions and playing with absolutely no discipline. On Monday the Chargers decided to continue these mistakes for 4 more years.

I don’t trust Norv Turner to coach a turd out of my rear, let alone a team of championship caliber players to the Super Bowl. I have nothing more to say to Spanos and A.J. Smith. Maybe I should move on to the Seahawks next year.

http://www.buzzmachine.com/2010/01/17/the-right-to-link/

Mon, 18 Jan 2010 05:23:55 -0800

http://www.buzzmachine.com/2010/01/17/the-right-to-link/:

A must read explanation of the Link Economy of new media and why Rupert Murdock is an idiot.

The most important video you will watch this year. I know the...

Tue, 12 Jan 2010 17:52:36 -0800

The most important video you will watch this year.

I know the year only began, but I guarantee that this video will change the way you think about technology, the internet, your government and where we head in the future.

Say it ain’t so LT! Say it ain’t so! I cannot...

Sun, 10 Jan 2010 10:15:00 -0800

Say it ain’t so LT! Say it ain’t so!

I cannot believe he did this.

Diet Begins Today

Mon, 04 Jan 2010 14:17:13 -0800

My diet officially begins again today. I should be getting my twitter scale today as well, so I’ll be doing my first official weigh-in soon. If you wish, you can keep up with my progress at http://twitter.com/JasonsDiet

For the record, the day’s not even half over and I’m starved… But more because I’ve been too busy to eat than being limited by what I can eat.

Rush has been released from the hospital.

Fri, 01 Jan 2010 18:09:43 -0800

Rush has been released from the hospital.

Someone really has it in for the new Animal Hospital

Thu, 31 Dec 2009 11:47:00 -0800

Someone really has it in for the new Animal Hospital:

From Pedro at the AmbergrisVacations forums:

“I was in Ivo’s new coffee shop when I got a call from Laurie Jackson (my asst. manager) who said there were the police/immigration/customs personnel at the gate with a search warrant issued in SP of todays date,and I better come back immediately.

When I got back there was Laurie(the vet) with the immigration officer, who had not waited for me,investigating all her paperwork which he ruled was a 100% correct.I asked him if it had not been correct what would have happened and he said she probably would have been taken into custody.

The customs and the police went to my bar examined every bottle,then to the liquor store room,and then into my house-they found nothing illegal,and by this time they were apologising as they had been told to come by various luminaries on a charity board,who had been making lots of calls to the heads of their respective organisations-this could be looked as an abuse of privilege.”

Apparently someone really has it in for the new San Pedro Animal Hospital. Laurie the Vet used to be employed by the local animal charity SAGA. I would absolutely hate to find that they had anything to do with this. The Animal Hospital is a great addition to San Pedro and important for future ex-pats like myself.

Also, kudos to Laurie the vet and and Pedro for being on the up-and-up and being able to pass this detestable test by those with evil intentions.

My Diet - again

Thu, 31 Dec 2009 10:03:51 -0800

So if you’ve known me for any length of time, you know my weight fluctuates nearly as much as Oprah’s… and I recently noticed an odd correlation to my charted weight with the 5 year NASDAQ chart.

So it’s time again for me to take the financial sector to it’s knees with another diet.

Last year about this time I started a diet and managed to lose 65 pounds. I did it using a diet comprising of extremely bland food in exactly measured quantities with little exercise. It worked great for 6 months, but as soon as I got sick of the crappy food, I started regaining weight. I’m still netting 20 pounds less this New Years than last New Years, but all those clothes I bought at my low no longer fit.

So time to get back on the wagon. I really think I need to get back to the gym, so I’m going to start doing this with good exercise regimen. Also, I’m going to try to do it with a little less effort on measuring what I eat. I’ll just stay away from the bad stuff and stick to more of the good stuff. We’ll see how it goes.

The biggest diet change though that I’m going to make this year though is that I’m going to do this in public. I’ve created a new twitter account http://twitter.com/jasonsdiet where you will be able to track my progress. I’ll be tweeting from there throughout the diet, but more importantly my daily weigh-ins will be automatically tweeted from my new twitter enabled scale. I won’t be able to lie about it, even if I wanted to.

I’m calling out to my friends and family who are out there reading this. Please help keep me accountable. Diet starts Saturday and tweets should start rolling through on Tuesday.

The Evil of the Left

Wed, 30 Dec 2009 20:34:00 -0800

Rush Limbaugh was rushed to the hospital today with chest pains while vacationing in Hawaii. I learned about it while working when someone I follow mentioned it on twitter. Twitter, and other new and social media is a terrific way to learn about news before other traditional sources have picked up the story… but since it’s a medium very close to the person telling the story, it also exposes a lot about the people behind the information.

While watching this story develop on Google’s realtime search, many tweets made their way through, most of which were extremely disturbing…

@randomscreenname: Dear Rush Limbaugh, please die. Sincerely, The People of America.

@billpalmer: come on God, I don’t ask you for much, and I very rarely ask you to smite anyone. but please kill Rush Limbaugh tonight.

@LONGBRDSK8R #RUSH LIMBAUGH IF IT COMES DOWN TO THE POWER OF PRAYER YOU AIN’T GOT A SNOWBALLS CHANCE OF GETTING THROUGH THE NIGHT GOOD RIDDANCE PIG

@epik_fail I’m not as good or decent of a person as the rest of you, so I’ll just say it outright: I hope Rush Limbaugh fucking dies.

@Skullbuggy Please let Rush Limbaugh die. Please let Rush Limbaugh die. Please let Rush Limbaugh die.

@billionbucks Rush Limbaugh in a hospital? Maybe our health care system will fail enough to remove him from this world he bitches about everyday. #byebye

These are joined with literally thousands of other vile hatred filled tweets. When Kennedy was nearing the end, I don’t recall the right ever expressing a death wish for the man. Sure I disagreed with Kennedy and believed him to be bad for this country, but I never once tweeted wishing for his demise.

I do pray that Rush is ok and I really believe that we need to start praying for our country and our world. When you surround yourself with mostly good, honest and moral people you tend to forget just how evil the rest of the world can be.

Barney Frank agrees with Ron Paul on health care?

Wed, 30 Dec 2009 14:11:04 -0800

Barney Frank agrees with Ron Paul on health care?

Lip reading Star Trek… Whoever made this video was...

Tue, 29 Dec 2009 13:52:48 -0800

Lip reading Star Trek… Whoever made this video was extremely high.

Merry Christmas from Tay Zonday!

Wed, 23 Dec 2009 16:43:53 -0800

Merry Christmas from Tay Zonday!

SQL Server Pricing

Mon, 21 Dec 2009 14:23:00 -0800

I find SQL Server pricing to be very disturbing.

SQL Server Express is free for databases up to 4GB in size. That’s fine and all, but think about this… If you have a database that will grow to 24GB in size, it is actually cheaper to split your data in 4GB chunks across 6 rackmount servers (including purchasing new hardware) than it is to upgrade to a version of SQL Server than can handle the data.

Hey Microsoft, how about a version of SQL Server that is within range of us small business and hobbyist coders out here? And is database size really the differentiator that you should be using to force people to upgrade?

I understand limiting the amount of RAM in use by SQL Server and even the number of cores that will be used by the free version, but it’s pretty easy to pass 4GB of data. In fact, I think the limitation is darn near unreasonable.

I thought I had found a great solution to the problem. Microsoft’s website explains a product called SQL Server Web Edition which is priced at $15 per proc per month. Unfortunately, a call to Microsoft clarified that since I own the server, that price is not available to me. It can only be purchased as part of a hosting plan from a qualified web hosting service. Seriously Microsoft? Seriously???

I’d be happy to pay something reasonable on a subscription plan if it removed the restrictions and provided some sort of SLA with it. Unfortunately, MS doesn’t want my money unless they get a whole bunch of it.

Guess it’s time to upgrade to MySQL, PostgreSQL or Firebird.

I'm Feeling Lucky

Thu, 10 Dec 2009 11:28:00 -0800

I just found out that if you go to http://google.com, enter Jason Southwell in the search box and click the I’m Feeling Lucky button, you are directed immediately here to this here fine site.

Moral of the story, if you are reading this, you are in fact very lucky.

This chart illustrates just how few people in the Obama...

Tue, 08 Dec 2009 13:18:00 -0800

This chart illustrates just how few people in the Obama administration have ever had a real job in comparison to other previous administrations.

Bill Bresnan - Rest In Peace I have had the opportunity to work...

Thu, 03 Dec 2009 12:54:45 -0800

Bill Bresnan - Rest In Peace

I have had the opportunity to work with Bill’s companies for several years now. He always treated everyone with respect and believed in holding himself and his companies to the highest standards, both in business and in ethics.

He will be missed.

Hillary Clinton thinks we were attacked by Afghanistan.

Thu, 03 Dec 2009 08:57:15 -0800

Hillary Clinton thinks we were attacked by Afghanistan.