The first indication that your site got penalized is the drastic decrease of traffic, if you were getting 100 visits a day to your site and all of a sudden it dropped to 50 visits a day there’s a big chance that you got penalized by Google. Now, to know for sure you need to check in your analytics account your search engine traffic, because you might be getting a lot of social media traffic and it can drop all of a sudden, but that would have nothing to do with Google.
So, if you are using Google analytics go to traffic sources>search>overview, if you are using a different tracking system then you will have to figure it out on your own. Once you are on that page and you are seeing only search engine traffic then look for major decreases, if you find one then your site probably got affected by an update of Google. To specifically know what is the update you need to know when and what update was released.
Go to http://moz.com/google-algorithm-change and track the date when you saw a decrease in traffic and look was there an update there in a range of 3 days or so. For example, if you see a drop on May 22, 2013 then you have got the Penguin 2.0 penalty, if November 21, 2012 then you got a Panda penalty and so on.
If you can’t track back the drop to any update and it happened recently then you will have to try and guess by whether you did something unusual, maybe you bought some backlinks or built some crappy ones or maybe you published a blog post that wasn’t unique, especially look at the guest posts you might have published, but don’t get too crazy and start deleting everything.
So, if you find out that you got the Panda penalty, check out your content, if Penguin then check your backlinks, if any other then Google about the update.