Popup windows are always disturbing, but if you get a warning out of the blue that you might have a security problem it's doubly disturbing.
Vendors of what purports to be security software are taking advantage of nervous users to trick them into downloading their software, or at least visiting their web site.

Ever see a window like this pop up while you were web surfing?

It may look like a warning from Windows, but it's just an ad. At least they have put a faint "advertisement" in the bottom-right, but the point of the design is to mislead the user into thinking that there is a problem and that clicking "Yes" will help with it.
Turns out that clicking anywhere in this ad, whether on the phony "Yes" or "No" buttons or anywhere else, takes you to the vendor web site where you can download their product.
We won't get into whose product it is or if the product is worth the time of day. The interesting thing is that you can design a popup web page to resemble a dialog box that a user might encounter if they actually have a Windows problem. More realistic ones than the one above have been created for more dishonest ends, such as simulating program menus.
How do you recognize that a window is a popup web ad and not what it appears to be? This can be difficult. First, even if a window presents a dire message, don't panic. Note that if you look at the title bar and status bar, the top and bottom of the window, you can tell that it is an Internet Explorer window.
This is a clue, but not conclusive proof that the window is not a security warning. It is possible that Windows, or a third party product like your antivirus software, could use such a window for legitimate purposes, but I'd call it a long shot. At this point you can look on the window presumptively as an advertisement.
If you're still curious though, right-click on it and select properties. You should be able to see the site from which it came. A real IE-based dialog box from Windows would have a non-standard address starting with something like 'res:'. A web ad will have a real web address.
The important thing for you to do is not to react quickly to messages that come up out of nowhere, but to stop and scrutinize them. At this point you're already ahead of the game.