Why is CS_DROPSHADOW a class style and how do you use it with a Dialog/DlgProc?

Recently, I've been doing some UI work. I program for Windows in C++ on bare-metal Win32 API. No MFC, no WTL, for WPF, no .NET, nothing. I frequently end up having to write my own UI widgets ("Windows Custom Controls" in Microspeak) because most control libraries assume you're using one of the above technologies. That, and most custom controls have annoying feature or performance deficiencies. So, I'm pretty good at hacking on the Win32 API. I read Raymond Chen every day and somewhere I have blackmail photos of Ales Holecek (pronounced "Alesh") at night in bathing trunks, in case things demand drastic measures.

One widget I've been working on lately is what I call a popdrop panel. It's kind of like how a ComboBox (Drop-down-list) works -- a minimal UI most of the time showing the current state, but when the user wants to change the state a temporary UI appears as a temporary layer atop the surrounding UI offering more extensive controls. This layer disappears when the user is done interacting with it. I couldn't find a good way to do this with existing controls, so I resolved to make my own. This UI is typically used in areas of limited screen real-estate and complex interaction where we need to save space. In my case, this is often within a window that already is complexity-managed with a set of Tabs. The popdrop panel contents are usually several complex custom controls themselves, so the popdrop is basically a sub-Dialog itself created in the resource editor and reused numerous times throughout the program.

So there are the requirements. In order to reinforce the temporary layer effect of the popup/dropdown UI, I wanted to employ the drop-shadow look used by the ComboBox and Menus. There's lots of ways to do this by hand, see Codeproject or CodeGuru for examples, but I don't like to reinvent the wheel more than is necessary. Fortunately, in WinXP and later (which is all I care about) we have the CS_DROPSHADOW class style. Done deal, right? In the details, therein be the devil.

First, where do you apply CS_DROPSHADOW? Ok, it's a class style. You need to register a new custom class so you can specify this class style, as it applies to all windows of that class, not just an individual window you choose. I already create several custom Window classes (not to mention all the custom widget classes) since you specify the top-left SysMenu icon on a per-class basis too (why, I may never understand). Ok, so, I start framing out some code with RegisterClass and a WNDCLASS structure. But then, I realize that I can't just whip up a new class from scratch -- this is a Dialog and it needs to use the canonical dialog box class in order to get the standard DlgProc and all the spiffy control tabbing handling and such. Hmm. Ok, slight change of plan. I'll just SUBclass the existing dialog box class and create an exact variant that has the CS_DROPSHADOW set. Easy enough. This calls for some GetClassInfo() to clone the existing WNDCLASS so I can override just the style member without disrupting anything else. Hmm. What IS the class name of the default dialog box class anyway? A little digging and Spy++-ing reveals it is the obscurely-named "#32770". Huh. Mysteries of the Universe.

Now that I have the window class customized, I just need my dialog to use it. Some Googling uncovers the frighteningly obvious CLASS statement in a DIALOGEX resource. I chuck my subclassed class name in there, recompile and test. Hrm. No dropshadow. And, I've lost the tab-gradient style on the background of the dialog that I normally get by calling EnableThemeDialogTexture(hwndDlg, ETDT_ENABLETAB);

Ok, let's bang our heads for a while. Fast forward to where I have a headache, and the code still isn't working. On a whim, I wonder if maybe dropshadow doesn't work with WS_CHILD windows, and I turn my popdrop panel into a WS_POPUP. Bingo, now I have a drop-shadow. Ok, so I write some child-to-screen coordinate translation code to position the popdrop panel's initial location as if it WERE a child even though it's not. I'm not worried about having it track when the parent moves as a child would, because to move the parent you'd need to click the parent and defocusing the popdrop is going to cause it to dismiss anyway. For what it's worth, I've had to solve that "track like WS_CHILD even though you're really WS_POPOP" before manually. I wish there was a window class that tracked relative to the parent (like WS_CHILD) but could move outside the client are of the parent (like owned windows: WS_POPUP with hWndParent). After discovering the limitation, I annotate it in the Window Styles and Class Styles MSDN documentation for the next victim.

Ok, what happened to my tab-themed background in the dialog? We're _using_ the unadulterated WndProc (DlgProc, really), same as any other dialog, but something is different. Obviously our Class Styles are slightly different, but I can't see why anyone would intentionally break theming just because we added CS_DROPSHADOW. As best as I can tell, the Theme Engine is probably sniffing the ATOM of the window class (a la GetClassLong(hwndDlg, GCW_ATOM)) and only doing its magic if it matches the internal #32770 class.

So, maybe we can use the theme engine to draw it in ourselves. Now I really do subclass the derived #32770 WNDCLASS a bit more, adding a custom WndProc that only intercepts WM_PAINT and passes everything else along. In WM_PAINT, I use DrawThemeBackground() with TABP_PANE to slap in the proper tab background gradient. Now it's starting to look right. But wait...

Now my transparent-background controls (like STATIC controls) are failing to show the proper background theme. I'd already had to subclass hack these long ago to call DrawThemeParentBackground() so they looked correct on a variety of themed backgrounds. It's another known problem with various workarounds. So, why isn't it working now? Ahh, of course. According to the theme engine (which implements DrawThemeParentBackground), my dialog doesn't get the tab-dialog themed gradient. I had to hack it in in WM_PAINT myself, but DrawThemeParentBackground isn't using WM_PAINT, it's being done by the oblivious Theme Engine. Grr. Several more scalp follicles are lost forever.

"But then, in the midst of my preparation for hara-kiri, it came to me." -Chris Knight (Val Kilmer), Real Genius

Finally I realized what terrible hackery it was going to take to make this all work. I cloned my DIALOGEX resource and gave it a new ID (IDD_POPDROP_SHELL). The original (IDD_POPDROP) was set to WS_CHILD instead of WS_POPUP and removed the DS_MODALFRAME style that was giving it a visible border and added WS_VISIBLE. I also removed the CLASS statement from the original. Now, I create and position the clone (IDD_POPDROP_SHELL) and then create the original IDD_POPDROP as a child with the IDD_POPDROP_SHELL window as its parent. The IDD_POPDROP_SHELL casts the dropshadow, and then contains and is entirely filled by the IDD_POPDROP child window. Because IDD_POPDROP is implemented by the original unadulterated #32770 window class, it behaves properly as a dialog and gets properly themed. I was able to remove the WndProc subclassing and WM_PAINT handling that hadn't quite worked previously. Since IDD_POPDROP is a child, it shows and hides automatically when its parent IDD_POPDROP_SHELL window is shown and hidden. Cake, meet Eat It Too.

So, at the end of the day, here are the unanswered questions, and my attempts at answering some of them.

  • Why is CS_DROPSHADOW a Class Style and not a Window Style?
    A: I know CS_DROPSHADOW is probably playing funky games with window layering and making a temporary layer on which to draw the dropshadow behind and down/right from the real window extent. It doesn't explain to me why it is class specific, but maybe some internal Windows limitation dictated this. I know WS_EX_COMPOSITED also dallies in the realm of layering and partial opacity, and it's an Extended Window Style that can be applied to individual window instances without messing with their class (and all the grief that caused).

  • Why don't STATIC controls (and GROUPBOX controls, which are a BUTTON with some STATIC) draw the proper themed background in WinXP, as is evident when they are placed on a tab page with ETDT_ENABLETAB?
    A: As best as I can tell, sloppy coding and poor testing. I can find no valid reason why you would want the as-provided behaviour or why it would be considered correct. Every poor sap in the world has had to (re)discover this bug and work around it, painfully. It has never been documented in official documentation that I can find, nor has it been fixed in any service pack. I believe Vista behaves the same way, thus cementing it as "annoying intended" behaviour, even if it really isn't. Somebody prove otherwise.

  • Why don't window classes derived from #32770 get the proper theming?
    A: Slightly less sloppy coding and testing. This is more of an exotic case. You could argue that once you're deriving things, the theme engine can't really know what exactly you want. But, the essence of derivation in programming means that a derived entity should behave like the original in all ways except those that the deriver explicitly changed. I think the problem was that the Theme Engine guys had a monumental task thrown at them, and they did the best they could to bodge it onto the existing infrastructure by some clever hacks, and sniffing the #32770 ATOM was one of them. The fact that Windows NT 5 (as WinXP was known originally) was colossally late already probably exacerbated the problems and didn't allow sufficient testing to show that this would bite people who might be forced to subclass #32770 to work around other issues.