Well, you can flip through a book before you buy it. If there is a likelihood of confusion that the layout would make a consumer think the book is a WotC book and not a 3PP, then there's at least an argument for trademark.I'm not sure if it can be a trademark when it is not visible to the consumer during the course of trade (eg because it is a table in the middle of a book). WotC, in the GSL, claims very 4e-related trademarks (in clause 5.2) but the 4e layouts are not among them.
My sense is that the 4e layouts are the subject of copyright (as pictorial or graphic works).
Any given page from a 4e book is covered by copyright, taken together. When you separate the content from the layout, things get a bit tricky. The content, the words and pictures, are certainly protected by copyright. Even if you lay it out differently, you're still infringing (at least if you don't have a exception).
But what about just the layout? What if I write all new content and just make it look like WotC product? I haven't taken a good look at them, but I suspect that the EU's strong unfair competition laws would call that palming off, a big no no.
In the U.S.? My sense is that the layout of a book would considered trade dress. Particularly if you are, in fact, using it to distinguish your products (RPG books) from others. Which is how they were using them. I can spot a WotC 4e D&D book from 50 paces. They're designed that way for a reason, so people know which books are official and which ones aren't. Maybe the GSL doesn't claim them, but I strongly suspect they haven't waived their rights to it.