I guess it was bound to come up. I'm with you on damage dealing though - if that's what I want to do, Warlock isn't how I'm doing it.
When I think back on all the Warlock related threads I've participated in over the last months, I think the issue of short rests is probably the biggest optimizer. Are you going to get two shorts per adventure day? Yes? Great. No? Pick another class.
Past that we are really talking about two major items, well three - pact, spell slots and invocations. Let's tackle pacts further down. The spell slots thing isn't really that complicated - they are high level slots, so make the best use of their value, and make sure you use them. The invocation thing is a little more nuanced. My preference there is to work with the at-will invocations to see what kind of synergy and effect I can work up, mostly combined with skills, feats, and cantrips. I think the per rest invocations are mostly less useful, or at the very least less useful to plan synergies for.
The four at-will invocations that stand out for me get you: disguise self, silent image, alter self, and arcane eye. All of those somewhat mitigate for the infiltrator style build, so that's generally where I go. I like the utility of at-will, and it's easy to build around and get good in-game use out of, even in dungeon crawls, provided there are intelligent monsters involved. I will mention that one way I optimize for that build is to dip 1 to 3 levels in rogue, but that's certainly not a requirement, and I obviously won't be tooting that horn in this thread.
One of the stumbling blocks to the infiltrator build is the reliance on illusion, which is very DM dependent. Provided you have a DM that is going to let you stretch your wings with Illusions, then I think that would be part of my core Warlock build.
Back to pacts as a core chargen decision. Tome or Chain? I like the buffed familiars, but I suspect the Tome for ritual casting is the more useful, as you can still have a normal familiar plus all that ritual casting goodness.