There do not exist any regular pure motives over which are not essentially self dual. Here is why. gives rise to a compatible family of Galois representations for each rational prime such that the characteristic polynomial of Frobenius is independent of this choice. By purity, the eigenvalues of are algebraic integers lying in a CM-field such that for some integral weight and any complex embedding . In particular, if is a root of , then is a root of . Since has coefficients in , it follows that is also a root of , from which one may deduce that (edit: up to the appropriate constant which makes the RHS monic – this doesn’t affect any of the arguments). Yet this implies that , by the Cebotarev density theorem. (Caveat: it really says that the -adic avatars of are essentially self-dual. Perhaps deducing the result for actually requires the standard conjectures.)
This argument no longer applies if one relaxes the conditions slightly; there do exist non-self dual motives of rank three with coefficients; Bert van Geemen and Jaap Top found some explicit examples with coefficients in an imaginary quadratic extension of . The point where the argument above fails is that it identifies the polynomial with the complex conjugate polynomial , which need not equal anymore.
Stefan Patrikis and Richard Taylor use a similar argument in their recent paper to prove a nice result. Start with a regular pure motive over (so by the above remarks, it is essentially self dual). Suppose that the corresponding -adic Galois representation:
is not absolutely irreducible. One may ask: are the irreducible constituents themselves essentially-self dual? They show that the answer is yes. Let denote the corresponding characteristic polynomials. If lies in , then the same argument above applies to . But it may be the case that the representation only decomposes over an extension of . By looking at the eigenvalues, it trivially follows that each of the may be defined over some CM field . More importantly, by a technical argument which I will omit but which is not too difficult, one may find a fixed CM field which contains all the polynomials (one may even do this [in some sense] independently of , although we won’t use that here). Consider the Galois representation , where is acting on the coefficients. Let be a root of . Then is now a root of , and so and coincide. Since , we deduce that is a sub-representation of . In particular, and are both sub-representations of . But the Hodge-Tate weights of and are the same! (Literally, the Hodge-Tate weights of are the Hodge-Tate weights of where , but since is a representation of , conjugation by is conjugation by a matrix, so there is an isomorphism .) It follows (from the regularity assumption) that and then the argument above implies that is self-dual.
One may use this argument as follows. As in BLGGT, one may find a prime such that all of the are residually irreducible, and so (if is sufficiently large) are also potentially modular (by BLGGT again). In particular, either all of the are reducible or they are irreducible for a set of density one set of primes. Moreover, any regular motive over is potentially modular, which is only three adjectives away from the complete reciprocity conjecture!
Patrikis and Taylor do something slightly more general, instead of pure regular motives over , they consider essentially self-conjugate regular compatible systems (with coefficients) of for some CM field . For reasons alluded to above, the coefficients live in some CM-field . This extra generality (mostly) adds some notational complexity to the argument above. (To see the type of complications that arise, consider an elliptic curve with CM and then restrict to the CM field . Then any reducible constituent is related not to its complex conjugate acting on , but the complex conjugate of this where complex conjugation is now acting on the coefficients and on the Galois group .) As expected, one obtains (using BLGGT) some nice consequences, like potential automorphy of regular polarizable compatible systems, as well as irreducibility (for a density one set of primes) of Galois representations associated to RAESDC automorphic form .