Tokenization is a rather simple concept. Take a field representing classified information and store it in a protected secure area. Make a reference ( token) to that protected information. Now instead of storing the classified information, create a declassified reference to it. Indeed, this solves a huge problem as it is easier to secure a small area rather than a large area. When the classified information is required to be transported to other secure systems, the instruction set to do that can use the token instead of the raw information.
In practical terms in credit card systems it involves storing credit card information in a protected vault. A token to the credit card number in the vault is created that is used by systems outside the vault. The only property the reference token needs to possess is that it must not be possible to derive the original value from the token. Thus monotonically increasing numbers, time stamps, database row numbers, etc meet the definition for good credit card tokens. A scrambled credit card number, some classified substring of the original number, etc make poor tokens for obvious reasons. The PCI Council community has taken this simple and well established concept and beaten it to death with white papers, guidelines, blog posts, discussion forums and expert opinions.
PCI tokenization guideline info supplement
I have some specific consternation with the example in section 2.1.1 wherein it states that A one-way non-reversible cryptographic function (e.g., a hash function with strong, secret salt) as an example of an acceptable token. There is no explanation on how the salt should be kept secret i.e. should it be protected like a cryptographic key? should it be rotated ? Of course any standard reference on secure system design warns against using a static salt for hash functions. Hashing low entropy fields like credit card numbers is not recommended without salting. However, the salt like a nonce must be used only once for every record and with that there is no need to protect the salt at all.
Credit Card State of the State
Credit card security is broken by design. With the current design, Merchants need to accept private consumer information, then they are told they need to be ‘compliant’ and are forced to follow a ‘walled garden’ approach. A whole industry has been created to audit the walled garden. However, proven security primitives exist that make this handling of private information in the food chain completely unnecessarly. Several proven models exist ( think blockchain!) that eliminate the need for a payment system to have a centralized walled garden approach. This is what one gets with monopolies and duopolies i.e. no incentive to move ahead with the times!