Security
The Basic Challenge is a protocol to allow an application to convince itself that it has been given legitimate permission to execute by a license server. The application gains this conviction by having the application send a challenge to the server and having the server send back a simple function of the challenge and the shared secret. The application checks that the server's response was appropriate for the given challenge. The application and the license server share a secret, and maintaining the privacy of this secret is essential if the mechanism is not to be subverted.
The encryption steps of the Basic Challenge are simple and one-way. Even knowing the entire challenge algorithm (as documented in this appendix) does not compromise the secrets being passed through the algorithm. Nor does it compromise the level of security offered. The License Service API is only as secure as: (1) the secrets themselves; and (2) the steps taken by the software publisher to prevent patching. Even having a more complex challenge/response algorithm for the challenge response does NOT raise the level of security offered. The primary goal is to provide a sufficient level of protection such that the effort to defeat is obvious and intentional. Such an overt effort can serve as evidence in copyright violation prosecutions by organizations such as the Software Publishers Association.
The fundamental weakness of this mechanism is that an attacker can examine the application code to recover the shared secrets, then use it to forge a perfectly correct license system flow for whatever challenge the application happens to issue. This attack cannot be avoided, because the application code is distributed to end users and there is simply no place in it where one can safely embed a secret.
Another possible point of subversion in the application code is where the application validates the challenge response. An attacker can locate the code which does the comparison and modify it. This attack is possible in virtually any environment where the attacker can examine/modify the application binary.
These weakness are pointed out so that the software developer can take defensive steps that may be desired. A software developer may choose to use a more sophisticated challenge mechanism, but it may not be supported by all license systems.
The Software developer can take some steps to reduce attacks such as security by obscurity (make it difficult to identify the critical comparisons, etc.) and anti-virus techniques (checksumed code, etc.). These measures won't make the code undefeatable, but they can make it more difficult.