I gave a another shot at the paypal API Today. Since I am most interested in encrypted web payments, after signing up with a business account, I went through the steps of generating a private key and corresponding public certificate.

$ openssl genrsa -out hostname-paypal-priv.pem 1024
$ openssl req -new -key hostname-paypal-priv.pem -x509 \
    -days 365 -out hostname-paypal-pubcert.pem

# Useful command to check certificate expiration
$ openssl x509 -in hostname-paypal-pubcert.pem -noout -enddate

I then went to my paypal account, follow to > Merchant Services > My Account > Profile > My Selling Tools > Encrypted Payment Settings > Add and upload hostname-paypal-pubcert.pem. Write down the CERT_ID, you will need it later on to create payment buttons.

The paypal API comes in many flavors but so far it is only important to understand PayPal Payments Standard vs. PayPal Payments Pro. When you use the first one, PayPal Payments Standard, a visitor will be redirected from your website to paypal's website for payment processing. When you use the second one, PayPal Payments Pro, a visitor can enter payment information directly through your site.

Paypal provides a sandbox to be used to develop and debug your code. Unfortunately the sandbox is quite broken. Some critical links, like "Merchant Services", branch out of the sandbox into the live paypal site. That makes it impossible to upload certificates in the sandbox and thus test your code there.

django-paypal

After uploading a certificate, I search through django packages for an integrated payment solution. django-merchant supports multiple payment processors including paypal. The django-merchant paypal setup documentation deals with PayPal Payments Pro. I am not quite sure django-merchant supports the PayPal Payments Standard. Either way, since it is mostly a wrapper around django-paypal as far as paypal support is concerned, I started there and configured django-paypal itself first.

Through the source code of django-paypal, there is a reference to a paypal with django post using the m2crypto library for encryption.

# install prerequisites
$ apt-get install python-virtualenv python-m2crypto
$ virtualenv ~/payment
$ source ~/payment/bin/activate
$ pip install Django django-registration django-paypal django-merchant

# create a django example project 
$ django-admin startproject example
$ django-admin startapp charge
$ diff -u prev settings.py
INSTALLED_APPS = (
   ...
+'paypal.standard.ipn'
)

+# django-paypal
+PAYPAL_NOTIFY_URL = "URL_ROOT/charge/difficult_to_guess"
+PAYPAL_RETURN_URL = "URL_ROOT/charge/return/"
+PAYPAL_CANCEL_URL = "URL_ROOT/charge/cancel/"
+# These are credentials and should be proctected accordingly.
+PAYPAL_RECEIVER_EMAIL = ...
+PAYPAL_PRIVATE_CERT = ...
+PAYPAL_PUBLIC_CERT = ...
+# path to Paypal's own certificate
+PAYPAL_CERT = ...
+# code which Paypal assign to the certificate when you upload it
+PAYPAL_CERT_ID = ...

$ diff -u prev urls.py
urlpatterns = patterns('',
+ # The order of 
+    (r'^charge/difficult_to_guess/',
+     include('paypal.standard.ipn.urls')),
+    (r'^charge/cancel/', 'charge.views.payment_cancel'),
+    (r'^charge/return/', 'charge.views.payment_return'),
+    (r'^charge/', 'charge.views.paypal_charge'),

$ python manage.py syncdb

# Running the http server
$ python manage.py runserver
$ wget http://127.0.0.1/charge/

Testing IPNs

For each payment processing request, paypal asynchronously calls your web server with the status of that request. That is the second part of the payment pipeline that needs to be tested before going live.

I decided to give a second chance to the Paypal Sandbox on IPN testing. I went through > Test Account > Create a pre-configured account > Business.

"> Test Tools > Instant Payment Notification (IPN) simulator" looks like a promising candidate so I went ahead and entered my site's url for the ipn handler, selected "Express Checkout" left all default values and clicked "Send IPN", result:

IPN delivery failed. Unable to connect to the specified URL. Please verify the URL and try again.

As it turns out, paypal will not connect to your web server on a plain text connection. The error message is just very cryptic. I proxyied the django test server through Apache to support https connections.

$ cd /etc/apache2/mods-enabled
$ ln -s ../mods-available/proxy.load
$ ln -s ../mods-available/proxy_http.load
$ ln -s ../mods-available/proxy.conf
$ diff -u prev proxy.conf
- 	   ProxyRequests Off
+      ProxyRequests On

        <Proxy *>
                AddDefaultCharset off
                Order deny,allow
                Deny from all
+               Allow from 127.0.0.1
        </Proxy>

+       ProxyVia On

$ diff -u prev ../sites-available/default-ssl
+       ProxyPass /charge/ http://127.0.0.1:8000/charge/
+       ProxyPassReverse /charge/ http://127.0.0.1:8000/charge/

+<Location /charge/>
+  Order allow,deny
+  Allow from all
+</Location>

That worked and I could see the paypal request in my apache and django logs. Though now I hit the following error:

IPN delivery failed. HTTP error code 403: Forbidden

Classic django error related to the csrf middleware and a little bit of csrf_exempt magic does the trick.

$ diff -u prev /usr/lib/python/site-packages/paypal/standard/ipn/views.py
+from django.views.decorators.csrf import csrf_exempt

+@csrf_exempt 
@require_POST
def ipn(request, item_check_callable=None): 

The IPN simulator is now showing a success.

Further notes

At some point I encountered HTTP 500 returned code from django without any log showing up. That happened because an import statements was not found. The longest time I spent was to figure out how to display the cause of the error. I finally did it like this.

$ diff -u prev settings.py
LOGGING = {
    'handlers': {
+        'logfile':{
+        'level':'DEBUG',
+        'class':'logging.handlers.WatchedFileHandler',
+        'filename': '/var/log/django.log',
+        'formatter': 'simple'
+        },
    'loggers': {
        'django.request': {
    ...
        },
+       # Might as well log any errors anywhere else in Django
+       'django': {
+           'handlers': ['logfile'],
+           'level': 'ERROR',
+           'propagate': False,
+       },

I was interested to find out how did django-paypal verify the IPN is actually coming from paypal. Looking through the source code I traced the answer from paypal/standard/models.py:verify to paypal/standard/ipn/models.py:_postback. django-paypal post the IPN back to paypal and checks the return code. Wow! I'd trust the DNS server I am using.

django-paypal is using django signals to trigger some other code that should run on an IPN notification. It can be setup as follow:

$ diff -u prev charge/models.py
+ from paypal.standard.ipn.signals import payment_was_successful

+def paypal_payment_was_successful(sender, **kwargs):
+    logging.error("!!! payment_was_successful for invoice %s", sender.invoice)
+payment_was_successful.connect(paypal_payment_was_successful)

With such code models.py need to be imported/executed before an IPN notification is triggered otherwise the signal handler is never set. That's usually not a problem when you trigger the payment pipeline urls in order (charge, ipn). That is something to be aware of though when starting django and directly running the paypal IPN simulator. Signals won't be added and thus triggered. Because of csrf_exempt patch and the signal setup issue, it might be better to add a wrapper to paypal.standard.ipn.views.ipn inside the charge django app.

Some interesting documentation from Record Keeping with Pass-through Variables, you should not that the following variables are passed through paypal back to your website:

• custom
• item_number or item_number_X
• invoice

Originally before using django-paypal, I looked through Paypal Java SDK. The setup required to download a crypto package from bouncycastle and export private keys in pkcs12 format.

# Compiling the code sample
$ curl -O http://www.bouncycastle.org/download/crypto-145.tar.gz
$ tar zxf crypto-145.tar.gz
$ export JAVA_CLASSPATH=~/crypto-145/jars/bcprov-jdk16-145.jar
$ export JAVA_CLASSPATH=$JAVA_CLASSPATH:~/crypto-145/jars/bcpg-jdk16-145.jar
$ export JAVA_CLASSPATH=$JAVA_CLASSPATH:~/crypto-145/jars/bctest-jdk16-145.jar
$ export JAVA_CLASSPATH=$JAVA_CLASSPATH:~/crypto-145/jars/bcmail-jdk16-145.jar
$ javac -g -classpath $JAVA_CLASSPATH ButtonEncryption.java \
  		com/paypal/crypto/sample/*.java

# Converting the private key (remember password for next command)
$ openssl pkcs12 -export -inkey hostname-paypal-priv.pem \
  		  -in hostname-paypal-pubcert.pem \
		  -out hostname-paypal-priv.p12

# Encrypting a paypal button
$ cat testbutton.txt
cert_id=Given when key uploaded to paypal website
cmd=_xclick  
business=sales@company.com
item_name=Handheld Computer  
item_number=1234
custom=sc-id-789  
amount=500.00
currency_code=USD
tax=41.25
shipping=20.00
address_override=1
address1=123 Main St
city=Austin
state=TX
zip=94085
country=US
no_note=1 
cancel_return=http://www.company.com/cancel.htm
$ java -classpath $JAVA_CLASSPATH ButtonEncryption \
  	   hostname-paypal-pubcert.pem \
	   hostname-paypal-priv.p12 \
	   paypal_cert_pem.txt \
	   pkcs12_password \
	   testbutton.txt testbutton.html

I have not completed this work yet but here are the initial notes I currently have on using crypto++ to interface with paypal processing system. Some background articles that turned out to be useful are Cryptographic Interoperability: KeysApplied Crypto++: Block Ciphers, crypto++ CBC Mode and crypto++ key formats.

# Private key that can be loaded through crypto++
openssl pkcs8 -nocrypt -topk8 -in hostname-paypal-priv.pem \
		-out hostname-paypal-priv.der -outform DER