Vamsi Pavan’s Place

Open source curl is one of best & stable http client tool as well as library. In this article, we’ll see how can we use libcurl library as well as curl command line utility for two way client authentication.

Using libcurl library for HTTPS client authentication:

Here are the basics steps for the client setup before going into actual libcurl code.

 1. Extract client certificate and client private key files in PEM format from the client keystore.

We have client.p12 keystore file in pkcs#12 format and it’s pass phrase. Following commands can be used further.

 1.1 Extract client certificate without key.

openssl pkcs12 -in client.p12 -nokeys -out clientCert.pem

 1.2 Extract client private key without cert.

openssl pkcs12 -in client.p12 -nocerts -out privateKey.pem

with PEM password. For both above commands ask for keystore pass phrase and only while generating key without cert ask for PEM password (new password will be setup for that key).

 1.3 Extract both client certificate & key in single file.

openssl pkcs12 -in client.p12 -out combinedClient.pem -clcerts

This prompts for both keystore & PEM pass phrase. Generated file contains both cert & key.

 2. Option step to verify generated files to crosscheck.

openssl x509 -noout -modulus -in clientCert.pem | openssl md5

d7207cf82b771251471672dd54c59927

openssl rsa -noout -modulus -in privateKey.pem | openssl md5

Enter pass phrase for privateKey.pem:

d7207cf82b771251471672dd54c59927

Both these md5 outputs are same and that confirms both are good to go.

 3. Libcurl client code for client authentication.

Using following snippet code for curl easy handle. Actually, for client certificate we don’t need to set pass phrase, still setting the same PEM pass phrase of client key.

curl_easy_setopt(curl,CURLOPT_SSLCERT,”clientCert.pem”);

curl_easy_setopt(curl,CURLOPT_SSLCERTPASSWD,”changeit”);

curl_easy_setopt(curl,CURLOPT_SSLCERTTYPE,”PEM”);

curl_easy_setopt(curl,CURLOPT_SSLKEY,”privateKey.pem”);

curl_easy_setopt(curl,CURLOPT_SSLKEYPASSWD,”changeit”);

curl_easy_setopt(curl,CURLOPT_SSLKEYTYPE,”PEM”);

Some times, setting client certificate & client key like this by extracting separately will not work with libcurl. It ends with following error sometimes.

*err unable to set private key file: ‘C:\privateKey.pem’ type PEM*

Reasons are unknown atleast for me. In that case, solution I found is that use your combinedClient.pem file we generated above in place of client key and client pass phrase in above code snippet (client certificate is optional if you are using combinedClient.pem file). I tried manually appending both client cert & key, but that’s didn’t help, so better generate with command only.

 4. Curl command line tool for client authentication:

Following curl command sends C:\myrequest.xml file content as binary HTTP request content with headers SOAPAction & Contenty-Type fields and client cert & client key set to the final url with verbose mode.

$ curl –data-binary @”C:\myrequest.xml” –header “SOAPAction: ” –header
“Content-Type: text/xml” –cert c:\clientCert.pem –cert-type PEM –key
c:\privkey.pem –key-type PEM –cacert c:\ca-bundle.crt https://mydomain.myco.com:443/soap -v

It prompts for PEM passwd and then following error.

* About to connect() to mydomain.myco.com port 443 (#0)
*   Trying 69.181.219.20… connected
* Connected to mydomain.myco.com (69.181.219.20) port 443 (#0)
Enter PEM pass phrase:
* unable to set private key file: ‘privateKey.pem’ type PEM
* Closing connection #0
*curl: (58) unable to set private key file: ‘privateKey.pem’ type PEM*

And then I tried appending both private key along with cert in a single file with format —–RSA CERTIFICATE START —– & —-RSA CERT END —– then immediately with ——CERTIFICATE START —– & —–CERTIFICATE END —– and tried following.

$ curl –cert testCert.pem –Verbose -H “Content-Type: text/xml”
https://mydomain.myco.com:443/soap
* About to connect() to mydomain.myco.com port 443 (#0)
*   Trying 69.181.219.20… connected
* Connected to mydomain.myco.com (69.181.219.20) port 443 (#0)
Enter PEM pass phrase:
* unable to set private key file: ‘testCert.pem’ type PEM
* Closing connection #0
*curl: (58) unable to set private key file: ‘testCert.pem’ type PEM*

Finally, I used the above mentioned command for combined file and then I got the response properly back as below. Following command is used to send both key & cert in a single file with option –cert and –cacert option to set cacert bundle file.

$ curl –cert combinedClient.pem –data-binary @”request.xml” –Verbose -H “Content-Type: text/xml”
–cacert “ca-bundle.crt”  https://mydomain.myco.com:443/soap

* About to connect() to mydomain.myco.com port 443 (#0)
*   Trying 69.181.219.20… connected
* Connected to mydomain.myco.com (69.181.219.20) port 443 (#0)
Enter PEM pass phrase:
* successfully set certificate verify locations:
*   CAfile: ca-bundle.crt
  CApath: /usr/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*        subject: C=US; ST=New York; L=New York; O=myco; OU=NDIS; CN=mydomain.myco.com
*        start date: 2011-06-23 00:00:00 GMT
*        expire date: 2012-07-29 23:59:59 GMT
*        common name: mydomain.myco.com (matched)
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of
use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 International Ser
ver CA - G3
*        SSL certificate verify ok.
> POST /soap HTTP/1.1
> User-Agent: curl/7.19.6 (i686-pc-cygwin) libcurl/7.19.6 OpenSSL/0.9.8n zlib/1.
2.3 libidn/1.18 libssh2/1.2
> Host: mydomain.myco.com
> Accept: */*
> Content-Type: text/xml
> Content-Length: 586
>
< HTTP/1.1 200 OK
< Date: Tue, 27 Sep 2011 09:16:19 GMT
< Server: ACE XML Gateway
< Content-Type: text/xml
< Content-Length: 498
<
* Connection #0 to host mydomain.myco.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
<?xml version=”1.0″ encoding=”UTF-8″?><env:Envelope xmlns:env=”http://schemas.xm
lsoap.org/soap/envelope/” xmlns:dlws=”http://mydomain.myco.com”>
<env:Body><dlws:MyResponse><dlws:statusCode><dlws:c
ode>0</dlws:code><dlws:description>Success</dlws:description></dlws:statusCode><
dlws:requestId>f828131e-bab0-4f50-96d0-a2512c7926d4</dlws:requestId><dlws:respon
seId>1317114979-190385186</dlws:responseId></dlws:MyResponse></env:Bo
dy></env:Envelope>

From all this, the learning I guess is curl behavior is not very sure with respect to client authentication. Some times it works perfectly if we send client cert & key separately, but some times it works only with combined file.

This article covers to setup both client and server for SSL authentication.

Consider we have java server and keytool utility comes with java sdk.

To create a server certificate follow these steps:

    1. Create the keystore.
    2. Export the certificate from the keystore.
    3. Sign the certificate.
    4. Import the certificate into a trust-store: a repository of certificates used for verifying the certificates. A trust-store typically contains more than one certificate.

From the directory in which you want to create the keystore, run keytool with the following parameters.

    1. Generate the server certificate.

    <JAVA_HOME>\bin\keytool -genkey -alias <server-alias> -keyalg RSA -keypass changeit -storepass changeit
    -keystore keystore.jks

    When you press Enter, keytool prompts you to enter the server name, organizational unit, organization, locality, state, and country code. Note that you must enter the server name in response to keytool’s first prompt, in which it asks for first and last names. For testing purposes, this can be localhost. The host specified in the keystore must match the host identified in the host variable specified in /etc/hosts. keystore file can have .jks or .p12 extension. Here we are specifying two pass phrases for each of key & keystore.

    2. Export the generated server certificate in keystore.jks into the file server.cer.

    <JAVA_HOME>\bin\keytool -export -alias <server-alias>  -storepass changeit -file server.cer -keystore keystore.jks

    If you want to have the certificate signed by a CA, then we need to generate CSR.
 
    2.1 Generate a Certificate Signing Request (CSR)

    <JAVA_HOME>\bin\keytool -certreq -sigalg RSA -alias <server-alias> -file <csr-filename>

    Send the contents of <csr-filename> for signing to CA. Then you get another server.cer file from CA after it is CA signed.

   2.2 Generate trust-store for server

    To create the trust-store file cacerts.jks and add the server certificate to the trust-store, run keytool from the directory where you created the keystore and server certificate. Use the following parameters:

    <JAVA_HOME>\bin\keytool -import -v -trustcacerts -alias <server-alias> -file server.cer
    -keystore cacerts.jks -keypass changeit -storepass changeit

    Information on the certificate, such as that shown next, will display. Output of the above command will be.

    Owner: CN=localhost, OU=Sun Micro, O=Docs, L=Santa Clara, ST=CA, C=US
    Issuer: CN=localhost, OU=Sun Micro, O=Docs, L=Santa Clara, ST=CA, C=US
    Serial number: 3e932169
    Valid from: Tue Apr 08
    Certificate fingerprints:
    MD5: 52:9F:49:68:ED:78:6F:39:87:F3:98:B3:6A:6B:0F:90
    SHA1: EE:2E:2A:A6:9E:03:9A:3A:1C:17:4A:28:5E:97:20:78:3F:
    Trust this certificate? [no]:
    Enter yes, and then press the Enter or Return key. The following information displays:

    Certificate was added to keystore
    [Saving cacerts.jks]

With this we have created a server certificate and configured trust store. Depending on the application you host, you need to make sure of these server.cer file for server certificate along with it’s pass phrase & cacerts.jks for trusted-store.

Creating a Client Certificate for Mutual Authentication:

To create a keystore named client-keystore.jks that contains a client certificate named client.cer, follow these steps:

    1. Generate the client certificate.

    <JAVA_HOME>\bin\keytool -genkey -alias <client-alias> -keyalg RSA -keypass changeit
    -storepass changeit -keystore keystore.jks
   
    2. Export the generated client certificate into the file client.cer.

    <JAVA_HOME>\bin\keytool -export -alias <client-alias>
    -storepass changeit -file client.cer -keystore keystore.jks

    Add the certificate to the trust-store file cacerts.jks in the server. Run keytool from the directory where you created the keystore and client certificate. Use the following parameters:

    <JAVA_HOME>\bin\keytool -import -v -trustcacerts -alias <client-alias> -file client.cer
    -keystore cacerts.jks -keypass changeit -storepass changeit

    The keytool utility returns this message:

    Owner: CN=J2EE Client, OU=Java Web Services, O=Sun, L=Santa Clara, ST=CA, C=US
    Issuer: CN=J2EE Client, OU=Java Web Services, O=Sun, L=Santa Clara, ST=CA, C=US
    Serial number: 3e39e66a
    Valid from: Thu Jan 30 18:58:50 PST 2003 until: Wed Apr 30
    19:58:50 PDT 2003
    Certificate fingerprints:
    MD5: 5A:B0:4C:88:4E:F8:EF:E9:E5:8B:53:BD:D0:AA:8E:5A
    SHA1:90:00:36:5B:E0:A7:A2:BD:67:DB:EA:37:B9:61:3E:26:B3:89:46:
    32
    Trust this certificate? [no]: yes
    Certificate was added to keystore

With this we have created client cert and imported in server’s trusted-store, completed client setup. Coming back to server again.

To check the contents of a keystore that contains a certificate with an alias <server-alias>,
use this command:

keytool -list -keystore keystore.jks -alias <server-alias> -v

To check the contents of the cacerts file,
use this command:

keytool -list -keystore cacerts.jks

With this, we have covered server and client setup for SSL certificates for both one way & two way authentication.

Next section, we’ll see how can we use libcurl as http client for HTTP(S) authentication with both libcurl API & curl command.

Before going into any details of SSL, best thing is to know the following terminology and file formats. HTTP(s) is basically used for secure transactions in HTTP like payments etc.

HTTPS : HTTP protocol on Secure shell which is encrypted to outside for any communication exchange. That way, it is quite secure in HTTP communication.

Keystore : keystore is basically a placeholder for a pair of public & private keys. It can hold multiple such pairs.

Key : It’s basically a string which is encoded in base64 format (I guess).

PKCS12 : This is the format of keystore file we create by default using openssl or keytool. You can see the respective commands in next part of this series. All most all the browsers expects keystore should be in this format to import the respective certificates. Normally, all keystore files will have extension .p12 in general.

PEM : This is the format of a file that can be certificate or key or keystore.

DER : This another format similar to PEM. But PEM is more popular in use.

Certificate : This basically a digital signature generated using the above pair of public & private keys. This represents some identity of a machine in the secure HTTP world of internet.

Self signed : After we create a certificate, the next step would be some one has to sign it. If you create a certificate and you are signing the same certificate, then it’s called self signed. After this, you need to copy your certificate to ca-bundle.cer file in both the machines. This way, you are trusting your own signed certificate.

CA signed : There are list of CAs (Certificate Authorities) available in the internet who has the right to sign a cert and it is considered to be trusted among all the internet. All most all operating systems and browsers have their certificates listed in that default ca-bundle.cer file as trusted. Though these CA signed certificates are very costly.

cert/cer : Standard extension for certificate files which can be either PEM or DER format.

keytool : a utility tool to generate these keystore/keys/certificates supplied along with java sdk.

openssl : similar to keytool another open source utility tool.

One way authentication in SSL : In this case, server machine hold a certificate for any HTTPS authentication. When client sends requests to server, first server send this CA signed certificate to identify itself. Client machine check this certificate to it’s default list of trusted certificates from a file called ca-bundle.cer and depending on that it can further communicate to server. If it find that the certificate is not there in that list, it can have two choices. Either it can proceed further for communication ignoring security aspects or it can stop further communication as it finds it’s not a trusted server.

Two way authentication is SSL : In this case, both server and client machines have their own CA singed certificates. When client sends request to server, it gets server certificate. It then verifies the certificate trusted or not, then sends back with it’s own certificate, then server verifies the same and depending on verification, communication starts on HTTP(S) medium. This method is kind of tricky, you need to configure you HTTP client (by which you will be sending request to server, normally a web browser) with client certificate as well as private key too.

We’ll see the commands to create these keystores, keys, certificates and along communicate with servers using curl (open source http client) in our next part of this series.

For Part 2 click here.

Legend goes that King Veeraballa of Vijayanagara once lost his way in
forest. Hungry and tired, he came upon a lone hut in the thick forest
where he met an old woman.When he asked for food, she gave him baked
beans (’Benda Kalu’ in Karnataka).The King found this humble meal taste
better than the richest fare.To commemorate this incident, he called the
place “Benda Kalu Ooru” (place of baked beans). Bangalore today is
getting popular though for a different variety of Beans-JavaBeans .

Basics of validating xmls with a given schema in C++.

1. Create parser instance.

SAX2XMLReader* parser = XMLReaderFactory::createXMLReader();

2. Set required features to parser instance as follow.

// Enable the parser’s schema support
parser->setFeature(XMLUni::fgXercesSchema, true);

// Schema validation requires namespace processing to be turned on.<br />parser->setFeature(XMLUni::fgSAX2CoreValidation,true);<br />parser->setFeature(XMLUni::fgSAX2CoreNameSpaces,true);

3. Set schema location using setPropery api call with/without namespace. If we want use ‘ExternalSchemaLocation’ property we need to append Namespace with a space char and then schema file path.

// Define the location of the schema.
XMLCh* schemaLocation = XMLString::transcode(”/directory/path/myschema.xsd”);
parser->setProperty(XMLUni::fgXercesSchemaExternalNoNameSpaceSchemaLocation,schemaLocation);

Current parser version requires the path in below format.

XMLCh* propertyValue = XMLString::transcode(”myschema.xsd”);
ArrayJanitor janValue(propertyValue);
parser->setProperty(XMLUni::fgXercesSchemaExternalNoNameSpaceSchemaLocation,propertyValue);

Another important thing to remember is - always file path should be in “file:/// “. If you don’t follow this format, parser won’t complain anything but validation/parsing don’t go well. Fortunately, if you are using in java, File API class provides you getURL() call to get the path in file protocol.

4. Now, set the content as well as error handler to parser instance. Remember always use custom handler by inheriting from DefaultHanlder while setting these. For error handler, inherited methods error(), warning(), fatal() needs to be overridden, otherwise parsing/validation errors go unnoticed without catching the exceptions.

parser->setContentHandler((ContentHandler*) myContentHandler);<br />parser->setErrorHandler((ErrorHandler*) myContentHandler);

5. Finally, parse api call.

// Do the parse<br />parser->parse(*xmlInputSource);

6. Now complete code would be.

Now, we’ll see the common errors we face during validation development. For most of the errors, we should make sure that Schema is having or not having target namespace. According to that parser/validator behave further.

1. Character ‘<’ is grammatically unexpected
Cause: missing required tag.

2. [cvc-elt.1: Cannot find the declaration of element
Cause: no namespace is found in the xml or Parser’s schemalocation property doesn’t have namespace attached or Schema files were missing at the specified path for parser’s schemalocation property.

This list will be updated in future too.

There was once a washer man who had a donkey and a dog.

One night when the whole world was sleeping, a thief broke into the house, the washer man was fast asleep but the donkey and the dog were awake.

The dog decided not to bark since the master did not take good care of him and wanted to teach him a lesson.

The donkey got worried and said to the dog that if he doesn’t bark, the donkey will have to do something himself. The dog did not change his mind and the donkey started braying loudly.

Hearing the donkey bray, the thief ran away, the master woke up and started beating the donkey for braying in the middle of the night for no reason.

Moral of the story ”Donkey shall Donkey’s work and Dog Shall do Dog’s work’
—–
PS: The extension of this story is :
—–
The washer man was a well educated man from a premier management institute.

He had the fundas of looking at the bigger picture and thinking out of the box. He was convinced that there must be some reason for the donkey to bray in the night.

He walked outside a little and did some fact finding, applied a bottom up approach, figured out from the ground realities that there was a thief who broke in and the donkey only wanted to alert him about it.

Looking at the donkey’s extra initiative and going beyond the call of the duty, he rewarded him with lot of hay and other perks and became his favorite pet.

The dog’s life didn’t change much, except that now the donkey was more motivated in doing the dogs duties as well. In the annual appraisal the dog managed a ” meets requirement” Soon the dog realized that the donkey is taking care of his duties and he can enjoy his life sleeping and lazing around.

The donkey was rated as “star performer”. The donkey had to live up to his already high performance standards. Soon he was over burdened with work and always under pressure and now is looking for a job rotation…

If you have worked in a corporate environment, I am sure you have guessed the characters of the new story.

There was a salt merchant who used a donkey to transport salt bags. On their way they had to cross a stream of water slightly less than knee deep.

Accidentally the donkey fell one day in the waters and to its surprise it noticed reduction in its load and hence forth it made it a practice to do it daily.

Merchant was helpless as he was losing some salt in the transport resulting in loss. He decided to teach the donkey a lesson and one day loaded it bags of cotton instead of salt.

The donkey was thrilled as the initial load itself was very light, nevertheless greed was part of now intelligent donkey and it dipped in water as usual …………….. and somehow it made it home.

Moral of the story , assume all are smart in this world !

One day a farmer’s donkey fell down into a well. The animal cried piteously for hours as the farmer tried to figure out what to do. Finally, he decided the animal was old, and the well needed to be covered up anyway; it just wasn’t worth it to retrieve the donkey.

He invited all his neighbors to come over and help him. They all grabbed a shovel and began to shovel dirt into the well. At first, the donkey realized what was happening and cried horribly. Then, to everyone’s amazement he quieted down.
A few shovel loads later, the farmer finally looked down the well. He was astonished at what he saw. With each shovel of dirt that hit his back, the donkey was doing something amazing. He would shake it off and take a step up.

As the farmer’s neighbors continued to shovel dirt on top of the animal, he would shake it off and take a step up. Pretty soon,
everyone was amazed as the donkey stepped up over the edge of the well and happily trotted off!

MORAL :
Life is going to shovel dirt on you, all kinds of dirt. The trick to getting out of the well is to shake it off and take a step up. Each of our troubles is a steppingstone. We can get out of the deepest wells just by not stopping, never giving up! Shake it off and take a step up.

Remember the five simple rules to be happy:

1. Free your heart from hatred - Forgive.

2. Free your mind from worries - Most never happen.

3. Live simply and appreciate what you have.

4. Give more.

5. Expect less

and the last bust not the least…

6.For success always network..network the in thing in today’s business

Basics of validating xmls with a given schema.

1. Create parser instance.

SAX2XMLReader* parser = XMLReaderFactory::createXMLReader();

2. Set required features to parser instance as follow.

// Enable the parser’s schema support
parser->setFeature(XMLUni::fgXercesSchema, true);

// Schema validation requires namespace processing to be turned on.

parser->setFeature(XMLUni::fgSAX2CoreValidation,true);

parser->setFeature(XMLUni::fgSAX2CoreNameSpaces,true);

3. Set schema location using setPropery api call with/without namespace. If we want use ‘ExternalSchemaLocation’ property we need to append Namespace with a space char and then schema file path.

// Define the location of the schema.
XMLCh* schemaLocation = XMLString::transcode(”/directory/path/myschema.xsd”);
parser->setProperty(XMLUni::fgXercesSchemaExternalNoNameSpaceSchemaLocation,schemaLocation);

Current parser version requires the path in below format.

XMLCh* propertyValue = XMLString::transcode(”myschema.xsd”);
ArrayJanitor janValue(propertyValue);
parser->setProperty(XMLUni::fgXercesSchemaExternalNoNameSpaceSchemaLocation,propertyValue);

Another important thing to remember is - always file path should be in “file:///

4. Now, set the content as well as error handler to parser instance. Remember always use custom handler by inheriting from DefaultHanlder while setting these. For error handler, inherited methods error(), warning(), fatal() needs to be overridden, otherwise parsing/validation errors go unnoticed without catching the exceptions.

parser->setContentHandler((ContentHandler*) myContentHandler);

parser->setErrorHandler((ErrorHandler*) myContentHandler);

5. Finally, parse api call.

// Do the parse.

parser->parse(*xmlInputSource);

6. Now complete code would be.

    SAX2XMLReader* parser = XMLReaderFactory::createXMLReader();
parser->setFeature(XMLUni::fgSAX2CoreValidation, true);
parser->setFeature(XMLUni::fgSAX2CoreNameSpaces, true);

//* Enable strict validation
parser->setFeature(XMLUni::fgSAX2CoreNameSpacePrefixes, true);
parser->setFeature(XMLUni::fgXercesValidationErrorAsFatal, true);    //* Enable the parser’s schema support
parser->setFeature(XMLUni::fgXercesSchema, true);
parser->setFeature(XMLUni::fgXercesSchemaFullChecking, true);
parser->setFeature(XMLUni::fgXercesDynamic, false);

XMLCh* propertyValue = XMLString::transcode(m_sDefSchema.getMBCSCopy());
ArrayJanitor<XMLCh> janValue(propertyValue);

//* Define the location of the XML schema.
if(isNS)  //with/without namespace
//Property name - http://apache.org/xml/properties/schema/external-schemaLocation
parser->setProperty(XMLUni::fgXercesSchemaExternalSchemaLocation,propertyValue);
else
//Property name - http://apache.org/xml/properties/schema/external-noNameSpaceSchemaLocation
parser->setProperty(XMLUni::fgXercesSchemaExternalNoNameSpaceSchemaLocation,propertyValue);

Now, we’ll see the common errors we face during validation development. For most of the errors, we should make sure that Schema is having or not having target namespace. According to that parser/validator behave further.

1. Character ‘<’ is grammatically unexpected
Cause: missing required tag.

2. [cvc-elt.1: Cannot find the declaration of element
Cause: no namespace is found in the xml or Parser’s schemalocation property doesn’t have namespace attached or Schema files were missing at the specified path for parser’s schemalocation property.

This list will be updated in future too.