Using ASCII mode for FTP of text files

Last week, I ran across a common problem -- I had transferred a INI file from Windows to Solaris in binary mode and the file did not work on Solaris. This was obviously due to the difference in newline characters.

I then realized that if we transfer in ASCII mode, then the FTP program automatically converts between windows and unix line feed characters :)
More info at these links:
http://www.editpadpro.com/tricklinebreak.html

Solaris tips - find out RAM and HD size

A few quick commands to help developers new on Solaris 10:

To find out the amount of RAM on the system.
> prtconf / grep Mem

To find out the Hard Disk capacity
> iostat -E / grep Size

To find out the file-system size
>df -h

To find out the CPU usage
> mpstat 12 5
> ps -e -o pcpu -o pid -o user -o args
> top

The following links give more info on common commands:
http://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html
http://www.geocities.com/arndike/Solaris_probe_process.html
http://projects.ericshalov.com/technotes/notes/solaris

Getting the Client IP address on a J2EE server

The HTTPRequest object has 2 methods 'getRemoteAddr()' and 'getRemoteHost()'. These methods would return us the IPAddress/HostName of the last proxy or the client. So if the client is behind a proxy, then we would get the IPAddress of the proxy.
Some proxies (known as transparent proxies) pass the actual IP address of the client in the header HTTP_X_FORWARDED_FOR.
So on the server side we can code something like this:

if (request.getHeader("HTTP_X_FORWARDED_FOR") == null) {
String ipaddress = request.getRemoteAddr();
} else {
String ipaddress = request.getHeader("HTTP_X_FORWARDED_FOR");
}

But if the proxy is an anonymous proxy, then even this won't work. So the only way to get the Client Address correctly is using an Applet to capture the IP address of the client. For this, the client should be trusted and signed.
Another option that I came across in a forum is to create a Socket connection back to the web server from which you came and asking the Socket for the local address:

URL url = getDocumentBase();
String host = url.getHost();
Socket socket = new Socket(host, 80);
InetAddress addr = socket.getLocalAddress();
String hostAddr = addr.getHostAddress();
System.out.println("Addr: " + hostAddr);

Secure FTP options

I was looking for an secure option to FTP files to a remote server.
There are 2 options available: FTPS and SFTP....and I was a bit confused in understanding the differences between the two.

FTPS is FTP over SSL/TLS.
SFTP (Secure FTP) is FTP tunneled thru SSH.

SFTP provides an interactive interface that is similar to that of FTP. In addition, the command line version of SFTP is scriptable in that it allows you to specify a batch file to control the file transfer process.

If U are using VPN, then another option would be to constrain FTP access to circuits created over VPN connections, then as long as the VPN is secure end-to-end, FTP will also be acceptably secure.

File Encryption utility for windows

I was looking for a small light-weight encryption utility for encrpyting my personal information in files. There are a plethora of programs available on the internet, but I found the following ones cool and easy to use.

Kryptel Lite: http://www.kryptel.com/products/krlite/
Iron Key: http://www.kryptel.com/products/ikey/index.php

FTPing in Java

For performing FTP operations programmatically, I use the Jakarta Commons Net Package. This is a super-cool utility package that makes writting FTP programs a breeze.
The jar file can be downloaded from here.

Tip: The jakarta-commons-net package also requires the ORO package which can be downloaded here.

Another important concept is that of Active and Passive FTP. A good explanation of the differences can be found here.
As a general rule, passive FTP is used as it works behind firewalls too.

The windows FTP can also be run unattended using the '-s' option and specifies a script file that contains the FTP commands.

URL encoding and XML/HTML encoding

Recently in one of our applications we were passing XML data as a POST parameter to the server. For e.g. xml={xml string here}

We were using XML encoding to replace special characters in the XML. For e.g. '&' was replaced with '&amp;' '<' was replaced with & lt; etc.

But on the server side whenever we were trying to get the XML string as a request parameter, we were getting a truncated string. After putting in a sniffer, I found out the reason why this was happening - I was getting confused between URL encoding and XML encoding.

In URL encoding as we know, we need to encode special characters such as '&'. So we replace them with '&amp'. But what if '&' was part of the data; i.e. an address contained &. In this case the URL encoding function should just replace '&' with '%26'. Otherwise the server would treat it as a parameter separator and break the string on the first occurance of '&'.

Now our XML string (passed as a POST parameter) contained the & character, because all XML special characters were encoded with a '&' at the beginning. So the trick in this case was to perform URL encoding of the element values after the XML encoding. Confused??

Ok...consider the following XML element:
<name>Naren&Sons<\name>
Now the above element data contains an & and hence would result in a XML parsing error. Hence I XML encode the data. So it would not look like:
<name>Naren&amp;Sons<\name>

But when this string is passed as a HTTP GET/POST param, then it would be truncated.
Hence we have to URL encode it - replace & with %26
<name>Naren%26amp;Sons<\name>

So when the string reaches the server, the request.getParameter would perform the URL decoding and then we passed the XML string to Castor or any other XML parser.

Another good design to avoid all this confusion is to pass the XML in the request body stream and get the XML on the server side using request.getInputStream().

Even ASP.NET has 2 methods: HtmlEncode and UrlEncode.
HtmlEncode converts the angle brackets, quotes, ampersands, etc. to the entity values to prevent the HTML parser from confusing it with markup.
UrlEncode converts spaces to "+" and non-alphanumeric to their hex-encoded values. Again this is to prevent the the URL parser from misinterpreting an embedded ?, & or other values.