Making your own NetBeans modules

NetBeans 7 is a powerful, free IDE that provides lots of opportunities to customise. If you find yourself continuing to create code that does roughly the same thing (eg. PHP scripts), you may want to make your own intelligent, dynamic template file to make life easier. The technology used to do all this is NetBeans IDE API and FreeMarker - a templating system that provides heaps of functionality, only a small portion of which needs to be used in NetBeans when creating modules.

As an exercise, I decided to make my own PHP template file module that lets you create a new PHP file that connects to a MySQL database and table, and runs a simple query against it. Because I keep forgetting the code (and because most of my PHP scripts do MySQL access), I decided to set up a NetBeans module that allows to you to creates a new PHP file with the shell code for database table access filled in. To use the templated file, it's just a simple right-click to get a panel where you provide a database name, a table name and the fields you want. Out pops a valid PHP script that opens the database, runs a SELECT * FROM xxx etc. query against it, then prints a list of fields you have selected in the panel.

There are some pretty good tutorials to do all this on the NetBeans sites, but a number of gotchas as you utilise version 7 - some of the tutorials are slightly out of date for this version. This posting should hopefully help you resolve those differences, and create your own customised template. Follow the tutorial on the NetBeans site NetBeans File Templates Tutorial while you read this post - I'm really just flagging a few tips to get it going on version 7, with a real-life example.

We'll be using FreeMarker as a template language to link what you put on your 'new file' panel into the actual PHP file you create... this is the timesaving feature.

You'll need the FreeMarker plug in for NetBeans, and I don't believe it's available from the standard plugin sites contained in Tools/Plugins, so you'll need to install the .nbm file manually.

Get the Freemarker Samples .nbm by using the link on NetBeans and FreeMarker Tutorial

You should read this page top-to-bottom to get an understanding of FreeMarker's interface to NetBeans, and can even try out the tutorial to install a standard Java file creator if you wish. If you just want to download and install the .nbm file, get it at NetBeans FreeMarker Sample plug-in

Press download, store it somewhere, then use Tools/Plugin/Downloaded and click 'install' to add it to NetBeans. It comes with a few samples that are immensely useful in understanding this beast.

Unfortunately, it won't compile out of the box unless you remove an unnecessary dependency to org.netbeans.modules.templates. Right click the project (FreeMarkerSupport) and edit the libraries list to remove it.

As the tutorial goes on to say, add in the other required (but missing) dependencies: namely 'Java Project Support', 'Project API', and 'Project UI API'. You should now have a working FreeMarker plug in with a few samples (HTML file, Java file, and even a little form-letter writer).

Now, to create your new module, select New Project, with a type of NetBeans Modules/Module. Give a meaningful name eg. PHPMySQLTemplate, and an appropriate code base name eg. org.yourorganisation.phpmysqltemplate.

Once you've done this, you can create the relevant files, compile and (most importantly), right click the project after compilation to Install/Reload in Development IDE. Create a wizard outline by following instruction 4 on the NetBeans FreeMarker Tutorial.

This will give you most of the files you need namely: Java panels, a java Wizard Iterator, a module description html, and layer.xml. You will be adding your own empty file to use a template (.ftl) file.

Because the purpose is to create our own panel, don't follow instruction 6 - we will customise our panel to collect a couple of fields and use those in the template. (Instruction 6 leaves out panel creation by just substituting the default Java file panels).

Now: on to your own panels. This was a trial-and-error thing for me, but the FreeMarkerSupport Example files were enormously helpful.

For your xxxVisualPanel1.java file, just add the required text boxes/list boxes etc. you need to provide the dynamic data for your eg. PHP file.
There's nothing too complex in here, but two things are important:
1) Add your template variables eg.

static final String DB_NAME = "db_name";
static final String TABLE_NAME = "table_name";
static final Object[] FIELD_NAMES = {"field_names"};

2) Provide accessor methods for your screen text boxes eg.


public String getDBName() {
return jTextField1.getText();
}


Note the Letter example shows you how to get hold of ListBox contents if you need that too.

For your xxxWizardPanel1.java file, you need to provide more accessor functions to get hold of the dynamic variables eg.


public String getDBName() {
return ((PHPMySQLVisualPanel1) component).getDBName();
}

Most importantly, don't forget to update the StoreSettings method with your dynamic screen variables eg.


public void storeSettings(Object settings) {
((WizardDescriptor) settings).putProperty(PHPMySQLVisualPanel1.DB_NAME, getDBName());
((WizardDescriptor) settings).putProperty(PHPMySQLVisualPanel1.TABLE_NAME, getTableName());
((WizardDescriptor) settings).putProperty(PHPMySQLVisualPanel1.FIELD_NAMES.toString(), getFieldNames());
}


This step is the mechanism by which the dynamic screen variables get passed to the template file.

The xxxxWizardIterator.java file links it all together. In the instantiate method, you create a HashMap containing the dynamic variables off the panel. Follow the example file closely so you fill in everything. In particular, note the linkage done by these lines of code:

hashMap.put("db_name", wizard.getProperty(PHPMySQLVisualPanel1.DB_NAME));

This puts DB_NAME off the panel into the HashMap which will be provided to the FreeMarker engine to create a variable you can use in your template like ${db_name}. Sticking to the lower/upper case standards here is helpful ie. make your template variables lower case. All the other code outside instantiate stays as is.

The fun bit is creating your template. I used a PHP file, but of course your new file can be any type you like. The FreeMarker site has a list of all the macro instructions you can provide (eg. #ifs, and <#list). A useful site to explore FreeMarker syntax is at FreeMaker Template Cookbook. Again, follow the example .ftl files to help create a template that should run/compile properly as soon as you add it to your next project.

The layer.xml file is very important. Use an example and modify it from there. Important things here are:
folder xxx - tells NetBeans where the new file can be accessed eg. Templates/PHP
You must also add the following line:


<attr name="javax.script.ScriptEngine" stringvalue="freemarker"/\>


so that NetBeans knows you want to tell FreeMarker to use templating.
You can also make an .html file to describe your wizard file creation, and for completeness, you might want a 16x16 icon to show off your new file. There are two .html files in use: one describes the wizard itself, and the other, some information on the file you are creating.

Lastly, clean/compile and Install/Reload in Development IDE to install your template.

When you compile, be careful of some of the libraries you linked in. The main issue I had was the auto-import fixer brought in javax.tools.FileObject which is NOT the one you want: you want the wizard custom file object version:

import org.openide.filesystems.FileObject;


If all goes to plan, you can now add a new file to any project, filling in a couple of custom items on the panel, then see these correctly inserted at the right places in your new file. As I mentioned, I used all this to create a ready-to-go PHP file that accesses a database table and prints a few fields.

One last gotchya when it comes to using your new template: if you're using the ListBox, you'll need to actually select all the values you want: a list box full of data, but unhighlighted brings in sweet FA!

Given you are controlling the creation of your templated-file with FreeMarker and java code, you have opportunities to do all sorts of exciting things - my next idea is to actually pre-fill the table fields in the list box based on the selected database and table. This MySQL access code will go in xxxVisualPanel1.java file, once the user has chosen a database and table.

In summary, you can use NetBeans to 'intelligently' create new files for you using FreeMarker templating, and using Java code to interact with the NetBeans IDE. Nifty eh?

Sorting in C#

Sometimes we all yearn for the days of DOS sort.exe, or MVS JCL's DFSORT. Sorts were big in the 2-3GL world of COBOL and the like as indexed files were not a given, and sequential file processing was the rule. Batch processing typically ran 'jobs' in steps, and one of those steps was inevitably a sort. Downstream process logic had precise dependencies on input data sort integrity and big-iron OS's all came with their own SORT utilities. The beauty of the sort step was it's clarity: you knew exactly what you wanted, and it was pretty easy to get it.

Although sorts have dropped from their prime position in the pantheon of programming processes, they're still around. Uncommon, but when required, utterly unavoidable.

Almost all programming languages make provision for sorting, and C#'s no exception.

C# sorting comes in a few flavours. The simplest is the sort method that pops up implemented in all sort of useful .NET library set objects like Lists. Of course, this sort of solution is rarely practical given the restriction of having only one sort operand, and we more often end up having to delve into the world of interfaces and OO approaches involving object derivation from useful .NET library components.

I recently had need to do a slightly-complicated sort on multiple keys, handling strings, numbers and datetimes. Here's a quick guide for this kind of situation, with a couple of learnt-the-hard way snippets.

1) Create a custom sort function.
This function is tasked with loading your data into a List (or some other aggregating .NET object type), defining and using your new custom sort IComparer object on this list and lastly, outputting your newly sorted List object into the output file. Running the sort itself involves calling the .Sort method on your list object, passing in the custom comparer object as the parameter.

2) Create a custom IComparer object
The magic (if you can call it that!) happens here.
IComparer-like objects simply compare two bits of data and return a 1 if the first is 'bigger', -1 if it's 'smaller' and 0 if they're the same. It's up to you to implement the logic behind bigger and smaller and in the process you can analyse the records to sort to process date times, strings and numbers (or even other objects). The comparison logic resides in the Compare method you create in this object - it's compulsory.

Two things are important to remember to get this object working correctly:

a) ALL comparison logic paths must be tested. The compiler picks up the fact there are missing paths for you if you forget some, but I guess there's only so much it can do. To abuse a consulting world term, your testing must be 'MECE' : mutually exclusive, completely exhaustive.

b) You should explicitly test for NULL against all the bits of the records you want to compare.

You can see that if you want to compare multiple parts of an object/record, you're going to end up with lots of if's and else's...especially given you need to check NULLs as well. The code can get quite ugly!

The result of your hard work implementing custom sort functionality is quite impressive speed-wise. A few hundred thousand record file with 3 comparison operands runs in 5 or so seconds on my average-fast PC. This is your reward for obeying the .NET paradigm - a somewhat verbose but very well-specified and slightly abstract piece of code works efficiently!

Here's a slightly schematic chunk of code explaining the above.


public static void SortMyFile() {
// Read the whole file into a List
List myLines = new List();
using (StreamReader r = new StreamReader("the file.txt")) {
string theLine;
while ((theLine = r.ReadLine()) != null)
{
myLines.Add(line);
}

// Create the sort object and sort
MyVeryOwnComparer myVeryOwnComparer = new MyVeryOwnComparer();
// Call the List.Sort method with the sorter as parameter.
myLines.Sort(myVeryOwnComparer);

// write out your sorted list
// using StreamWriter or the like
.
.
}

// Your iComparer derived comparing object.
// Create the tests in the Compare method you must implement.
// Don't forget all paths to be tested and test for NULL explicitly.
public class MyVeryOwnComparer : IComparer
{
public int Compare(string x, string y)
{
// break up your x and y strings,
// convert elements as required and return
// 1's, -1's or 0's
.
.
.
}
}




Java and C together

It may seem antithetical, but your bloated java code can utilise your nice lean C libraries quite easily using the JNI interface.

There are two major tasks to get your C stuff into Java:

A. Modify and compile your C file using JNI hooks:

1) Make a special Java C header by calling javah on your .c file:

javah myOriginalCFile.c -o myJavaCFile.h

2) Make a new JNI C file out of your original C file and include the JNI interface and your just-made javah header file. Export your functions to Java by modifying the function signature. Use JNI's special types (eg. jdbouble) for anything coming into your function from Java:

#include <jni.h> /* Java Native Interface headers */
#include "myJavaCFile.h" /* Auto-generated header created by javah -jni */

JNIEXPORT jdouble JNICALL
myJavaCFunction(JNIEnv *env, jobject obj, jdouble x)
{
double x = 0.0;
/* Call the original C function */
y = myOriginalCFunction(x);
return y;
}

3) Compile your file into a shareable lib (architecure specific)

B. Use your new .lib in Java - there are 3 steps:

1) declare your C function in Java using the native keyword. 'native' says: it'll come from a library later. Do this in a static block of your class:

static {
private native double myCFunction(double x);
}

2) Get the library and load it. Note that you may want to set the library path either in the call arguments (-DLD_DYNAMIC_LIBRARY) or call set_library_path. Note that you don't need to include the file ending: Java will look for a .dll, .so, or .dylib depending on where you're creating your file.

System.loadLibrary("myLibName");
// or .dll or .dylib etc.
// System will check for right ending itself

3) Call it! Make a new object of your class, and use the function eg.

MyClass mc = new MyClass();
double myRtnValue = mc.myCFunction(2.33);

Tada.....C goodness for free!

For a more complete (and much better description!) of this process, see the java documentation from Sun at:
java.sun.com/docs/books/jni/html/jniTOC.html

For a link to a practical implementation of a bessel function:

www.nag.co.uk/doc/techrep/html/tr3_00/TRExample1.html

Bessel functions are solutions to Bessel's equation, and are useful for analysing wave propagation and decay in spherical or cylindrical coordinate systems: they are used extensively in signal processing and acoustics. Think drumskin vibrations varying over time. If you want to chart the y-displacement of the skin after your bang the drum, ask a Bessel function.

Why does Carbon form 4 covalent bonds instead of 2?

Most people are not in the slightest bit interested in the answer to this question. For the 1% who are though, this vexing little question is at the heart of organic chemistry - that bitch of subject that seems to drive people alternately mad/ecstatic. I always knew C formed 4 bonds, and got through uni organic chem and some other stuff without really knowing/remembering why. So....years later, I decided to sort myself out.

Here's the answer:

Carbon's valence shell is configured 2s2 2p2 (remember the p shell can contain up to 6 electrons, in a 3-D 2px, 2py, 2pz shape). Two half-filled p orbitals should mean stable molecules like CH2 - the 2 electrons from the H filling the 2px and 2py subshells. But....these don't normally exist ...instead carbon likes to form 4 covalent bonds with other atoms. Why 4 and not 2? Because the 2s2 2p2 shell hybridizes. After hybridization there are NO LONGER any s or p shells. This is really important to understand. They've gone forever!

2s2 2p2 becomes : sp3 sp3 sp3 sp3 (the hybrid).

What's an sp3 shell look like? sp3 shells look nothing like s or p shells...more like a very short stocky baseball bats. They're called sp3 because the s and the 3 p shells combine to make 4 sp3 orbitals. Because there are 4 of them, and they like to be as far away from each other as possible, C forms (single bonding) tetrahedral shapes ie. 4 triangle surface planes.

I'll try to put some picture in to explain this better...I admit that wasn't a great job. The most important thing I realised was that sp3 orbitals are not s and not p. They are completely different, and the old s and p orbitals have gone.

How does a double bond work? Similar deal: let's take ethylene as an example:

H2C=CH2

1 x s and 2 p orbitals create 3 x sp2 orbitals. The remaining p orbital on the 2 C atoms forms a pi bond.

A triple bond's no different: 1 x s and 1 x p form an sp orbital. 2 p's then form 2 pi bonds. Between the carbon atoms, you've got one sigma bond (from the sp's touching), and 2 pi bonds to make 3! (the sp hybrid has nothing to do with the normal p pi bonding). Easy eh?

Hopefully this explanation also clears up that confusing issue of naming the hybrid orbitals. sp3 for 4 bonds, sp2 for double, and sp for triple never made any sense to me, until I realised they're just the formula that represents how many of the old s and p bonds combined.

Where are our paths?

Lost when it comes to Leopard's paths? They're all important when you compile your own stuff and will drive you nuts at some stage, so after quite a bit of stuffing around (my incompetence, not theirs), I've decided to list the path to path in ORDER.

The key thing to note is that in Leopard, in addition to the standard /etc/paths, Apple have created a 'path_helper' script (called from /etc/paths), that picks up path directories from two new special locations.

1) A text file call /etc/paths
2) any file you put in /etc/paths.d

Before this, your own .bash_profile is read from ~/.bash_profile or, for root, /var/root/.profile in the standard way.

So...the order for your paths is:
~/bash_login or ~/.profile (if you have you path like xxxxx:$PATH)
/etc/profile (mine contains JAVA_HOME)
/etc/paths
/etc/paths.d/*

If you've installed your own software in /usr/local, you'll probably want to edit /etc/paths first, as this gets set up first too, and you generally want to override /usr/bin, /bin, /sbin and /usr/sbin to get your own stuff going first.

The general commands in your script are simply
XXXX=/your/path
export XXXX

Entropy PHP and xdebug on Leopard

Like everything else *nix, Leopard comes with a crippled PHP. It's a PHP nonetheless, but as soon as you want to write interesting things you'll find it doesn't have many important modules compiled in.

I mucked around with compiling my own into /usr/local/mysql but found the missing link was always GD....a bugger to compile in. Not wanting to use the bloated, and often outdated ports or fink versions, I relented and finally used a pre-compiled binary from entropy.ch provided by Marc Liyanage. This dude in Switzerland maintains a great binary, Leopard package installable version of PHP with lots of goodies. It's at
http://www.entropy.ch/home, and it sure beats stuffing around with endless compile options from the raw php code.

Of course, one other goody you'll want is xdebug by Derick Rethens - another deutschesprechende by the looks of it. This fantastic addition lets you debug your PHP scripts realtime, and hooked up to Komodo, Netbeans or Eclipse for instance, allows you to lazily trace through your code, inspecting variables (and changing them) as you go. It's a godsend to PHP developers everywhere and very popular, with new features added all the time. You can find xdebug at
http://xdebug.org

To get xdebug going in Marc's binaries you need to take care of a couple of gotchyas.
1) compile it with CFLAGS=-arch x86_64 as that's what Marc's is.
>CFLAGS=-arch x86_64
>export CFLAGS

2) make sure your path finds Marc's PHP first, not Apple's or any other PHP's you've got lying around. This is really important, as otherwise you'll enter version issue in php.ini reminiscent of DLL hell! You do the standard phpize, and ./configure --enable-xdebug --with-php-config=path to php-config.

Manually move the modules/xdebug.so into Marc's directory (mine was /usr/local/php5/lib/extensions/20090623).
The last gotchya was tricky. You load your extra modules in the entropy version php in the php.d directory - just add a new file eg. 50-extensions-xdebug.ini. It will get picked up, however MOST IMPORTANTLY, even though you might have set your extensions_dir in the main php.ini, the [Zend] section in the 50-extensions-xdebug.ini file requires the FULL path to xdebug.so. Nothing else will do.

Thanks to both these guys for making Leopard PHP development easy. Vielen Danke (or, as they say in der Schweize Danke vielmal) dufte Typen weil Ihr PHP coding leicht gemacht habt! (thanks guys you make PHP coding easy).

Ruby - getting the latest on Leopard

Wanna upgrade Leopard's default ruby binary? Unlike most of the other Apple inclGNUsions like MySQL and PHP, they couldn't really cripple this one. Still, we all like the latest greatest stuff right?

Per usual, the process can be painful. To cut a long story short, with a new 64-bit MySQL and PHP installation, you need to do this (in addition to your other ruby upgrade steps) to get the MySQL part going.


env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Otherwise - you'll bomb out as Leopard tries to compile a universal binary ie. a whole lot of binary stuff you don't want/need and can't use. The use of the --with-mysql-config flag is a nifty way to get these things to just WORK with your MySQL installation. Kudos to MySQL for this.